"What Works is based on the principle that good decision-making should be informed by the best available evidence. If evidence is not available, decision-makers should use high quality methods to find out what works." So says the Government website.
Stands to reason, doesn't it? Evidenced based public policy. Spend scarce resources on what works.
Increasingly often in education, it seems, the evidence comes from randomised control trials. Well maybe we should pause for a little thought?
For instance: "Understandings and Misunderstandings about RCTs" (with thanks to Andrew Old @oldandrewuk on Twitter for pointing out this blog post.)
"Policy makers and the media have shown a remarkable preference for Randomized Controlled Trials or RCTs in recent times. After their breakthrough in medicine, they are increasingly hailed as a way to bring human sciences into the realm of ‘evidence’-based policy. RCTs are believed to be accurate, objective and independent of the expert knowledge that is so widely distrusted these days."
The blog post refers to a paper "Understanding and Misunderstanding Randomized Controlled Trials". (August 2016. The link is to a free downloadable .pdf paper.) "Recent randomized trials in economic development have attracted attention, and the idea that such trials can discover 'what works' has been widely adopted in economics, as well as in political science, education, and social policy."
If this sort of thing interests you - and for all of us struggling to make the public policy case for conductive education, that might well be you - you'll find it a most interesting paper, if not a wholly easy read!
This is the Abstract:
"RCTs are valuable tools whose use is spreading in economics and in other social sciences. They are seen as desirable aids in scientific discovery and for generating evidence for policy. Yet some of the enthusiasm for RCTs appears to be based on misunderstandings:
- that randomization provides a fair test by equalizing everything but the treatment and so allows a precise estimate of the treatment alone;
- that randomization is required to solve selection problems;
- that lack of blinding does little to compromise inference;
- and that statistical inference in RCTs is straightforward, because it requires only the comparison of two means.
"None of these statements is true.
"RCTs do indeed require minimal assumptions and can op- erate with little prior knowledge, an advantage when persuading distrustful audiences, but a crucial disadvantage for cumulative scientific progress, where randomization adds noise and undermines precision. The lack of connection between RCTs and other scientific knowledge makes it hard to use them outside of the exact context in which they are con- ducted.
"Yet, once they are seen as part of a cumulative program, they can play a role in building general knowledge and useful predictions, provided they are combined with other methods, including conceptual and theoretical development, to discover not “what works,” but why things work. Unless we are prepared to make assumptions, and to stand on what we know, making statements that will be incredible to some, all the credibility of RCTs is for naught.