Tom Bennett

Home » Bad Science » Thinking in the right direction; just don’t put all your faith in RCTs. Ben Goldacre’s vision for evidence based learning.

Thinking in the right direction; just don’t put all your faith in RCTs. Ben Goldacre’s vision for evidence based learning.


But is it outstanding?

Bethnal Green Academy, a soy latte’s throw from both hipster Columbia Road and the surrounding estates, was the venue for the launch of Ben Goldacres’s new advocacy project: Building Evidence into Education. Nice looking school; it’s got BSF written all over every plane and pane. The livery outside the school shouted every second sentence of the latest Oftsed report. Most pleasingly it was styled in the Star Wars font (or as my kids call it, ‘That old film’). 

Hosted by Teach First, the Royal Marines of the teacher profession, we were first treated to an introduction by Dame Gove himself in what was, I thought, a remarkably short set. It was like booking Geoff Capes and asking him to open a jam jar. Goldacre followed;  a passionate and determined thinker and speaker, whose Bad Science series shook me, like Hume did for Kant, from my dogmatic slumber.
It’s a familiar saw for him: the need for Randomised Controlled Trials (RCTs) to become more prevalent in the social sphere (here represented by education), as they are in the medical profession. He made an interesting comparison: medical practitioners at first resisted RCTs because they were convinced of their own intuition, until it turned out that intuition often led us down the wrong path; eg using steroids for brain inflammation did more harm than help- the reverse of orthodoxy. Now, so are teachers (resisting RCTs, not harming the brain. Oh wait.)

Goldacre’s point is refreshing and disarming: we run trials like this for every pill and potion; why not policies, which in the galactic scale of things, dwarf even pharma for dollar tonnage. And as usual, there is a lot of practical wisdom in what he says. I am delighted that the DfE is looking into ways of conducting research that is robust and real, rather than fanciful and driven by dogma.
So why not have RCTs in the social sciences? One possible objection, the ethical, can be easily dismissed: issues of consent, and issues of knowingly withholding a potentially useful intervention from the control group. But we already introduce dozens of interventions in schools every single day. It isn’t as if we aren’t already drowning our kids in optimistic hoodoo. Why NOT randomise it once in a while? When was the last time we asked a kid’s consent before we popped a Thinking Hat on them, or told them to rub their brain buttons? 
I think there are, however, some serious problems with the use of RCTs in education.
 RCTs aren’t the answer to the question ‘What really works in schools?’ because differences in context can never satisfactorily be ironed out. The causal density of humanity is too high; there are too many factors to establish a reliable protocol that could hope to encompass the variables of the human mind. There are as many factors to juggle as there are grains of sand on the beach. Lazy research is sodden in bias and assumptions, wooly over interpretation and optimism. That said, there is room for quantitative research , some of which will be amenable to RCTs. I’ve heard social scientists say that the problem is merely one of design; I say the problem is the methodology itself. If you’re making quantifiable predictions about the physical world you need to provide unambiguous methods of establishing initial conditions as well as outcomes, possible causal mechanisms, and demonstrate reproducability. That isn’t easy.  

Also, the problems of cognitive bias, the Hawthorne Effect, the John Henry Effect, the Pygmalion Effect and so on are enormous obstacles to appreciating when an intervention is effective and when it is not.

This is definitely thinking in the right direction. Goldacres’s attempt to lasso the spoon bending of social science and pin it down to some kind of methodological rigour, is exactly what I, and many teachers, want to hear, and I absolutely support his attempts to reinvigorate the debate. But there are many obstacles to this, even before considering motivation.
One is time. I work pretty hard. I don’t expect a biscuit, because I bet you do too. I’m also pretty good with my time. And I barely have time to do, well…this kind of thing, and the only reason I do is because I cash in hours of sleep like chips in a casino. Doing research into education is going to be a niche pursuit until workloads go down. Like archaeology in the 19th century, it will remain the hobby of gentlemen.
And let’s be honest, most teachers aren’t scientists. They’re as prone to their pet prejudices and tender perversities as most people. Read the Tooley report if you want to see how badly education research can be conducted even by people who should know better. Now multiply that by 100 and you get, I would say, most of us. Many teachers would love to get involved with what Goldacre sensibly described as driving research.But would research benefit? Not until teachers learn what the scientific method is.

Dating for Nerds
So, what next? From the presentations, and the discussion one thing became apparent: this isn’t the launch of anything concrete yet. At this point, the program is advocacy. There was no funding committed, no projects starting tomorrow. From one perspective, it’s hard to see many education secretaries getting too worked up about RCTs- the average tenure of the Headmaster-in-chief is about 18 months. RCTs can take years; to be efficient, to be meaningful, they have to be as large and long as possible (as the inspector said to the janitor). Why invest in something that a) won’t bear fruit until after you’ve been moved to The Department of Silly Walks and b) Might disagree with your own pet projects? It’s always safer to simply pick research that appears to validate your own objectives.
And yet, and yet. Gove was, at least, there. The Dfe’s involvement with Goldacre shows, at least, a symbolic commitment to better research in education. And let’s not forget the £125 million DfE funding for the Education Endowment Fund (represented here by Dr Kevan Collins) a Sutton Trust start-up devoted to research, some of which involves RCTs. There might be some will there, but it isn’t very full-throated. 
So much in education is an abstract, an artefact of art as much as engineering. We can barely agree as to what educated means. Or learning. Or thinking. Or engagement. Or creativity. These aren’t amenable to metrification.How do we study what we cannot catch?
But to sound a more positive note: anything which seeks to firm up the wiliest of educational research is an asset.  Goldacres’s profile should help turn a search light upon the relationship between research community and Chalkface warriors. He’s absolutely right when he says that there needs to be more communication between the practitioner and the research communities. Teachers make bad researchers, and often, researchers make bad teachers, and the assumptions are apparent in many articles of research I read- where assumptions and biases that would have you laughed out of the physics club are common.
One thing’s clear: it’s a mess. I’m not so sure, as Mark Keary and Ben Goldacre implied, that we’re on the cusp of a Golden Age of research bounty in education. They’ve been saying that about the social sciences since they were invented, and microscopes and abacuses, which had been so generous in the natural sciences, were turned to the human sphere. We’re still waiting for the Industrial Revolution in social science.
One thing teachers need to do, exactly as Goldacre said, is to familiarise themselves with the the principles of science; to arm themselves with at least a basic understanding of what it means to say an intervention is true, or probably true, or probably not. God knows we need to, given the deluge of garbage that we’ve endured, justified by silly hat research.
So I wrote a book about it. It’s called Teacher Proof, and it’s out this June. 


  1. Laura McInerney says:

    Did Goldacre use the word Randomised *controlled* trial? Only it doesn't appear in his report from today anywhere (which I had presumed is what he had read out, but perhaps not). The report mentions 'randomised trials' not randomised controlled ones, which has led me to wonder why that is?!

  2. Anonymous says:

    I added “Teacher Proof” my Amazon list two weeks ago, right after an infuriating seminar on Differentiated Instruction (which name-checked multiple intelligences natch). Can't wait to read it!

  3. Anonymous says:

    As if things weren't bad enough, do we really have to use ANYTHING modelled on NHS practice as some kind of shining errrr…. lighthouse?

  4. Anonymous says:

    As a medicinal chemist who is now a teacher, I've read Goldacre's books with interest. I am struggling to see how the methods used to test drugs, which involve biochemistry and measurable outcomes, can be applied in a classroom. Still, I'm open to anything! If it helps the kids and I don't get crushed by the workload, bring it on.
    Isn't it typical that Gove gets a non-teacher to preach to us about this, though?

  5. JoeN says:

    My long standing scepticism and suspicion of gurus of any variety found absolute confirmation in Daniel Kahnemann's “Thinking Fast and Slow” where he details brilliantly why vociferous figures like Ben Goldacre are statistically, and psychologically…literally untrustworthy.

  6. Tom Bennett says:

    He mentioned RcTs repeatedly when he spoke so I don't know why the text didn't contain it.

  7. Tom Bennett says:

    And I can't wait for you to read it. Funnily enough, I wrote it after a lifetime dealing with rap in the education sector, so we're not so different you and I.

  8. Tom Bennett says:

    A lighthouse deters surely? But yes, I don't think the medical model and the educational model map particularly successfully.

  9. Tom Bennett says:

    I don't thik it's a Gove thing. To be fair to BG he did say a few times 'I know I'm not a teacher and I can't tell you how to teach' etc In general people LOVE to tell us how to do this.

  10. Tom Bennett says:

    I'll investigate this marvellous book. Thanks for the tip.

  11. NottsAST says:

    I'm not sure I would agree with your reading of this. The book is mostly about not falling into the cognitive traps our brains are necessarily conditioned into. Dr Goldacre is a physician and scientist, not a guru. I really liked Kahneman's book, but have you considered that maybe he is a 'guru', and that you shoud be sceptical about what he says too?

  12. Gnnnaarrr your making my head hurt….

  13. JoeN says:

    I think if you read what Kahneman has to say about vociferous pundits, in any industry or field where there are measurable outcomes, and the relationship between their headline grabbing sound bytes and those outcomes, you'll agree Ben Goldacre fits the archetype perfectly.

    Kahneman is in contrast a career academic who can point to an immensely impressive track record of peer reviewed, investigative and analytical research in a number related fields, all without the aid of a newspaper…(and I can see I'm treading thin water here Tom!)

  14. Anonymous says:

    Fantastic article, Tom. Got so many Gove quotes, references and bookmarks i fear I will start calling my colleagues Michael or 'Govey' as he sometimes calls himself….can you recommend any papers / policies about learning styles? Busy writing a paper on learning styles…..

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s


%d bloggers like this: