Tom Bennett

Home » educational research

Category Archives: educational research

Thinking in the right direction; just don’t put all your faith in RCTs. Ben Goldacre’s vision for evidence based learning.

But is it outstanding?

Bethnal Green Academy, a soy latte’s throw from both hipster Columbia Road and the surrounding estates, was the venue for the launch of Ben Goldacres’s new advocacy project: Building Evidence into Education. Nice looking school; it’s got BSF written all over every plane and pane. The livery outside the school shouted every second sentence of the latest Oftsed report. Most pleasingly it was styled in the Star Wars font (or as my kids call it, ‘That old film’). 

Hosted by Teach First, the Royal Marines of the teacher profession, we were first treated to an introduction by Dame Gove himself in what was, I thought, a remarkably short set. It was like booking Geoff Capes and asking him to open a jam jar. Goldacre followed;  a passionate and determined thinker and speaker, whose Bad Science series shook me, like Hume did for Kant, from my dogmatic slumber.
It’s a familiar saw for him: the need for Randomised Controlled Trials (RCTs) to become more prevalent in the social sphere (here represented by education), as they are in the medical profession. He made an interesting comparison: medical practitioners at first resisted RCTs because they were convinced of their own intuition, until it turned out that intuition often led us down the wrong path; eg using steroids for brain inflammation did more harm than help- the reverse of orthodoxy. Now, so are teachers (resisting RCTs, not harming the brain. Oh wait.)

Goldacre’s point is refreshing and disarming: we run trials like this for every pill and potion; why not policies, which in the galactic scale of things, dwarf even pharma for dollar tonnage. And as usual, there is a lot of practical wisdom in what he says. I am delighted that the DfE is looking into ways of conducting research that is robust and real, rather than fanciful and driven by dogma.
So why not have RCTs in the social sciences? One possible objection, the ethical, can be easily dismissed: issues of consent, and issues of knowingly withholding a potentially useful intervention from the control group. But we already introduce dozens of interventions in schools every single day. It isn’t as if we aren’t already drowning our kids in optimistic hoodoo. Why NOT randomise it once in a while? When was the last time we asked a kid’s consent before we popped a Thinking Hat on them, or told them to rub their brain buttons? 
I think there are, however, some serious problems with the use of RCTs in education.
 RCTs aren’t the answer to the question ‘What really works in schools?’ because differences in context can never satisfactorily be ironed out. The causal density of humanity is too high; there are too many factors to establish a reliable protocol that could hope to encompass the variables of the human mind. There are as many factors to juggle as there are grains of sand on the beach. Lazy research is sodden in bias and assumptions, wooly over interpretation and optimism. That said, there is room for quantitative research , some of which will be amenable to RCTs. I’ve heard social scientists say that the problem is merely one of design; I say the problem is the methodology itself. If you’re making quantifiable predictions about the physical world you need to provide unambiguous methods of establishing initial conditions as well as outcomes, possible causal mechanisms, and demonstrate reproducability. That isn’t easy.  

Also, the problems of cognitive bias, the Hawthorne Effect, the John Henry Effect, the Pygmalion Effect and so on are enormous obstacles to appreciating when an intervention is effective and when it is not.

This is definitely thinking in the right direction. Goldacres’s attempt to lasso the spoon bending of social science and pin it down to some kind of methodological rigour, is exactly what I, and many teachers, want to hear, and I absolutely support his attempts to reinvigorate the debate. But there are many obstacles to this, even before considering motivation.
One is time. I work pretty hard. I don’t expect a biscuit, because I bet you do too. I’m also pretty good with my time. And I barely have time to do, well…this kind of thing, and the only reason I do is because I cash in hours of sleep like chips in a casino. Doing research into education is going to be a niche pursuit until workloads go down. Like archaeology in the 19th century, it will remain the hobby of gentlemen.
And let’s be honest, most teachers aren’t scientists. They’re as prone to their pet prejudices and tender perversities as most people. Read the Tooley report if you want to see how badly education research can be conducted even by people who should know better. Now multiply that by 100 and you get, I would say, most of us. Many teachers would love to get involved with what Goldacre sensibly described as driving research.But would research benefit? Not until teachers learn what the scientific method is.

Dating for Nerds
So, what next? From the presentations, and the discussion one thing became apparent: this isn’t the launch of anything concrete yet. At this point, the program is advocacy. There was no funding committed, no projects starting tomorrow. From one perspective, it’s hard to see many education secretaries getting too worked up about RCTs- the average tenure of the Headmaster-in-chief is about 18 months. RCTs can take years; to be efficient, to be meaningful, they have to be as large and long as possible (as the inspector said to the janitor). Why invest in something that a) won’t bear fruit until after you’ve been moved to The Department of Silly Walks and b) Might disagree with your own pet projects? It’s always safer to simply pick research that appears to validate your own objectives.
And yet, and yet. Gove was, at least, there. The Dfe’s involvement with Goldacre shows, at least, a symbolic commitment to better research in education. And let’s not forget the £125 million DfE funding for the Education Endowment Fund (represented here by Dr Kevan Collins) a Sutton Trust start-up devoted to research, some of which involves RCTs. There might be some will there, but it isn’t very full-throated. 
So much in education is an abstract, an artefact of art as much as engineering. We can barely agree as to what educated means. Or learning. Or thinking. Or engagement. Or creativity. These aren’t amenable to metrification.How do we study what we cannot catch?
But to sound a more positive note: anything which seeks to firm up the wiliest of educational research is an asset.  Goldacres’s profile should help turn a search light upon the relationship between research community and Chalkface warriors. He’s absolutely right when he says that there needs to be more communication between the practitioner and the research communities. Teachers make bad researchers, and often, researchers make bad teachers, and the assumptions are apparent in many articles of research I read- where assumptions and biases that would have you laughed out of the physics club are common.
One thing’s clear: it’s a mess. I’m not so sure, as Mark Keary and Ben Goldacre implied, that we’re on the cusp of a Golden Age of research bounty in education. They’ve been saying that about the social sciences since they were invented, and microscopes and abacuses, which had been so generous in the natural sciences, were turned to the human sphere. We’re still waiting for the Industrial Revolution in social science.
One thing teachers need to do, exactly as Goldacre said, is to familiarise themselves with the the principles of science; to arm themselves with at least a basic understanding of what it means to say an intervention is true, or probably true, or probably not. God knows we need to, given the deluge of garbage that we’ve endured, justified by silly hat research.
So I wrote a book about it. It’s called Teacher Proof, and it’s out this June. 

Let’s say something so stupid we have to call it science: obnoxious, loud children ‘learn better’

My learning style involves everyone else not learning.

NEWSFLASH: A study of 12,000 baby Tyrannosaurus Rexes found that the larger, more aggressive calves with longer, sharper teeth tended to outperform their more civil, amenable peers in standardised herbivore intimidation tests (SHIT). This contradicts the commonly held view that agreeable, polite mega-carnivores would be more successful. This raises questions about the need for T-Rex mentors to consider other methods…..(continues in a similar vein forever and ever until everyone dies crying)

Comedy Gold dropped into my email box this morning; a timebomb of stupid that I normally associate with people ‘standing on street corners, selling coloured pencils from a tin cup,’ to quote the Hitch. Only this time it was a paper written by m’learned educational scientists at Durham University, and blazed on the BBC news website here. The claim was that children who shout out in classes do better than their kinder, lovelier peers.

If Charles Darwin read this report he’d be chewing his long beard like Gandalf and wondering what all the fuss was about. ‘Competitive, unpleasant organisms unfettered by social mores or self-restraint get more of what they want’ isn’t exactly news. Nor is ‘teachers forced by obnoxious organic alarm clocks to focus attention on hitting the snooze button until Hell freezes over.’

‘The lead author of the research, Peter Tymms, head of Durham University’s school of education, said that among children with ADHD symptoms, those who got excited and shouted out seemed to be more “cognitively engaged and as a result learn more”.
“Perhaps those children also benefit from receiving additional feedback and attention from their teacher,” suggested Prof Tymms.’

TEACH ME! TEACH ME!

YOU THINK? Noisy, greedy little attention-hoovers grab attention, do they? Well, KNOCK ME DOWN WITH A FEATHER. Who knew? Maybe they should ALL be like that, eh? Those quiet little f*ckers, keeping their heads down and working hard, what do they know, eh? Despicable little arse-limpets. They should step the HELL up and get some attention for themselves. Except….except if they all did that….then there wouldn’t BE a survival advantage any more because the teacher would be split like a kaleidoscope into a million shards of fragmented attention. So for anyone to grab more attention than their peers they would have to….SHOUT LOUDER AND LOUDER UNTIL THERE WAS NOTHING LEFT IN THE COLD BROKEN UNIVERSE BUT THE BAYING SCREAMS OF CHILDREN HOWLING UNTIL THEIR THROATS HAD TURNED TO STONE.

Let’s not do that.

‘Ms Merrell, CEM’s director of research and development, said she wanted to carry out further studies to see how pupils could be encouraged to shout out as part of the lesson.’

OH BOY OH BOY, I bet the teachers will be QUEUEING ROUND THE BLOCK for Ms Merrell to start doing her experiments in their class. Perhaps after that we can test the effect of letting kids fling faeces at the teacher like Barbary Apes. Form a line, comrades.

“We need to look more closely at this behaviour and how the interaction can be managed in the classroom.” 

The data suggests that this is optimal.

No, chum; we don’t. I, and every teacher, know how this interaction can be managed in the classroom. You don’t allow it. If we allow the loud mouths and the intemperate to dominate the time and attention of the teacher, to guzzle the oxygen of attention, to monopolise discussion and personal resources, then all we do is create a war of all against all, where the loudest, neediest and least agreeable come first. Which is fine if we are breeding the next generation of politicians, transvestite cabaret artists or X-Factor participants. Less so if we seek to sculpt human beings.

Other aspects of this paper:

1. The title: ‘ADHD and academic attainment: Is there an advantage in impulsivity?’ Are they kidding me on? Are they really asking that question? Normally Educational Social Science needs to be pretty brave to show its face round my manor, but this takes the whole packet of biscuits. Are they having me on? Ask anyone with an ounce of language and experience and they would tell you the answer to the question: yes, of course there is, sometimes. But it still makes you a dick. I look forward to the follow up: ‘stealing things- does it mean you get more stuff?’

For F*CK’S sake.

2. ‘However, it is also recognised that these behavioural problems can lead to children being excluded from pre-school and school from an early age, and that early interventions are promising.’ Or to put it another way, kids who shout out and misbehave get in trouble a lot, and if you work hard with them to control themselves at an early age, they can improve. Which we already know. How do we know? We work with kids. Isn’t that grand?

School Uniform? More research is needed.

3. In the ‘alternative hypotheses’ section of the paper, the writers scratch their undoubtedly pointy chins, and ponder their own fallibility:

‘Could an unmeasured variable, such as the intelligence, confidence or knowledge of some young children be acting as the key causal factor? This is a possibility; only intervention studies could firmly establish a direct causal link.’

Say it isn’t so! You think…you think it might be something else? You think maybe it wasn’t just the shouting out that gave them an educational advantage? I love the cautious, wide-eyed uncertainty allowed to creep in at the end, as if they were scared to even ask the question: is it possible that human behaviour is so complex, that the causal density for potential explanations is almost infinite, and the possibility of exacting a clear predictive mechanism is next-to impossible? Let me answer that one for you: yes, it is possible.

It’s possible that you wrote this paper before you wrote it. It’s possible that you found exactly what you were looking for. It’s possible that the first ten cars I see tomorrow will be red: does that mean that Friday generates a glut of scarlet Hyundais? If, every time I wake up on a Sunday morning I have a sore head, a wet mattress and a pocket full of sticky pennies, can I infer that Sunday mornings cause incontinence, cephalalgia and amnesia, or should I look to the pile of empty Laphroaig bottles surrounding my bed? 

4. Most importantly, this paper actually only makes the claim that pushiness is a competitive learning advantage in the group that already displays inattentiveness; in other words, being quiet and well behaved still outweighs all the other factors in terms of educational achievement correlation, as you might expect. It’s just that the shouty kids who are inattentive do better than the quiet kids who are inattentive. Now that’s not so much of a claim,but in my opinion, the paper frames these themes in such as way as to present them in a more newsworthy and provocative, bone-headed light. The news article certainly does.

T-Rex: surprisingly good Value-Added.

Social science, I despair of you. You’re not really a science are you? Like an ape pretending to be a prince, you can wear the robes and hold a sceptre, but you’ll never pass the laws. You can hide behind as many abstracts, tables and T-scores as you like, but you aren’t physics. You aren’t chemistry. You aren’t even biology. Until educational research remembers what it can do- act as a commentary on human experience, suggest and frame that experience- and what it can’t- discern causation in anything  more than the most rudimentary sense when it comes to human desires and direction- then I’ll file papers like these squarely under ‘that’s nice, did you have fun writing it?’ and turn to the classroom for the laboratory, the research, and the Petri dish of my teaching.

But more than that, I despair of the way that papers like this are presented in the national media. Normally sober and I’m sure, reliable journalists put things like this under the magnifying glass, cranking up the claims and turning a fairly standard investigation into data, into something beyond even the claims of the writers. The paper itself is no worse than many others, and was, I’m sure the result of many hours of dedicated enquiry and analyses. It isn’t the author’s fault that the factual conclusions of the research are so unremarkable, although it is their fault that the suggested implications are so knuckle-headed. But the BBC online news portal seems to be increasingly hyperbolic, as if no Thursday could be complete without a claim so absurd that the Krankies would have blushed to imagine it. 


And of course, the real heart ache is that people will pick up on research like this, and attempt to shoe horn it somehow into the classroom, advising impressionable teachers to let their kids shout out because of some imagined benefit it brings. What kind of people? Normally people who have never taught in a classroom, never will, and yet have the kind of stones to tell people like you and me how children best learn. Get back in your pit, weasels. We have a job to do.


Teacher Voice.


Thanks to @Bio_Joe for passing on the paper to me, and highlighting the alternative hypothesese.
Read the paper yourself here. It’s a barrel of laughs. At least it’s short.