Home » Bad Science
Category Archives: Bad Science
Thinking in the right direction; just don’t put all your faith in RCTs. Ben Goldacre’s vision for evidence based learning.
|But is it outstanding?|
Bethnal Green Academy, a soy latte’s throw from both hipster Columbia Road and the surrounding estates, was the venue for the launch of Ben Goldacres’s new advocacy project: Building Evidence into Education. Nice looking school; it’s got BSF written all over every plane and pane. The livery outside the school shouted every second sentence of the latest Oftsed report. Most pleasingly it was styled in the Star Wars font (or as my kids call it, ‘That old film’).
Also, the problems of cognitive bias, the Hawthorne Effect, the John Henry Effect, the Pygmalion Effect and so on are enormous obstacles to appreciating when an intervention is effective and when it is not.
|‘Guess where this has been?’|
|In other news: thinking burns calories.|
Remember Brain Gym? It was a now-discredited theory that pressing your brain buttons and doing warm ups would somehow improve the cognitive development of your learning conversation, or something similarly moronic. It would be laughable, except that a sizeable purse of public money was spent promoting this ridiculous snake-oil. You know, money that could go to orphans and homeless people and that. I should know, because I was one of the recipients- I was a cultish recruit on the now-defunct Fast Track program (motto: Be the inspiration- from the classroom to the staffroom, which should give you some kind of idea how much we were hated), sort of a predecessor to Teach First. They threw money at us, really chucked it as hard as they could. One of the now-unimaginable training bonuses was a three day residential where we learned NLP (another dubious bag of serpents and spanners) where we were taught the uncertain joys of Brain Gym, and had it recommended for dissemination in the real world.
Fortunately the stake has been fairly firmly planted in the heart of the Brain Gym vampire, especially after Ben Goldacre’s famous assault on it in Bad Science. But not before thousands of schools had wasted their time, and most importantly that of the students, on pointless, pointy-headed miracle crystal exercises that made extraordinary claims to efficacy but without concomitant extraordinary evidence. Any efforts accrued from Brain Gym could be replicated from giving your pupils a break every now and then and getting them to stretch their legs a bit. Which, you know, people do anyway, unless you treat your students like laboratory beagles (and even they get very long fag breaks).
|And THAT’S what it’s all about.|
But the dead do not stay in their graves; they rise, they reek, and roam the edu-sphere, looking for new necks, fresh blood and brains. Fans of feeling sad and slightly intellectually superior weren’t disappointed this week if they read the news that ‘Moving to rhyme may boost pupil results.’ What the research appears to be telling us this week is that doing exercises set to nursery rhymes helps children to develop. I AM NOT MAKING THIS UP I READ IT ON THE BBC WEBSITE THEY DO NOT F*CK AROUND:
The Primary Movement project involves getting nine-year-olds to do set exercises to nursery rhymes and will be tested in 40 schools in north-east England. The exercises mimic the earliest reflexes made by babies and foetuses. The theory is that children can be held back if such reflexes persist. Trisha Saul from the Primary Movement project said: “Some of the songs and the nursery rhymes will be familiar, it’s the movements that are different. “These are designed to replicate movements the foetus makes in the womb and the baby makes in the first six months of their life.”
Now, why on earth would anyone think that this was a thing that actually existed, rather than simply being a whimsical daydream of a theory? Because the ubiquitous research suggested it, in a study by the Queen’s University Belfast in 2000. Blimey, they’ve taken their time to get round to doing anything, haven’t they? Or maybe it’s a testimony to the distance back anyone had to dig in order to get any academic support for this latest foray into the desperate world of educational fairy tale research. Maybe not.
The small scale little known research project found that children who carried out systematic physical exercises for a year gained 15-20 months progress in reading compared to a control group which did not do the exercises.
‘Small scale’. ‘Little Known’. It’s not looking good, is it? They could have said ‘obscure’, but I think the Beeb draw a line somewhere. Trisha Saul, from the Primary Movement Project, said this:
“It’s a bit like a caterpillar turning into a butterfly, but the butterfly still has bits of the caterpillar attached to it.”
That’s EXACTLY what it’s like. What?
|‘I am a confident, independent learner.’|
I had a look a Primary Movement’s website (the body coordinating the project, which is taking place in 40 schools in England and Wales). It’s far from illuminating, although it links to a sole credit- a press release from Queen’s University, Belfast (oddly enough from 2006, and the article published by the no-doubt beyond beyond reproach Journal of Research in Special Educational Needs– available online only, of course- was in the November 2005 issue, so I don’t know where 2000 comes from).
What it DOES offer is a number of courses that you can apply to take. There isn’t a price list on it. I’m guessing it isn’t free, for either the Foundation or Advanced Level certification. The website advises that as a parent, you should check if your local teacher is trained properly in the method, and suddenly it’s getting a bit mystical and Alexander techniquey, and only the elect are chosen etc.
There is an interesting name that comes up again: Dr Martin McPhillips, who developed the Primary Movement program, and also appears as the author of five out of the seven published papers supporting the work of the PMP so boy, he’s busy. As far as I can see, the research produced by the PMP seems to focus on children with SEN. I might be wrong, of course.
‘The Primary Movement programme developed at Queen’s University, Belfast has been shown to have a significant impact on reducing reflex persistence. It has been evaluated in a number of formal studies that have been published in peer-reviewed scientific journals.’
In a school-based study of children in their first year at primary school, it was found that the Primary Movement programme had a significant effect on the development of fine motor control(9). In another large, school-based study, involving more than one thousand children, it was found that the Primary Movement programme had a significant effect on ATNR persistence. This led to improved academic attainments in reading, spelling and mathematics
|‘I endorse brain gym. And anything else.’|
That last sentence interests me, because that’s where programs like this intersect with my work as a teacher. The PMP is, I’m sure, beyond reproach, has impeccable academic credentials, and works solely to promote the well being of children. Its authors and board are undoubtedly motivated by nothing but the noblest of motives. That last sentence is quite a claim. I wonder how they reached that conclusion?
My concern is that it is far from clear that instigating a program, however well endorsed, of physical exercises has anything like a substantial effect on a child’s learning ability, and if it does, can be replicated on anything more than laboratory conditions, on all, or even merely most children. And that it is far from clear that such a program has any significant difference from any other program of simple physical exercises. That the suggested increases in learning can be accounted for solely by reference to the exercise program, and can’t be accounted for by other means, such as the children and the teachers feeling that there should be some kind of benefit. Maybe, maybe, maybe. That’s the problem with this kind of research.
The problem remains with ALL forms of educational research; controls aren’t real controls; exact conditions can’t be replicated and tested against. High causal density in human interactions means that causal relationships can rarely, if ever, be inferred from any pool of data, and researcher bias becomes overwhelming in both the design, execution and interpretation of any such project.
Meanwhile, in austerity UK, trials like this receive funding.
Social science. It’s not a real science, is it?
I believe that children are the future. So; no change, then. Brain Gym, Bad Science, and the generational story of stupid.
Excellent article in today’s Guardian from the excellent Ben Goldacre, who writes (again) about the car-crash of stupid that is Brain Gym, and the fact that, despite its reliance on the kind of science normally found in the average Green Lantern comic, it persists in being adopted by schools that really should know better. There’s nothing more I can add about Brain Gym, other than to say that anyone who still believes it has any utility beyond keeping credulous writers in employment deserves to be placed in a corner wearing a conical hat.
But there’s a point that deserves examination: Goldacre’s optimism that, despite the persistence of- often adult- participants in the kind of moonshine and snake oil that so entrances educational experts, the future holds a possible golden age of intelligence. Why? Because as he puts it:
‘Information is more easily accessible now than ever before, and smart, motivated people can sidestep traditional routes to obtain knowledge and disseminate it’
|Bouchard: You BASTARD.|
Information certainly is a lot more available these days, oh boy it is. I remember a GCSE (or equivalent thereof) homework where we were invited to either fill in a work sheet, or research a biography of Pierre François Bouchard, the engineer who discovered the Rosetta Stone. Being somewhat friendless and socially gauche, I of course chose the research project, intuitively apprehending its innate complexity, challenge, and geek points. The family picture encyclopaedia was sadly not up to the task; a trip to my local library revealed nothing except for the painful lack of diversity in the Scottish public lending establishment. A further venture to the grand Mitchell Library in the city centre was more helpful. I handed the report in with pride, only to discover that I had carefully recited the life history of another Pierre Bouchard who, unhelpfully, was a contemporary in time and space. Some stories don’t have happy endings.
The point is, that with a few stylish finger flourishes, Dame Google can furnish me with more information about the intrepid French tomb raider than I could possibly digest in a month. Instant, endless raw data is now an axiomatic principle of our daily lives. Pub arguments about statements of fact now last only as long as the time it takes to wave an Iphone. Pub Quiz Masters have to emit EMP pulses just to level the playing field; that, or allow everyone an identical phone and speed recite the questions, testing participants on digital dexterity and search-term efficiency instead of scrappy long-term memory and retention of trivia.
The homework task I described would now perfectly exemplify the IT revolution we have experienced, and surely guarantees Mr Goldacre’s claim that even children- or perhaps especially so- will be the inheritors of a new age of intellectual rigour and penetration; that armed with the facts, reason and empirical scepticism, New Age rhetoric, Religious cant, and superstition will evaporate like fog in a November night, replaced by precision, degrees of certainty and cool, unexcitable reason. In fact, children should surely be the drivers of this intellectual revolution, given the alleged proficiency and taste they show for digital manipulation, and the familiarity they evidence with the online adventure.
|Brilliant! You look like a moron!|
But not so, not so; if only it were. This reporter is at Ground Zero on Planet Earth, bringing you the latest, and from where I’m standing it isn’t pretty. It pains me to say that, while many children do conform to this description as brave and rigorous denizens of an information superspace, many do not. In fact, you could even say that many of them are just as naive as their slightly older counterparts. They hold beliefs, prejudices and fancies that are just as fanciful, unsubstantiated and unsupported as anything that their generational predecessors could conceive.
For example: t a recent conference for students at a school I was visiting, I showed a group of approximately 100 sixth formers a picture of the 1969 Moon Landing, above which I had written ‘Are we being fooled?’ Now bearing in mind that I had already announced to the audience that this session would be about conspiracy theories, and how most of them were totally unfounded, you might expect a high level of scepticism. Not a bit of it. When I asked the audience to put their hands up if they thought that Neil Armstrong and his fellow possessors of the Right Stuff had really made it to Earth’s little sister, only around a third of the group did so. OK, I reasoned; participant anxiety. So I rephrased the question in the negative: who thought they hadn‘t really gone, and that the Moon Landings were faked?
Around two-thirds of the group launched their own Apollos into the air, give or take. In my mind, I could hear the Ancient Gods of mysticism laughing at the pretender Reason. When I asked them why they believed such an apparently transparent piece of guano, the usual answers came out: the shadows from the lander; the motion of the flag, and so on and so on. All the usual, cranky suspects. I can’t be bothered to retread the well worn path of dispelling this crap, but the answers are out there- you know, on the Internet. I spent around half an hour blowing each conspiracy to pieces, showing them sources, experts, references.
By the end of the session, I asked them to repeat the show of hands. With a trembling lip I saw that the ratio was now an even 50:50. A victory for rationality? I didn’t think so. Even when I questioned them later, the nays still reverted to their default prejudices and positions: they ‘didn’t trust’ the official story; there was ‘something in’ the conspiracies. I was confused. Had I not explained the perfectly simple refutations of these quack ideas? No; they certainly understood.
I realised what was happening. The conspiracy version of events suited their emotional and cultural appetites; the idea that there is ‘something going on’ behind the scenes of history; that authority narratives are unreliable and manipulative; that doubting the official record enables the powerless to perceive themselves as something powerful; prophets of the truth, discerners of deception, and holders of wisdom. There was, it seemed, at least two worlds: the world as it seemed, and the world as we would prefer to imagine it….even if that world itself contains unpalatable truths, such as the idea that the leaders of every nation state would conspire to conceal the truth of the Moon Landings, or that billions of tax dollars were therefore being diverted to feed…what? The military-industrial complex?
|‘This looks well proof, innit?’|
It doesn’t just end at the Moon Landings; it couldn’t. Such an attitude has, in my experience, been the foundation of my experiences of working with this Net Generation, these citizens of cyberspace. I teach RS and philosophy, so I understandably attract the conversations of the mystical and counter-rational. That’s fine, and for the best; the oceans of the mind are not charted solely by reason and measurement. Other compasses are necessary to understand the black box of the human psyche. So I know first hand the way that children and teenagers often wrestle with issues of meaning, value and experience, empirical or otherwise. And I also know how many of them turn to ideas and belief systems that are not merely counter-experiential, but counter-rational.
Like the scores- and I mean scores- of serious young men who sidle up to me after lessons and ask me if I have heard of the Protocols of the Elders of Zion, and would I be able to discuss it with them? (The Protocols, for the sainted uninitiated is a fraudulent crypto-history/ manifesto, allegedly written by the Jewish Puppet Masters, and ‘accidentally’ reveals their global ambitions. It is as vile a forgery as you could hope to read, and it has existed for over one hundred years. Despite its utter annihilation as a fake many decades ago, it still enjoys a furtive and political life wherever anti-Semitism needs a prop. Ghastly). There are students who have enquired if I know anything about the New World Order; about the assassination of Diana by the Royals, of JFK by the CIA, and more recently, the enormous number of students who apparently believe that 9/11 was the work of a diabolic Israeli black-op. All of these, theories as empty and aimless as a barrel going over Niagara Falls, yet somehow, inexplicably resuscitated in the minds of the credulous child.
The anti-empiricism doesn’t stop there. I teach a unit on ‘War and Conflict’ to year 9 pupils; I thought it apposite to teach about the Iraq war which, to me at last, still feels relatively recent. Every class I have taught for the last few years has, for the most part, believed that the war was started because of a) Al Qaeda and b) 9/11. Most of my students believed that Hussein and Bin Laden were in some kind of Axis of Evil, and that Desert Storm was the direct response to the fall of the Two Towers.
Now I’d like to point out that these are the same children who are practically hard wired into fibre optic cables, so ubiquitous is their online experience. They could barely be more connected were they to have a wireless connection node implanted in their cerebella. But still, they seem as susceptible to the storm fronts of illusion and nonsense as any of their 8-bit forebears; just as swayed by fantasy and information tricks as anyone else. Why is that?
|‘…and if you look closely, you can see a pixie.’|
The answer lies in this basic truism: being drenched in facts doesn’t make you any smarter; it doesn’t provide the possessor of an iota more discernment or critical faculty. This is the difference between facts and knowledge. Many educational half-wits have hailed the Google generation as having been liberated from the tyranny of knowledge; that in an age of universal IT suffrage, any fact can be conjured from the ether like a magician’s dove, and we should therefore focus on teaching children skills, not facts. If everyone who thought that was reduced to a single person, I’d spanner them. Our data-rich descendants are fallible after all; the cyber natives are just as liable to err as anyone else.
And why on earth wouldn’t they be? They are human, or so I believe, although from a cursory reading of the over-excited froth that is regularly written about them by progressive educationalists, you could be forgiven for thinking they had developed an X-gene that made them the perfected inheritors of a Golden, Utopian tomorrow. I have some news: they’re just like us. They ARE us. If a student wants to find out if the world really is controlled by a race of Reptilian overlords, or a more mundane segment of the Primate clan, it isn’t as simple as just clicking on a link. Because more often that not, the link will take you a million miles away from anything resembling ‘life experienced’, and into ‘life imagined’, realms of fairy tale so fabulous that they resemble a fever-dream. Try it. Search Google for Who killed Diana? and see what kind of balance you find. May God have mercy on our souls.
Searching for the truth involves more than just asking a question: it involves asking the right questions. Not all questions will lead you to the real answers, or as close to the real answer as our epistemological faculties will permit. Asking ‘When will Satan end the world?’ won’t get you an inch closer to the veracity of that claim’s assumptions, not on Bing anyway.
The ability to tease apart speculation from cautious theory is a skill that has enjoyed varying respectability as the centuries have progressed. Some reliance on the evidence of our senses has always been with us, otherwise we wouldn’t have made it this far. David Hume famously exploded the irrationality, the ‘sophistry and illusion’ as he described it, of believing in anything that could not be experienced or rationally inferred. The growth of the scientific method has transformed our world, physically at least. But how much of it has penetrated the methodology of our everyday analysis?
I, too have attended training courses on Brain Gym (sponsored and paid for, I hasten to add, by the Fast Track, a by-now deceased arm of the DfE’s previous attempt to invite the brightest and best into education. Alas, they got me) and believe me, the first time around, it sounds perfectly plausible. Who’s got the time to go away and dig up the data that reveals it to be a load of shit? Besides, the DfE were paying for it, so surely they must have checked it out, right?
We’re not swimming in information any more. We’re drowning. Children and adults alike are drenched in a deluge of data that takes time, effort and perspicacity to distinguish the dreck from the good stuff. There will always be people patient and stubborn enough to want to get to the bottom of things, and with the advent of the Internet, the resources to do so are now available to more people- children included- to do so. The problem is, not many of them necessarily will bother. We’re lazy; we’re busy; we’re just too damned indifferent to the truth.
And it’s one of the reasons that an enormous intellectual cavern exists in educational theory, and why the charlatans, the hucksters and the spivs have moved in to fill it with empty theory and card tricks. the battle, I might point out, goes on…
I’m off now to pay tribute to my Scaly overlords, and see if the Internet can tell me how to build a tinfoil Fascinator to stop the CIA controlling my brain. If I start blogging about Personalised Learning Styles, you’ll know they got me, in which case put a diamond bullet through my heart and head for the Vatican. End of Transmission.
|‘Leave me! Save yourself!’|
An article on today’s BBC Education portal highlights the no doubt vertiginous decline in the average school child’s ability to lose gracefully or, indeed, to win gracefully. I wonder how they came up with a sufficiently large data set to arrive at this conclusion, given that school sports days appear to have gone the way of the Daily Sport and vanished from most school’s year planner, on the basis that the very rumour of competition will have less physically able children curled up in a foetal ball of misery and self-loathing akin to a diabetic fit.
But given that they actually found enough children who were allowed to actually compete with one another, the findings seem, on the surface, to be a cause for concern.
‘Two-thirds of parents of eight to 16-year-olds said their children reacted badly when they lost, the poll found.
A further two-thirds of respondents said parents behaved badly when watching children’s matches.
Some 1,008 parents and 1,007 children aged eight to 16 were questioned for the survey by Opinion Matters.’
How awful. And, I’m sure, quite true, quite true. Still, it always gets my whiskers twitching when I see survey findings being posted slavishly as headlines, despite the cowardy-custard get-out of putting it in between two lazy apostrophes to indicate a possible lack of veracity. Try ”David Cameron wears tights,’ an unnamed source claimed last night.’ The writer gets the neon headline, the reader gets mauled by a grisly, unforgettable image, and all is well, except for the reader’s orientation to any actual truth claims. So, how firm is this piece of educational news?
|‘After you.’ ‘No, after you.’|
Well, it’s been conducted by Opinion Matters. Who they? Well, if you’re Marks and Spencers, or Vitabiotics (‘where nature meets science!’) you’ll know all about them. They’re an ‘independent’ market research company (what does independent even mean in this kind of context? It seems to mean, ‘will work for money’) who, as their website proudly says, ‘make it our priority to aid our clients to generate headlines and create coverage that highlights and reinforces their branding and key messages in media.’ In other words, they conduct research that will get their clients into the press. You’ll have seen this kind of thing many, many times before- have a look at the Vitabiotic link above, where they advise the client that a survey or two- a bit of science- will get lazy journalists writing about the product, even though the survey in question relies on ‘perceptions’ and subjective opinions. So they’re used to dealing with sensitive subjects with impartiality and rigour. So far, so good. Have a look at the website. How do they sleep? (As Don Draper would say: ‘On a bed of money.’)
And who commissioned the research from these paragons of objective investigation and veracity? Surely not the Marylebone Cricket Club and the Cricket Foundation? It certainly was. One might reasonably ask why such caryatid columns of sportsmanship and integrity are interested in finding out how badly Britain’s youth react to winning and losing. Perhaps they’re just curious.
Or perhaps not. As we discover buried away in the legendary bowels of the article (which you get a chocolate biscuit if you’re still reading by that point) is the golden nugget that these august bodies have recently started ‘offering sportsmanship lessons to state schools’. So, no financial or commercial interest in this research at all. Still, at least the researchers were independent. Somehow.
|The data says ______.|
So there we have it: a story based on research commissioned by bodies who have a vested interest in the findings, conducted by people who are delighted to make sure that the findings are found, and published uncritically as fact by a public body funded by the license fee. The sad thing is that I probably have a lot of love for the Cricket Foundation. (I mean, it’s the Cricket Foundation, for God’s sake.) And the idea that children should be more sporting. And the idea that engaging kids with good role models in sport might be a good idea. It’s just as charmless as Liberace to see it all stirred into the same cauldron, sold as fact, and served up as news. I think we all deserve a bit better than that.
I enjoyed Panorama tonight; I always do. There’s something so intuitively respectable about the BBC’s venerable investigative magazine that I would default to unqualified admiration even if it were to tell me that spaghetti grew on trees. This week: Troops To Teachers (TTT)- Michael Gove’s drive to inject a bit of military discipline back into classrooms by aggressively recruiting and retraining ex-military servicemen. It apes the Troops to Teachers program in the US, launched 18 years ago after the first Gulf War, and since then it’s seen over 15,000 men and women swap green berets for cardigans with leather patches (or whatever the symbolic equivalent is in America).
If you watched the program you would be forgiven for assuming the the program is an unqualified success; we were treated to the example of Lordswood Boys’ School in England, which entertains no less than 1 in 12 staff from military backgrounds, which shouldn’t really be a surprise seeing as how the smallish Birmingham comprehensive has an assistant head who used to be in the Infantry, an ex-Sergeant from the Territorials, and a former sergeant major acting as a shooting instructor. Quite. Still, variety is the spice of life, and one thing that schools have to be praised for is diversity of strategies, trying different things, and adapting tactics to meet the needs of the local community. Looking at the prospectus and the Ofsted report, it seems a bit of a success story. Students like Hakeem Nawas spoke proudly of how it had transformed his self-esteem and motivation to be involved in Cadet activities, and Neil Macintosh, the aforementioned Assistant Head proposed that ex-military were ‘more resilient…less down-hearted…and more robust.’ As Mandy Rice Davies, said, ‘Well, he would, wouldn’t he.’
Actually I have no issue with this: in fact I admire many of the principles that inspire it. I particularly liked how the servicemen spoke about how they maintained order- they didn’t have to raise their voices, they said. The students agreed. ‘They just look at you,’ one said. I know what he means. Screaming your head off is usually a sign that you’ve blown your stack, and for most kids it’s better than telly. Speak silently, they say, and carry a big stick. I couldn’t agree more. Who do you respect more- the small dog with the big bark, or the silent dog with the claw hammer behind his back? Exactly.
|Troops to Hogwarts|
Then we were off to Huntingdon Middle School, in Virginia’s Newport News City (honestly- I wouldn’t make up a name like that because you wouldn’t believe it), where a clutch of ex-military had taken over their classes like Desert Storm. The story here was the same, it seemed- lines of biddable, disciplined and enthusiastic students queued up to enter classrooms, and we were presented with crocodiles of marching students who were noticeably not selling crack pipes to grandmothers or auditioning for The Wire. Glee, maybe.
Geoff Lloyd, the poster boy for this school’s TTT project spoke proudly about bringing ‘discipline into an undisciplined world,’ and frankly, I couldn’t agree more. His robust, direct attitude to being a responsible adult in a classroom full of students who need clear boundaries and someone they can rely on was more inspirational than a dozen Dead Poets’ Societies or Dangerous Minds. I would put him on my fictional Heroes of Education list, but unfortunately he’s a real person, so he’ll have to content himself with a notional award instead.
So far, so good. As I say, I actually applaud many of the aims of this program. I think that what many of the ex-servicemen said made perfect sense- courage, responsibility, discipline and carrying your own water. Amen to that, brother.
And then- with the inevitability of the Sun rising- came the research. Because that’s what we do whenever we want to justify something: we wheel out the academics who biddably endorse whatever is being flogged to us. And that’s when it got interesting for me. William Owings of the Old Dominion University sat in an agreeable, respectable setting and enthusiastically waved the flag for the TTT program, his eyes twinkling as he did so. He twinkled a lot. ‘Ex-military stay in the profession twice as long as non-servicemen,’ we were told. ‘Troops in the T2T program outscore all other teachers,’ it was said. Owings also provided my favourite quote of the show- T2T had provided a ‘stellar performance,’ he said. Twinkle, twinkle.
Now that didn’t strike me as the careful, cool, neutral perspective of the scientist, I thought. And as soon as someone starts to mention educational research, my spider sense starts to tingle, and frankly I start to sweat a bit. Because, as regulars to this blog will be painfully aware, I’m allergic to the way that some educational research is used to hustle strategies and big ideas that are composed, it is eventually seen, of equal parts moonshine and optimism. As a teacher of some years, I’ve been making a list of the Initiatives and Great Ideas that the hucksters of education try to flog us, and my hackles start to mambo whenever someone calls along and says, ‘Hey, you guys! I have a great new idea for turning schools around! I just need your credit card number and your uncritical commitment…’ I’m just funny like that.
So I did a bit of, rooting around on t’interweb. Just what IS the Old Dominion University, anyway? It sounds awfully grand. And it is, I am sure, a paragon of academic vigour, rigour and propriety, even if its mission statement does say that ‘Our philosophy is simple: Knowledge should be productive. Research-driven solutions that make sound business sense.’ Which isn’t really a philosophy, is it? More of an admission that if something is worth something, it has to be worth money. Ah, it brings a tear to your eye, doesn’t it? .
As I say, I’m sure it has the noblest intentions. It also has an interesting link to the Troops to Teachers program, as its website says: ‘The state office for Virginia TTT is located on the ODU campus.’ That’s the state office. Of course, that doesn’t suggest that the Old Dominion University might be a less than partial witness to the efficiency of the TTT program. I’m just saying, that’s all. Isn’t that a marvellous coincidence, though?
So I did what few civilians have done before: I had a peek at a couple of the papers quoted on their website as showing terrific, supportive data that confirmed the TTT program as a winner, and the ones that William Owings was quoting so freely on Panorama. You can find two of them here and here. They are, as most social science papers are, a thrill a minute, and I heartily recommend you print them off and read them on the way to work tomorrow. Unless you drive. Or listen to Coldplay while you read them.
The 2005 study was, broadly speaking, a survey of Teachers who had gone through the program, and of their supervisors. It asked if they felt that they had been appropriately trained to approved standards. It also asked supervisors of these teachers if they felt they were as good as, or better than teachers of similar experience who hadn’t come through the program. The answer was strongly in favour in both cases. How many were surveyed? A fair few. Over 2000 teachers and their supervisors were sent surveys. That’s not a bad study by any standards. Except that the response rate was 65%. We don’t know why the other third didn’t reply. We don’t know what attempts were made to convert those no-shows. We don’t know on what basis the surveys were sent. We can probably assume that surveys weren’t sent, or at least answered, by teachers who had dropped out of the program.
And it’s details like that, that make this kind of research so hard to value meaningfully. Big numbers are good, but without transparency about who answered, what their motives were, and show inaccuracies were avoided, the purity and reliability of this kind of data is always going to be hard to measure, let alone accept. I’m certainly not impugning William Owings, or any of his co-writers, but these are substantial, significant impediments to the development of social scientific research credibility.
Another problem is that this paper relies on perceptions- ‘how well do you feel …’ questions. These questions fall short, IMO of the clinical precision and neutrality of the genuinely inquisitive, and stray into the territory of market research. When did you stop beating your wife? Who’s to say that the TTT candidates were actually trained properly? What’s to prevent the supervisors betraying their own inclinations, preferences and prejudices through their own opinions. Nothing. Nothing at all. This isn’t the same as measuring the temperature at which mercury boils- it’s like interviewing a series of marathon runners at the finishing line and asking if they feel out of breath.
The paper does acknowledge some of this. Actually, it seems to acknowledge all of this:
‘the study does not provide evidence of T3’s self-reported or actual teaching behaviours. Neither does it provide empirical observations of school administrators watching T3s’ actual teaching behaviours. Nor does it provide evidence of students’ learning gains as a result of working for a period of defined time with T3s as compared with other teachers of similar experience. Further study of the actual teaching practices from T3 self-report or assessment of their students’ measured achievement, although very complex and difficult studies to undertake, would provide important information about T3s’ quality as well as feedback about how to strengthen T3 preparation.’
In other words, we know it’s all just opinion and self analysis. But we don’t think it’s a problem. Of course, opinion and subjective experience have a place in analysis; but it’s not the same place as objective, viewer-independent data. It doesn’t prove anything more than the people who responded felt the way they felt. It’s not corroboration that these teachers are better: it is what it is.
The other paper I looked at, from 2010 (and also by our hero from Panorama), focused on TTT candidates who went on to become Principals. This time it was 107 subjects; ah, boo, much smaller. Their supervisors (I didn’t even know Heads had supervisors) overwhelmingly (90% plus) said that they thought such principals were better on a variety of scales than similar, non TTT Principals. Yes, you may also find it unsurprising that supervisors, who I assume are involved in the selection and support of these principals, overwhelmingly thought that they were doing a jolly good job, and hadn’t they made excellent decisions hiring them? Again, we don’t know the conversion rate, the response rate etc.. I’m sure it was fabulous, given that 107 is a very small number. Still, the data comes out rather well, doesn’t it?
So is there nothing concrete at all to support the view that TTT candidates have a, if you will, tactical advantage over their civilian counterparts? Not a bit of it. Here it is:
‘In a 2008 Florida study comparing measured academic achievement of elementary,
middle, and high school students taught by TTTs, results indicate that compared to all
teachers, students served by Troops teachers performed about equally well in Reading and
achieved a small but statistically significant advantage in Mathematics. In comparisons
where each Troop teacher was individually matched to another teacher, teaching the same
subject in the same school, with approximately the same amount of teaching experience,
students served by Troops teachers achieved substantially and statistically significantly
higher in both Reading and Mathematics (Nunnery, et.al, 2008; Nunnery, et. al., 2009).’
Call me a gutless limey cynic, but ‘equally well in Reading’ and ‘a small but statistically significant advantage’ in Maths doesn’t exactly strike me as cause to start popping the champagne for the cause yet. Incidentally, the Nunnery paper mentioned above by Owings is co-written by…..yes, William Owings. And it wasn’t published in an academic journal, but, as the report says, ‘submitted to ‘Educational Administration Quarterly
October 2008′. I can submit a poem written on bog paper to the Sunday Times. Does that mean I can say it was printed? Have a look at the front page. It’s got a lovely ‘Troops to Teachers’ logo all over the front. I’m don’t have a Ph.D. in this exact subject, but I suspect that means they might have something to do with the report….
(I stopped reading it at that point, because I don’t believe in an afterlife, and I value every precious minute I possess.)
In fact, so do the previous two papers I mentioned, both of which are prefaced by the sentence, ‘A Report Prepared for Mike Melo, Director, Virginia Office of Troops to Teachers,’ and ‘Report to Dr. William McAleer, Executive Director, Troops to Teachers, Pensacola, Florida.’ So all of the reports mentioned were written for (can I presume commissioned?) the TTT itself. Hey, waitaminute…..
It’s not that I’m against the idea of ex-military training for schools: good luck to ’em, I say. And I think that there might be something in the idea that men and women who have experience with leadership, developing self-discipline and oiling rifles might have something useful to teach children (sniping, for instance). But it doesn’t do anyone any good to use research like this that seeks to support proposals with empirical claims that can at the very least be contested as meaningful or verifiable in any real sense. Michael Gove needs to look elsewhere for better arguments, and maybe we might start to take research based policy more seriously.