On education "research": a rant.

I've recently been doing my duty as an external examiner on a Master's programme, and it struck me that most of the stuff which the tutors criticised for being too sweeping or unsupported by peer-reviewed, published evidence, was the interesting stuff.

So I'm cutting to the chase on this issue, at least. Much of what follows falls short of the academic bar, but probably exceeds that of mainstream journalism...

Much (probably most--but how do you tell?) of what passes for educational research is crap. There are several reasons for this:

• No-one wanted to do it in the first place. Students, from final-year undergraduates, through master's and doctoral post-grad study are obliged to do dissertations, and so they have to find something to research. Frankly, the guidance they get is very variable, and often the research proposal (itself nowadays frequently formulated as an assignment for a Research Methods module) is poorly formulated. Usually it is far too big, often it includes value-judgements (are students with English as an additional language getting enough support?) and it's chosen for the wrong reasons. Those are usually related to its perceived ease, accessibility of subjects, and availability of resources--because of course this usually has to be funded out of the student's own pocket (unless it is full-time doctoral research as part of a team on a funded project.) Very rarely is the subject matter actually inherently compelling, or can it be seen to be useful.*

• Once the researcher is in post, she (no disrespect to female colleagues, they are simply in the majority in Schools of Education and education in general), she comes under pressure to produce "research". Actually it's more specific than that--she is under pressure to produce publications, particularly articles in peer-reviewed academic journals that are unlikely to be read, except by people like her who have no interest in the topic itself, but need to "read up" on it for a literature review. And frankly, many of the papers listed at the end of articles will not have been read, at least not carefully. They are there because they were included in other articles... 

• Of course, if she is lucky, she may get to work on a funded project. But in many cases in education there are no new resources for research, and indeed not even any remission from teaching time (and certainly no relief of administrative loading). So we can imagine a parallel with learning; there is deep research indeed, but more common is surface research, conducted purely as a means of getting a record of publication. 

• And the pressure to publish and the growing expectation that in order to be employed in the first place one needs a Ph.D (even one in an irrelevant discipline, or so specialised as to be useless in an education context) --those factors are perverse drivers to generate quantity rather than quality.

• Not only does that not augur well, but the researcher's grounding in methods is unlikely to be adequate. Most of it will come from a mandatory module on a Master's programme (there will probably have been a final-year undergraduate version, too--but frankly ...)  The typical "Research Methods" module will require two assessed pieces of work. One will supposedly test content knowledge; "Critically** compare two approaches to education research..." While the other will be a draft dissertation proposal, intended to focus the candidate's attention on the practicalities of dissertation research--but in practice serves to send the unjustfied message, "You now know enough about methods to apply them to a real project." 

• Most of that comes down to "I can get away with a few interviews or a focus group if I'm pushed." underpinned strongly by "anything but having to do sums!"

• I'm not a devotee of Randomised Controlled Trials as the Gold Standard for research in education--I part company with Ben Goldacre on this for reasons Andrew Old argued very well here and here--but the wriggling I see on the part of educational researchers just to get off the quantitative hook gets me squirming sympathetically.

• They frequently argue that quantification is inherently positivist--a sort of robotic curse which afflicts those who count. No. Their problem is that quantification opens the door to all kinds of nasties---sample size, and standard errors, and (the Hallowe'en horror) significance testing if you venture into the Wild Wood of cross-tabulation! So they settle for a few descriptive statistics if pushed, preferably illustrated by pretty graphs courtesy of Excel (or SPSS) which tell us nothing.

• Worse, they don't even know what they don't know (Rumsfeld, 2004) about quantitative methods.

• It's not that quantitative methods are inherently superior to qualitative--the problems are that the drivers to select methods are rarely about suitability--they are about comfort and confidence and flexibility and even malleability.

...and then there is the question of what happens to the research, even if it is some good. Dissemination is very poor.

• There's an unintended consequence of the poor quality of the work to begin with--the proliferation of not-very-good journals, which have arisen in order to exploit publish it. (After all, if it isn't published somewhere it doesn't exist, does it? ***) I'm not going into the all the academic publishing scams which beset us, but the net has of course made it much easier to pump out rubbish mediocre material via e-journals, and even to publish everything in return for a page fee. (Here is a recent newspaper treatment. and here is a list of them. [I can't testify to the accuracy of the details] I tell the tale of a bizarre prior personal encounter with this world here.) This is not of course confined to education. 

• And the downside of peer review is the number of studies which do not get published (or even submitted for publication in many cases). They may not be very good (see above), but that is not the whole story. The more popular the topic, the greater the competition. Not-very-dramatic results (or too-dramatic "outlier" results) are less likely to get published. Replication of existing research--which is a really important if unsung (and unfunded) process to challenge or confirm the received wisdom--is less likely to be published. And in education in particular, anything which challenges received wisdom, however wrong it is, is less likely to see the light of day. (See this paper on the importance of replication and how it works in the case of the famous but dubious "Pygmalion in the Classroom" study. But see also here (and here: up-date 29.11.13) for a systematic attempt to replicate findings in psychology.)

• Most of it does not get read. There is simply too much. (See David Colquhoun on this, buried in a long but interesting post on academic publishing in science; he makes the point that some journals foisted on the library of University College London as part of publishers' bundling deals just do not get read at all, and many more hardly ever. One of my own papers has racked up a grand total of just 38 citations since 1999--and that's not bad. That's why I cut out the middleman and publish directly online--but even that doesn't guarantee readers...)

• And much of what does get read is only read because it has been hijacked and hyped by charlatans and vested interests. Who may be cavalier--to say the least--with perfectly respectable initial findings.

• So we end up with research which gets published and deserves to be (hurrah!), that which doesn't get published and doesn't deserve to be (a category whose invisibility makes it very difficult to know about unless you happen to be someone like me who has had to mark and/or referee scores, if not hundreds, of papers over the years), also the Type 1 errors; research which shouldn't have been published but was anyway (which is not only unfair, but the status of being published confers a degree of credibility which is unlikely to be contested by reluctant researchers who--rightly--lack confidence in their own judgement; hence the egregious and embarrassing fiascos of "learning styles" and the like). 

• And of course research which should have been published and wasn't (Type II error)--not that anyone is in a position to pronounce definitively on that...

...and then the process of "chinese whispers" (that doesn't sound very politically correct),

• through students' half-understood lecture notes about papers they have not and never will read...

• or text-book summaries, which try to distil complex ideas into a 200-word paragraph, and lay out all the theories/research/half-baked ideas/sheer bullsh*t alongside each other as if they didn't overlap and/or contradict each other in the interest of "even-handedness"...

• or workshops from consultants on Continuing Professional Development days in schools and colleges; consultants who are aware above all of the bottom-line and the pre-eminent need to ensure they get another booking, so they are more entertainers than disseminators of accurate information... (in real life, of course, many of those consultants will moonlight as Ofsted inspectors, so they will be listened to avidly by members of Senior Leadership Teams, or whatever else they are calling themselves this week...) 

• who will latch onto the the latest fad and push out a garbled version of it to their staff, in the hope that it will give them the edge when Ofsted next come calling... 

• and the staff themselves, who are in the terminal phase of initiativitis, and almost as cynical as this piece, and who latch on the simplest and most banal aspect of what the "research" (the quotes testify to how far it is from the genuine article--however flawed) is purported to have recommended, in order to appear to be going through the motions...

and the net result is that even the originators of some of the most influential ideas disown the ways in which they have been implemented. See, for example, Dylan Wiliam's considered judgement on the state of Assessment for Learning (AfL).

What's the answer? In the AfL case, it may be found in meta-analyses, but only in part: their syntheses may be subject to just the same distortions as I have touched on above. And, as a perceptive course member raised with me a couple of weeks ago, they are designed to homogenise all the data, so you can't get at the needs of particular groups.

Treat all educational "research" with suspicion. We need it, but we can't assume it's trustworthy. Even mine. (And this post doesn't even pretend to the soubriquet.)

---

* There are some exceptions--chiefly action-research projects undertaken by practitioners, where publication is not the point, but improvement of practice is.

 ** "Critically" is critical, as it were. It's the magic word which confers M-ness on everything it touches, rather like the Higgs field confers mass...

*** Action research is the exception--it exists in order to influence practice, even if that is only the practice of the researcher. Publishability is a bonus.