On not trusting "the research"

The new Dean is energetic and enthusiastic and keen to raise the research profile of the Faculty, and has been trying to get me involved in bidding for EU funding for research in post-school education. I don't think I shall rise to the bait--I am after all now retired, and I have never been one for large-scale research projects.

However, the process, together with  my post of about two weeks ago about published research and the comment on it from a volunteer on one of the All Results Journals has set me thinking about the practicalities of doing research and the how they impact on the results and quite conceivably on the value of it all. I'm not talking about methodology as such, here, but the story of how research actually comes to be done in universities and beyond, and how we get to know about it and use it. I'm sure it varies according to discipline and institutional setting, of course, but some features are fairly common.

In education it is still possible to do research very cheaply. The same is probably true of mathematics and philosophy, but physics and engineering and other plant- and equipment-hungry disciplines are hugely expensive. That in turn has organisational implications, with an individual in an armchair or at a chalkboard (the mathematician's weapon of choice even now, I am assured) at one end of the spectrum and legions of post-docs and graduate students and collaborators directed by principal investigators of professorial status at the other. Those professors, of course, bemoan their fate in not being able to do much of the actual research themselves, because they are kept so busy bringing in the funding.

Even so, the stimulus to get me involved in a project was not the topic to be researched, but the possibility of  gaining funding. And funders have their own agendas, particularly in any area which touches on public policy as education does. The agenda may be up-front, as in cases where bids or even tenders are sought to research, say, the impact of a government initiative; or it may be less apparent in the case of bids to a research council for un-earmarked funds. In the latter case, the acceptance or rejection of an application for funds may be made on purely academic grounds, but even with published criteria for the judging of bids it is not always clear on precisely what basis the decisions are made.

There's nothing wrong with this, of course, and those who pay the piper are entitled to call the tunes; but it does mean that some areas of research have no chance of getting funded. I'm sure there could be some interesting work done on the place of selection by ability for secondary schools in the UK, but I doubt whether it will happen. And funders tend to want a pretty good idea of what the research will conclude before they commission it--yes, I know.

So the choice of research topic is not likely to be on the basis of the researchers' interests, beyond the titles of master's (and occasionally doctoral) dissertations. That may of course be a good thing. There really are priorities to be considered, and individuals making names for themselves are not the highest on the list. But there are few things less conducive to high-quality work than getting lumbered with something in which you have no interest whatever; a little motivation can go a long way.

And then there is the issue of what it is practical to research. Cost and time are major factors, naturally, but the limitations of the technical expertise of many researchers in education, particularly in quantitative methods, mean that many good ideas never see the light of day. And some studies are distorted by scaling back the scope to what is manageable in the time; the pressure of getting out the "product" --the papers rather than the new knolwedge they are supposed to contain-- makes it very difficult to undertake anything which will not bear immediate apparent fruit.

There's even more filtering on the basis of likely publication; if work flies so much in the face of fashion that a reputable journal editor is not likely to accept it, or (nowadays in the UK) it is unlikely to score highly on the REF, it probably won't even start.

And so to the "all results" issue, to which I referred in the previous post. Finding out what things are not true, and what does not work is valuable in its own right, but both researchers and their institutions and funding bodies (leaving aside the journals) are willing to make much of their lack of "success". As David Berreby blogged recently here:

Repeated experiments are an important test of whether a finding is "really out there" or an accident, so, as a number of psychologists have been saying lately to the public, it is kind of a problem that many experiments are never repeated. And that, when they are, failures to replicate are often consigned to the file drawer. A new hypothesis often excites people; the "null hypothesis"—nothing's out there, the prediction was wrong—not so much.

(But do read the rest of his post, too. And for a more general treatment of non-replicability and the "decline effect" see this superb New Yorker article now released from their paywall.)

Similarly there is serious work to be done combating erroneous beliefs, but in whose interest is it to do it? Here is an example of such work in medicine, where it matters more, but the vested interests are more powerful. ('Twas ever thus, as Frank Coffield explores in this paper, which expands on this point far more elequently than I can. Coffield has done his share on the de-bunking front, too, discussed here and here.)

And then there is the research which is "not quite there", which finds its way into the public and professional arena without benefit of peer review or even accountability. Like this blog, of course, and my websites and indeed most of the rest of what is on the web. Here, opinion takes over from evidence, dubious chains of inference thrive, snake-oil is assiduously sold, and we can lie through our teeth. It is difficult to evaluate such material (including that which appears in the commercial press).

If the failings of research are discussed in terms of Type 1 errors (finding something which isn't actually there--false positive), and Type 2 errors (missing something which is really there--false negative), then the probability is that a scenario such as I have described will lead to a form of Type 2 error--missing all the interesting stuff. There will be Type 1 errors too of course but those owe more to the charlatans than the failings of real "research".

In turn, this suggests that most of the research on which we would really like to rely just has not been done. I'm all for evidence-based practice, when the evidence exists, but we can't hope that "the research" (as if it were a consistent body of knowledge all pointing in the same direction) is sound enough to constitute a firm foundation for practice. Even Hattie and Marzano.

However, it's a better starting point than actively distrusting the research. Trust but verify. In other words, use it as the basis for your own action-research.