The Politics of Education Research & “What Works” Randomised Controlled Trials
D.C. Phillips, In The News, Methodology, Policy, Politics, Research Tuesday, June 23rd, 2015“Political and ideological differences, or perceived differences, are often argued-out explicitly, but sometimes the conflict is carried out via “proxies”.
In the case of the deep divisions about educational policies and practices, for the past couple of decades the battle has been pursued on both fronts – explicitly, but also via proxies in the area of research methodology, the proxies being advocates “pro” and “con” for the adoption of rigorous science as the model which educational research should aspire to emulate.
Thus, the use of the modifiers “empirical” or “rigorously scientific” have been vigorously contested; as one critic put it, to speak of empirical or scientific educational research is to be guilty of adopting “your father’s paradigm”, which presumably is not complimentary (Lather, 2004). In the eyes of critics like this, these labels carry important and contestable socio-political connotations and are not merely methodological in nature – they convey a false picture of research as being objective or value-free; they give a false sense that policies supported by the research are the only ones possible and that these policies are assailable only by the deluded; and finally, use of these labels leads to the denigration of modes of inquiry that are rooted in the humanities or in feminist theory, or that can be gleaned from the work of such late natives of the Continent as Foucault, Derrida, Baudrillard, and Gadamer, among others. In 1997 the then editor of the British Educational Research Journal wrote in support of a broadly postmodern methodological alternative to the dominant empiricist/scientific mode.
The arguments offered by the critics to justify such things as the claim that objectivity needs to be rejected even as an ideal, or that it is an imposition to require that research findings or interpretations of findings need to be adequately warranted, are too often “quick and dirty” or “deep and murky”, or else they are non-existent and their place taken by name-calling – it is not uncommon for those who push for objectivity as an ideal or insist upon competent warrants (a Deweyan notion), to be called “positivists”, the implication being that this is a “knock-down” argument.
It is safe to say that on all sides of the debates over methodologies and their implications, there is mutual incomprehension mingled with distrust. As indicated above, the critics of traditional empirical work see it as bankrupt, as an arm of conservative social policy, and they reject its supposed virtues of objectivity and scientific rigor as missing the point. Instead, educational research should move, as Lather mysteriously asserts, “toward a Nietzschean sort of ‘unnatural science’ that leads to greater health by fostering ways of knowing that escape normativity” (Lather, 2004, 27).
On the other hand, mainstream empirical educational researchers – who see themselves as a species of applied social scientists – do not see the point of the more radical criticisms of their work mentioned above, and not surprisingly many simply pay no heed. This group of researchers is not unified, however, for as will be seen in the following section a philosophically dubious wedge has been driven into them recently by no lesser agent than the US Federal government.
Adding fuel to the fire – indeed, turning it from a fire into a conflagration – and conferring a degree of face-validity on the political analyses of research given by the critics, a subset of empirical educational researchers have been in the eye of the storm because of their single-minded advocacy of scientific rigor, which they characterize extremely narrowly. Crucially, they have identified rigor – and sometimes scientific worthiness – with the use of one particular research method, which has lost them the support of empirical researchers who use other methods, and who might otherwise have been allies against the radical attacks on empirical research outlined above.
This so-called “gold standard” methodology is the randomized controlled experiment or field trial (RFT), which of course is an instantiation of at least one of J.S. Mill’s “methods of logic”. Chief among its virtues – the one that endears it to the US government – is the (supposed) fact that it can authoritatively establish that a program or intervention or treatment did, or did not, cause an observed effect; this, it is claimed, is precisely the kind of information needed for the rational shaping of educational policy.
Researchers who are advocates of the “gold standard” often go on to deliver a very backhanded compliment to their empiricist colleagues; for they suggest that other empirical research methods are useful, at best, only in “building up” to RFTs or in “augmenting” them (Boruch and Mosteller, 2002, 4) – the idea being, roughly, that such things as qualitative case studies might generate interesting causal hypotheses, but these remain merely hypothetical until supported by rigorous RFTs. To add insult to injury, two of the leading advocates for the use of RFTs have been quite denigratory when making a passing reference to the contributions that could be made to this research enterprise by philosophers, social theorists, and the like: “Even throat-clearing essays at times contribute to understanding.” (Boruch and Mosteller, 2002, 4)
Insults aside, it is important to note the serious narrowing of the purpose of inquiry that is embodied here – the aim of rigorous educational research simply is to establish whether or not treatments or programs are causally efficacious, and other aims common in science such as unraveling causal mechanisms or developing and testing theories, or even the initial development of treatments that are likely to work, are ignored (see Phillips 2006).
It is also worth pointing out that although many of the vocal advocates for use of the “gold standard” are researchers based in the USA, the same approach has strong support in the UK and Europe where “evidence based policy” and “what works” are catch-phrases, and where it is held that the “evidence” and the “working” are determined most reliably by strong experimental studies paralleling those that are performed in the medical sciences to establish which particular treatments are effective. A few philosophers of education are starting to pay attention to the assumptions here, though not particularly effectively.
As hinted above, there has been an astounding twist of fate in the USA, one that gives further support to the charge that methodology and politics are closely intertwined in fields like education. Around the turn of the millennium the arguments and pronouncements of this pro-“gold standard” group of researchers found favor among those in power in conservative political circles in Washington who view the field of educational research as lacking rigor, and as being too influenced by ideology (views for which, admittedly, there is some evidence from an evaluation of research in the UK, see Tooley and Darby, 1998). Unfortunately this ideological sensitivity on the part of the politically-powerful does not extend to recognition of how their own ideology influences policy and (mis)shapes research. Thus, since the early years of the Bush (junior) Administration, Federal educational research funding in the USA has been available only for the conduct of RFTs, which means that empirical work using any other methodology goes unfunded. This is a draconian policy indeed, especially when promulgated in the name of scientific rigor. The US National Academies of Science, through its executive arm, the National Research Council (NRC), tried to liberalize this funding policy by issuing a report that made room, in the name of science, for such things as ethnographic research, but to no avail (NRC, 2002; for documentation of the relevant pieces of legislation, and debates over both the federal funding policy and the motives behind the NRC report itself, see Phillips, 2006) It also is of some interest that the NRC report was interpreted by some of the radical critics of research as being in fact part of the conservative political weaponry being deployed to suppress dissident voices, rather than being an abortive effort to liberalize the notion of “science” that was being abused by supporters of the “gold standard”.
In the years since the promulgation of the “gold standard” funding policy, the nature of rigor in various types of quantitative and qualitative research has been much discussed, as has the nature of science, and the mischaracterization of the history and methods of science that follow upon adoption of the RFT as the “gold standard” of scientific work. It has been pointed out that Galileo, Newton, Darwin, Crick and Watson, and Hawking, would not be regarded as doing rigorous, fundable scientific work under the US Department of Education definition (Phillips, 2006).”
D.C. Phillips (2009) Empirical Educational Research: Charting Philosophical Disagreements in an Undisciplined Field in Harvey Siegel ed. The Oxford Handbook of Philosophy of Education , pp.386-389.