Lessons from London Schools, Self-Promotion and the Myth of Education Research
Academies, ARK, CfBT, Charter Schools, EdReform, EduBusiness, Free Schools, In The News, League Tables, Management Consultants, McKinsey & Co., OECD, Ofsted, PISA, Research, Teach First, Teach for America, Teach for Australia, Testing, Unions Monday, June 30th, 2014
We were a little undecided as to whether or not to write this post because of the youthfulness of those involved in Lessons from London Schools: Investigating the Success (LLS), the study that we will critique, and not wanting for youthful enthusiasm to be overly dampened by what is to be said about the LLS study. We then wish to state that we sympathise with those like the LLS authors who wish to make a difference to schools in England, but feel that what this study represents is far too important to let pass even if the critique’s message may initially perhaps sting some pride a little.
The LLS study has featured this week in national and local UK media (e.g. BBC News) and been a feature of social media. It contains 6 chapters and is over 130 pages long.
LLS investigates “the claim that London schools have improved dramatically since 2000” (p.6) and ‘generates five key findings’:
1. London schools have improved dramatically since 2000.
2. The improvement cannot be explained in terms of the advantages that London has over the rest of England.
3. The improvement was assisted by a set of factors that we describe as ‘enabling’, these include issues relating to resourcing: finance, teacher recruitment and school building quality. Improvement in these areas enabled improvements to flourish but London’s success was not fundamentally caused by these factors.
4. Four key school improvement interventions provided the impetus for improvement – London Challenge, Teach First, the academies programme and improved support from local authorities. Our research identifies common features that link together all of these interventions.
5. The improvement of London schools depended upon effective leadership at every level of the system.
In the post that follows, the key findings of LLS are disputed. However, in our critique the intention is not to suggest that other possible findings either discounted in LLS or overlooked completely are somehow more important, but rather our intention is to illustrate, with LLS as a good example, a more fundamental problem with education research itself. In this sense, LLS is a manifestation of a much wider and more concerning malaise.
At the outset, however, it is worth stating that the critique of LLS won’t centre on the question of judging school improvement in terms of exam results and OFSTED inspection grades (p.8). Whether school quality can effectively be encapsulated in a test score or inspection grade will be passed over although the ‘unintended’ consequences of high-stakes testing will be central to the critique below.
The critique of LLS and what it represents will centre on two inter-related things: the backgrounds of the organisations and authors of LLS and secondly the particular interventions singled out as being primarily responsible for London’s supposed improvement since 2000. The outcome of their critique will then form the basis for broader rumination on the current state of education research and whether such research is possible or even desirable in the form that studies like LLS represent.
To begin, take the claim (p.7) that the academies programme has been one of four key school improvement interventions providing the impetus for improvement in London since 2000. For those that don’t know, academies (and ‘free’ schools) are schools in England run free of local authority control and funded directly by central government. They are also very controversial.
LLS is published by CfBT. CfBT Schools Trust, “set up as a separate entity within CfBT Education Trust,” is currently closely involved with the academies programme. According to CfBT’s website, for example, “CfBT Schools Trust partners a wide range of schools in England from free schools to academy conversions. We work closely with schools to deliver specialist support where required to improve performance.” CfBT Schools Trust also “helps schools in England to become academies whether as a member of the CfBT Schools Trust or as a standalone academy requiring project management support for the conversion process.” CfBT Schools Trust is an established free school provider in England. We work in close partnership with parent and community groups to support free school applications, oversee the opening and management of the schools.”
Given how committed CfBT is to the academies and free schools programme, what might we conclude about its hopes for LLS? We might, for instance, think CfBT would want LLS to speak as highly as possible about the types of things, such as its work in academies and free schools, CfBT is undertaking and think it similarly odd especially given CfBT’s support for LLS if LLS’s findings did not align with CfBT’s work. Turkeys don’t vote for Xmas, as they say.
Of those responsible for LLS, it is not only CfBT who we might think have a vested interest in academies being regarded as a key school improvement intervention. Take Centre for London, another organisation responsible for LLS.
Of Centre for London’s trustees, for example, one, Lucy Heller, “is the managing director of ARK Schools, a children’s charity which runs a network of inner-city, high achieving academies. She oversees the running of 27 academies in disadvantaged communities throughout the UK. She is responsible for a model of education which is driving current government education policy.”
Organisations that have profited from the academies programmes also support the Centre for London. Take PWC and SERCO, for instance, or take University College London (UCL), who run their own academy school.
If we look to the authors of LLS we find similar professional investment in the academies programme. For example, a quick internet search tells us that Laura McInerney, who also writes for the Guardian and other publications, is a known advocate of academies and free schools. In these publications and on her blog she has frequently written in their support, has authored a book on free schools and has recently returned from the University of Missouri where she was studying free schools, academies and charter schools.
A second key school improvement intervention providing the impetus for improvement in London schools since 2000 was said in LLS to be Teach First. Teach First is an alternative teacher accreditation programme modelled on Teach for America and part of the Teach for All network. Teach First recruits are placed in schools in disadvantaged areas on two-year contracts and given responsibility for a class of their own without the level of training typically expected of trainee teachers.
Take Ms. McInerney again. The internet tells us that she became a teacher through Teach First and taught in London for six years. Strangely, LLS doesn’t tell us this, however, and stranger still given the openness about other details of the LLS authors’ backstories (p.4).
Interestingly, other co-authors of LLS also have strong connections to Teach First but again strangely this information isn’t found in LLS either. For example, Eleanor Bernardes was Coordinator for Literacy and Approaches to Learning at Teach First between September 2009 and May 2013. Loic Menzies is a graduate of the Teach First programme where he taught Citizenship between 2003 and 2005 and since 2013 he has been an Impact Committee Member for Teach First.
The Teach First connection to LLS is identifiable elsewhere. For instance, given its history, it is not unreasonable to think Teach First is a McKinsey & Co offshoot, and the management consultancy McKinsey is a supporter of the Centre for London. Sam Freedman, Director of Research, Evaluation and Impact at Teach First is one person explicitly thanked in the LLS report’s acknowledgements (p.5) and so too is Rebecca Allen, a quantitative researcher with a favourable opinion of Teach First.
Given the close links between LLS and Teach First it is then perhaps no surprise that on 27th June 2014 a TES article on the release of LLS begins not as you might expect with a headline about the report per se but begins with a headline (‘Charity wants to repeat the ‘London effect’ around Britain’) about what LLS means for Teach First and where we are told at the start of the article that,
“Hundreds more schools will be able to obtain staff from Teach First as the recruiter of high-flying graduates aims to repeat its success in London in more parts of Britain…News of Teach First’s expansion comes as a report credits the charity with being one of four “key” reasons for the incredible transformation of the capital’s state schools since the turn of the millennium. The scheme has already grown outside London, but now it wants to work with bigger groups of schools in their local areas. Sam Freedman, research director at Teach First, said: “While we have seen significant improvements across many schools in England and Wales, and especially in London and other big cities, a number of challenges remain, particularly in coastal and rural areas.””
We should recall too that academies are thought more likely to employ Teach First recruits. This is partly because sponsor academies will typically be found in areas of disadvantage where Teach First recruits are sent but also we might think because Teach First shares much of the corporate ethos and philosophy behind the academies programme and other social enterprises behind academisation such as CfBT. In advocating academies, in other words, Teach First benefits.
These are not the only connections to highlight between LLS and Teach First, incidentally. For example, one of the studies (Hutchings et al., 2006) cited by LLS in its attempt to substantiate the claim of Teach First’s effectiveness in London since 2000 was conducted at an institution (i.e. Canterbury Christ Church University) where an author of LLS (i.e. Loic Menzies) is currently employed and where in 2002 a Teach First alternative accreditation programme was set up at the same time as Teach First was itself being established. The connection between this University and Teach First is then a very long one.
One further thing to highlight is that the evidence drawn upon in LLS to support the claim that Teach First is one key driver of improvement in London’s school performance since 2000 is Teach First’s own data (e.g. see p.80, 82 Teach First, 2013; See also Allen and Allnutt, 2013, p.3: “Thanks are due to Teach First and the Department of Education for providing the data that is used in this study”) or is data drawn from studies which Teach First’s major donors have funded (Muijs et al., 2010: “This study was commissioned as part of the Maximum Impact Programme (MIP), funded for Teach First by the Goldman Sachs Foundation” (p.3)). The LLS report is therefore data-dependent on the very organisation it is trying to evaluate.
So why draw attention to the obvious links between academies and Teach First and the organisations and authors behind LLS? To explain why this matters I shall quote Ben Goldacre, UK medical science author, media commentator, and champion of the use of randomised-controlled trials (RCTs) and systematic reviews in education. Goldacre (Ch.1 ‘Missing Data’ Bad Pharma, 2012, p.1) writes, “Before we get going, we need to establish one thing beyond any doubt: industry-funded trials are more likely to produce a positive, flattering result than independently-funded trials…this is one of the most well-documented phenomena in the growing field of ‘research about research’.”
While Goldacre’s book is about the pharmaceutical industry, as a scientific practice we can quite reasonably think that in this regard education research is no different. We could therefore just as easily write: “Before we get going, we need to establish one thing beyond any doubt: industry-funded trials IN EDUCATION RESEARCH are more likely to produce a positive, flattering result than independently-funded trials…”
Above all, in other words, to be legitimate any kind of research needs to be independent. Yet as we have seen LLS is clearly not independent of those (i.e. advocates of academies and Teach First) who we might reasonably think have a vested interest in seeing in LLS certain interventions promoted rather than others. Even if this has not been the intention, if studies such as LLS are not fully independent there will always be the suspicion of the education industry promoting its own ends through so-called research such as this. It is naïve to think anything else.
This undeniable lack of independence is however not the only reason for taking what LLS and other similar studies find with a large pinch of salt. There is the second inter-related issue that was introduced above: the highlighting of the four key improvement interventions in particular.
In this regard, one initial point to make concerns the nature of the world that must pre-exist for a study of this kind to make any sense. For a study like LLS to identify any key improvement interventions the world, and our relationship to it and our relationship to others, must be characterised by causality.
Causal thinking is the kind of everyday thinking whereby it is thought if we do A, then B, but if we don’t do A, then not B. If I pull this lever, something happens at the end of it, but if I don’t pull that lever, then nothing happens, in other words.
Causally is how we normally see the world and is also how the world must work for LLS and other studies to make any sense. It is then not surprising to find causal thinking found throughout LLS. For instance, we are told that LLS “analyses the nature and causes of the changes in London schools and demonstrates that it is possible to tackle the link between poverty and underachievement” (p.2); “There is a need, therefore, for an investigation of the nature and possible causes of the changes in London’s schools” (p.6); “As ‘policy in action’, none of the major London reforms were planned with a concurrent rigorous evaluative element or any randomised controlled trial (RCT) element. While this limits the ability of any subsequent review to make firm claims for cause and effect, our approach allows us to present the most detailed analysis to date of the London story” (p.7); “THE EXCEPTIONAL IMPROVEMENT WAS ASSISTED BY, BUT NOT FUNDAMENTALLY CAUSED BY, RESOURCING ISSUES RELATING TO FINANCE, TEACHER RECRUITMENT AND SCHOOL BUILDING STOCK” (p.10) etc. etc.
What isn’t considered by LLS nor typically by any study of a similarly causal nature, however, is that any social change such as an increase in test scores or inspection grades may not have a cause, lever or reason at all. It isn’t ever considered that what for example has happened since 2000 has simply been a random event.
That it might be a one-off random event isn’t considered because the everyday world of the education researcher is typically one of causes and effects and it is their job to identify them, and identify them they must. They have to identify them even when it might just be that causality is not responsible in any way for the improvement say in London’s test scores and grades. Such change might simply have happened to be this way for London schools since 2000. There might simply be no reason whatsoever.
If for argument’s sake we agree that causality is at play in the world that education research investigates and one or more interventions are responsible for test score and inspection grade improvement in London from 2000, we can still object to the LLS research outcomes. We might argue, for instance, that even with their interventions and the polite coverage they give to those interventions popular elsewhere but that they reject (e.g. Gentrification, Ethnicity, Opportunity, p.9), LLS could still be overlooking something. Indeed, we might say that they are overlooking something seemingly minor, and not typically a variable that the consensus acknowledges, but still a variable of great importance.
Consider for example what the French mathematician Blaise Pascal famously wrote centuries ago about Cleopatra’s nose: “Cleopatra’s nose, had it been shorter, the whole face of the world would have been changed” (1669). What Pascal writes here is extremely telling for his comments could be construed as meaning that while we might think we have identified the typical causes of something such as school improvement we cannot discount completely the fact that in comparison to the causes social scientists usually work with (e.g. gender, class, race, income etc.) something’s cause may be something altogether different and minor (e.g. such as the shape of a woman’s nose). And if we can’t discount this possibility, if there may always be another explanation however trivial, any conclusions that studies such as LLS reach can never be fully convincing. Period.
There’s more, however. Those behind LLS, and other similar studies, do not consider something else that is similarly undermining of their research enterprise. In the case of LLS, they do not consider whether the four interventions they identify might be holding school improvement back. It might be of course that without the academies and Teach First programmes, for example, schools in London would have done even better since 2000.
Of course, it might not be possible to establish this in the UK but we might take note of international developments and where similar school choice and alternative accreditation reforms have taken place for here the evidence is decidedly unconvincing. In the US, for example, charter schools have been a feature of schools there since the 1990s but there the results are mixed. In Sweden, OECD PISA scores have fallen since free schools were introduced. In Chile, school choice is being rolled back. In the UK, free schools have been opened but then closed, some academies have been successful but others have not. The 12th grade National Assessment of Educational Progress (NAEP) report in the US suggests overall scores for high school seniors are flat and reading scores in 2013 were lower than in 1992, the year after the first charter school opened in America. It is worth remembering too that no leading OECD PISA participant has a Teach For All organisation such as Teach First.
There is then good reason to think that since 2000 interventions such as academies and Teach First in London might just as likely have been making things worse not better and good reason for thinking that if there hadn’t been these interventions things in London might have been a whole lot better. Even if we discount all of the above, and we follow LLS in thinking that the interventions they identify (e.g. academies, Teach First) are the only ones that have counted, follow LLS in thinking that these interventions have had a positive causal impact and follow LLS in thinking that London schools have been better with them than without, we might still be concerned with what LLS have found. We might think for example that the knock-on effects of these interventions on education have been too damaging for such interventions to warrant continuation even if they had been successful in terms of test scores.
We might think that placing such an onus on test scores and inspection grades as a means of assessing the success of interventions such as academies and Teach First recruits, for example, incentivises teaching to the test and gaming the system and disincentivises offering a broader range of subjects and curricular opportunities to pupils. We might then think that even if changes to school management and teaching personnel have raised London school performance scores this is still a price not worth paying because London kids are missing out on a full and sufficiently broad education.
Incidentally, other problems associated with interventions such as academies and Teach First have also been identified. These include their greater cost to the taxpayer, their frosty attitudes towards union representation, the lack of democratic and local oversight, lack of accountability for their use of public funds, their exacerbating of teacher turnover, possible negative impact on wages, potential threat to pensions and terms and conditions, and contribution to the neoliberal corporatisation of education.
Faced with the possibility of such damaging consequences, we might try another way of assessing the quality of these interventions, where unlike with sole reliance on test scores, the potentially harmful effects of interventions can be fully taken into account. Or, alternatively, because of the harm these interventions might be having on curricular, teacher autonomy and local communities we might instead consider scrapping these key interventions altogether.
Ultimately, though, the major problem with any causal study of LLS’s kind is that there are countless other potential independent variables that could be said to account for why school performance scores have improved. It might just well be Cleopatra’s nose after all.
What this means of course is that the right answer to the question of what is driving school performance in London is not self-evident in the data. The data needs interpretation, in other words, but as is also well-recognised interpretation opens the door to interest and bias. This means the data is fitted to the interpretations that we prefer rather than data telling us what is the true.
There is certainly disagreement about where our interests and biases are said to emanate from i.e. whether they are innate or socio-cultural, for example. But there is pretty much wholesale agreement that bias and self-interest is a fact of life not only for researchers but for us all.
So not only might there be many numerous possible reasons for school performance improving in London in 2000, the interventions that are chosen by researchers such as the authors of LLS are not self-evident findings but findings unavoidably reflective of personal interests and bias. It is important to say, incidentally, that the unavoidability of these prejudices applies as much to those who say that variables such as poverty, years of experience and class size are crucial to schools performance as it does to those who say teacher quality, school type and leadership are. All of these key findings reflect researcher interest and bias.
It is then not that academy organisations, their supporters, Teach First and so on are especially or anymore biased and self-serving than anyone else (We might think that the current relationship between Teach First and some younger members of the education research community exposed above is slightly suspicious, perhaps). Rather it is that by focusing on the interventions favoured by those with a close connection to LLS, LLS draws attention to something much more fundamental i.e. bias and self-interest and something that unavoidably afflicts all sides of education research.
The inevitability of self-interest and bias pervading research studies such as LLS is why because they are thought much less susceptible to bias, RCTs are currently the gold standard for research (as LLS admits (p.7)). Another favourite of education researchers, Karl Popper, also recognised long ago that we make our observations agree with our theories (what he called ‘Bacon’s problem’) and it was Bacon’s problem that led Popper to propose his popular criterion of falsifiability/refutability/testability (Myth of the Framework, 1994, p.88).
What ought then to be our response to the education research predicament that LLS could be said to exemplify? We might as many urge seek solace in RCTs. However, doing this limits us to only those areas of study where RCTs are available and while they are growing in number, they are still too few in number to offer much insight, nor should we think this a panacea anyway given the unexpected and ever-changing particularities inherent in any educational context that RCTs choose to scrutinise.
Instead, we might in Popperian fashion seek to continually test our research claims. Yet while this might work for the hypotheses of teachers in situ in the classroom, it is obviously more difficult in terms of time, manpower and resources to constantly test and retest policy interventions of the kind highlighted in LLS.
Faced with the dilemma of inevitable bias and self-interest pervading our findings, but faced also with insurmountable problems with RCTs and falsifiability, we might throw up our hands and concede we are not in any position to say what, for example, has been driving school performance in London since 2000. While understandable, this is allowing the pendulum to swing too far the other way, however. Because we don’t know something fully doesn’t necessarily discount completely what we know.
We need a position then where the possibility of knowing something is acknowledged but where it is also acknowledged that what we know is more than likely subject to personal interest and bias and so our faith in what we know acknowledged as limited and not by any means the final word. Not being a final word on a subject such as school performance is to understand our research claims not as the ultimate truth therefore but as contributions to something else.
What this fallibilist position could be understood as is as an ongoing conversation. This pragmatic type of conversation is not characterised by a one-way expert-led declaration of truths as found in education research such as LLS but characterised by an open dialogue where all stakeholders in education, for example, are respected as equally knowledgeable but thought too equally circumscribed in what they know. This is a conversation that ought to encourage collaboration and working problems through together and be a conversation characterised not by narrowing things down to one or other set of interventions and favoured by this or that interested party but characterised by an opening up to the potentially invaluable insights of others so that all others may contribute.
Of course, an alternative response to the problems inherent in education research is to sweep issues of self-interest and bias under the carpet, issue disclaimers, add caveats, hedge bets and so persist with the myth, dogma and bad faith of the status quo. But is this really what we want? An ongoing, fallibilist, all-inclusive conversation of equals is surely better than that.
