Original content taken from LSE Impact of Social Science Blog
To critics across higher education, evaluation frameworks such as the REF and TEF represent mechanisms of control, the generation of a “target and terror” culture. Deirdre Duffy explains how the REF and TEF resonate most closely with impact evaluation, a form of evaluation that can prove useful for a simple cost-benefit analysis but can also be problematic as it encourages causal inference, is reductionist, and can dissuade creativity in outputs. However, evaluation itself need not be a bad thing. It can be reclaimed by academics and universities to combat both its misuse and undoubtedly negative impacts, and even be used to reveal and challenge the darker, less merit-worthy elements of HE.
Evaluation is political. It is an established part of “good governance”. It acts as a form of soft power where subjects of government are oriented towards particular goals. In higher education, the targets of this power are both academics and institutions. According to critics across HE, evaluation frameworks such as the Research Excellence Framework and Teaching Excellence Framework are mechanisms of control.
But while the depiction of the REF and TEF as governing through measurement (or “target and terror”) is accurate, as Duncan Green recently highlighted, rejecting evaluation wholesale is equally problematic. The reason for this is simple: there is more than one way to do evaluation.
With this spirit of openness to evaluation in mind, I want to discuss (i) the “type” of evaluation that the REF and TEF represent, and (ii) the potential to approach evaluation differently.
Evaluation R/TEF-style: impact, impact, impact
Unsurprisingly, current HE evaluative frameworks resonate most closely with impact evaluation. This form of evaluation research is principally concerned with outcomes and represents what Huey T. Chen labels “black box evaluation” where the “inside” of the “black box” is judged on the basis of a simple comparison between inputs and outputs . Black box evaluation is useful in scenarios where, for example, the primary goal is a simple cost-benefit analysis. Unfortunately, at a fundamental level, straightforward black box evaluation is less useful for pretty much anything else as it suffers from two key problems. First, it encourages causal inference (i.e. assume that X caused Y); and second, it gives no detail in the way of process (i.e. what led X to result in Y?).
It is also reductionist, a problem that impact evaluation tries to combat by “complicating the box”. More outcomes are identified and more inputs included. Adding multiple objects of evaluation (evaluands) results in the evaluative frameworks HE has now – think of the National Student Survey. While these extended evaluative frames represent a more context-sensitive “box”, they suffer from precisely the same problems as a much simpler version.
Impact evaluation can also be problematic where units of measurement are specified in advance. Focusing on one measure over others can dissuade any creativity in outputs. This may be desirable in certain circumstances but in HE is not and contributes to the prioritisation of certain kinds of output (journal articles!), for example. Pre-set measures or targets can also ossify the system itself as everything becomes oriented towards particular products (even where their value is questionable outside the system). There are further effects of impact approaches which are now well-recognised in HE: the generation of a “target and terror” culture and a fear of numbers.
The other sides of evaluation
Against the gloomy background of REF and TEF, it is easy to forget that there is much more to evaluation than impact and control. What distinguishes evaluation from other forms of research is not its focus on impacts and outcomes but its commitment to three research objectives: appreciating merit, engaging with the real world, and producing evidence to assist action. While impact evaluation in HE defines these commitments according to neoliberal logics (and promotes economic value as a result), writers on evaluation like Jennifer Greene and Melvin Mark et al. argue that “merit” and useful evidence need to be defined in terms of how much social injustices are addressed. Evaluation is part of participant action research – research which engages communities in identifying ways to resolve problems affecting them. Approaches such as transformative evaluation, utilisation-focused evaluation, and complexity evaluation promote ongoing, critical reflection on the workings of a programme or system with the aim of changing it for the better. Importantly, approaches like realistic evaluation and responsive evaluation recognise that they will never capture everything that is merit-worthy (or not!).
I do not want to suggest here that there is a “good” evaluation that HE has not discovered yet because we have been so obsessed about the “bad” evaluations. Neither do I want to claim that there is a “magic bullet” evaluation with none of the disciplining edges of the REF and TEF. Evaluations will always reflect underlying social-political and individual beliefs and, interpreted uncritically, even “social justice” evaluations can reinforce the problems they highlight.
But I do want to defend evaluation as a method and suggest that it can be reclaimed by academics and universities to combat both its misuse and undoubtedly negative impacts (an example of this is the prioritisation of particular types of academic output over others; another is the consumerisation of HE). Evaluation could, theoretically, be used to reveal and challenge the darker, less merit-worthy elements of HE. Take the growing precarity of new academics. What if HE institutions had to document, monitor, and improve the ways they had supported early-career academics, the mentoring programmes, and the processes they had for enabling early-career researchers to publish in the REF? Or what if the REF had to detail the workload of academics alongside their outputs?
Similarly, the TEF could bring positive change in HE if it compelled HEIs to implement, monitor, and publicly document their approaches for tackling the exclusion of black and minority ethnic students. Or if it promoted the efficacy of HEIs’ processes for preventing and responding to sexual assault and harassment as an indicator of “excellence”.
All of this seems incredibly idealistic (and, to a cynic, naïve!). But that is the key point to remember. Evaluation is idealistic. It encourages people to look at “what is really happening” and think about what an ideal situation would or should be. As the REF and TEF become used to direct HEIs towards economic value, effectiveness, and efficiency we need to remember that the ideals of HE include much more than happy customers and value for money and insist that our evaluations do the same.
 It is worth noting that HE evaluation is reflective of the UK context. Despite attempts to introduce less impact-focused evaluative approaches – Ray Pawson’s work on “realistic evaluation” is worth reading here – impact analysis is favoured by central government. This preference is reflected in guidance on evaluation in central government which treats cost-benefit analysis and evaluation as almost synonymous.
Evaluation and Governing in the 21st Century: Disciplinary Measures, Transformative Possibilities, is currently available for pre-order with Palgrave Macmillan as part of the Palgrave Studies in Science, Knowledge and Policy series (edited by Dr Katherine Smith and Dr Sotiria Grek, University of Edinburgh). Her ORCID iD is 0000-0001-9941-1587.
Original post URL: