Assessing the Impact of Voluntary Services for Justice-Involved Women: What Have We Learnt from Evaluation and Research?
By Kevin Wong, Reader in Community Justice and also Associate Director at the Policy Evaluation and Research Unit (PERU)
Answering the question that neatly doubles as the title, this blog draws on the collective decades long experience of my colleagues, Caroline O’ Keeffe, Jackie Lowthian, Linda Meadows, Rachel Kinsella as well as my own involvement – in evaluating voluntary sector women’s services (hereafter women’s services).
It summarises the key arguments from our article published as part of a recent special issue of the Prison Service Journal to coincide with International Women’s Day 2025. We commend the issue as a whole as well (of course) our paper.
The overarching themes: “women, criminalisation and the need for differential approaches that recognise the part that gender plays in women’s pathways into the criminal justice system”[1], receive lucid articulation and interrogation. In short, a fascinating and insightful read.
What’s the point of evaluation?
The purpose of evaluation is, in the context of government guidance, to: “systematically investigate the effectiveness of policy interventions, implementation and processes, and to determine their merit, worth, or value in terms of improving the social and economic conditions of different stakeholders.” [2] Our evaluation reporting[3] spans several years where broadly the same findings and recommendations have emerged, while at the same time we have observed limited change. The cumulative knowledge about justice-involved women which many researchers have contributed to – which for example underpinned the government’s Female Offender Strategy,[4] has yet to fully find its way into commissioning and practice.
Limitations of current approaches
Evaluation aims, objectives and questions enshrined in invitations to tender, focus evaluative effort but also box-in evaluators. They give rise to dashed hopes of “clinching evidence”,[5] invest in X will return reduced recidivism of Y percent. After years of seeking the holy grail of clinching evidence and explaining (apologising) why our efforts have failed, we admit to a disenchantment with a policy orthodoxy for justice-involved women, that manifests as:
- Firstly, an over-fixation with reducing reoffending as the end goal of women’s services; and
- Secondly, an unthinking rush towards randomised experimentation as the only evaluation method that can confirm their worth.
What’s the alternative?
The complex landscape of women’s services, policy and commissioning steers us to advocate for a realist approach to evaluating women’s services. The question of “what works” with its seductive simplicity becomes the more nuanced ‘what works, for whom, in what circumstances, to what extent and why’[6]. Realist evaluation recognises the complexity and changeability of social programme implementation and delivery, that the mechanisms – which underpin women’s services, (practitioners interactions with women, women’s reasoning, the processes which affect their behaviours and so on) are affected by context (women’s characteristics, local infrastructure, socio-economic conditions, access to services, the requirements of other services, the co-operation or lack of co-operation of services; family and peer relationships). They generate outcomes intended and unintended: women secure housing close to supportive family, have better access to their children; however, it also takes three rather than one bus to attend their probation appointments and so they begin to miss them. Services will be optimal for some women but not others, but this could alter if circumstances change.
Realist evaluation works with the non-linear complex messiness of the social world as it is, rather than the tidy linear version in policy makers and commissioners’ heads. If we commission service A to do B + C then Y will happen. There are no magic bullets to deal with the complexity of marginalised women’s lives. Realist evaluation describes a ‘realist’ evidence-based pathway chain which through theory elicitation, then testing facilitates programme targeting and programme improvement.[7] This is a heuristic, however, it is instructive, the point of evaluation here is programme improvement, enabling service(s) to adjust and refine provision. Borrowing from Pawson and colleagues, women’s services are:
“…dynamic complex systems thrust amidst complex systems, relentlessly subject to negotiation, resistance, adaptation, leak and borrow, bloom and fade.” (p23)[8]
Even if women’s services themselves don’t change, things around them do: a new funding regime; new national and local policies/strategies emerge. Adaptation is constant.
What should policy makers do?
We recommend a return to first principles, provided by two essential government texts: Government Social Research (GSR) ethics guidance,[9] and the Magenta Book, Central Government Guidance on evaluation.[10]
Outcomes for women’s services. The types of outcomes and measurement of such should align squarely with what women want and what women’s services do rather than what commissioners would like them to do. In short, outcomes should be co-produced – which is clearly supported by Principle 1: Research should have a clear user need and benefit and Principle 5: Research should enable participation of the groups it seeks to represent.[11]
Research method. Realist evaluation approaches have the potential to provide a more sympathetic way of doing what evaluation can do well, not provide clinching evidence but instead facilitate service improvement and adaptation and provide a better understanding about what changes/outcomes have occurred and why. Figure 2.4 of the Magenta Book (p33) sets out a decision tree to select the most appropriate method for impact evaluation based on intervention type. [12] The characteristics of women’s services falls squarely within the conditions for adopting a theory-based evaluation approach, which realist evaluation fulfils. Moreover, a more careful consideration of research method operationalises Principle 5: Research should enable participation of the groups it seeks to represent which the guidance confirms “not only helps to ensure the effective dissemination and impact of research findings, but also is an important step in determining the most appropriate and effective research methods.” (p.5) [13]
Peer research. As one manifestation of co-production in research and operationalising the collaboration ethos of new public governance,[14] peer research involving justice-involved women has much to commend it. As a method it clearly upholds ethics Principle 5: Research should enable participation of the groups it seeks to represent[15]. Two of my colleagues trialled it over twenty years ago as a then novel approach, it has since become more widely adopted. However, effecting change, in this case evaluation practice, requires investment of additional resources, but also care in implementation and a willingness to forgo control as a professional researcher and share this.[16]
How to do this are provided in other texts precluded from inclusion here by word limit. [17] Experiences of these co-produced practices, specifically, training and supporting peer researchers to undertake research with women has demonstrable benefits: reducing the gap between researcher and researched; women can be themselves, say what is true for them rather than a filtered version; offering a richer, alternative insight that a ‘professional researcher’ may not be able to elicit. For the peer researchers, it allowed them to forge a professional pathway from their experiences, it gave them purpose, self-worth, status and an alternative identity. Of course, being a peer researcher is not an unalloyed good. The challenges they experience – confusion of identity; going from the ‘high’ and self-affirming experience of being a researcher to returning to prison and resuming their status as prisoner speaks to the challenges of such peer roles.[18] Careful attention to managing these contradictions is important.
[1] Lowthian, J. (2025) Editorial. Prison Service Journal March 2025 No 277
[2] Government Social Research Unit (2007). The Magenta Book: guidance notes for policy evaluation and analysis. HM Treasury.
[3] Kinsella, R,. Meadows, L., O’ Keeffe, C., Wong, K. (2023). Evaluation of the ‘wrap around service’ for the mayor’s office for policing and crime (MOPAC), Manchester Metropolitan University. Unpublished. Kinsella, R., Clarke, B., Lowthian, J., Ellison, M., Kiss, Z., Wong, K. (2018). Whole System Approach to Women Offenders Final Evaluation Report, Manchester. Manchester Metropolitan University. O’ Keeffe, C., Ellingworth, D., Lowthian, J., Clarke, B., Wong, K. (2016) Evaluation of the Whole System Approach for Women Offenders Progress Report. Sheffield Hallam University.
[4] Ministry of Justice (2018). Female Offender Strategy. London: Ministry of Justice
[5] Hough, M. (2010). Gold standard or fool’s gold? The pursuit of certainty in experimental criminology. Criminology & Criminal Justice, 10(1), 11-22.
[6] Pawson and Tilley (1997) Realistic evaluation; Pawson R, Greenhalgh T, Harvey G, Walshe K. (2005) Realist review–a new method of systematic review designed for complex policy interventions. J Health Serv Res Policy. 2005 Jul;10 Suppl 1:21-34. doi: 10.1258/1355819054308530. PMID: 16053581.
[7] Pawson, R. (2017) Evidence-based Medicine & Evidence-based Policy: The world’s most perfectly developed method & the 79-pound weakling? University of Leeds.
[8] Pawson R, Greenhalgh T, Harvey G, Walshe K. (2005) Realist review–a new method of systematic review designed for complex policy interventions. J Health Serv Res Policy. 2005 Jul;10 Suppl 1:21-34. doi: 10.1258/1355819054308530. PMID: 16053581.
[9] Government Social Research. (2021). GSR Professional Guidance: Ethical assurance for Social and Behavioural Research in Government. GSR
[10] HM Treasury. (2020). Magenta Book Central Government Guidance on Evaluation. https://assets.publishing.service.gov.uk/media/5e96cab9d3bf7f412b2264b1/HMT_Magenta_Book.pd
[11] Government Social Research. (2021). GSR Professional Guidance: Ethical assurance for Social and Behavioural Research in Government. GSR
[12] HM Treasury. (2020). Magenta Book Central Government Guidance on Evaluation. https://assets.publishing.service.gov.uk/media/5e96cab9d3bf7f412b2264b1/HMT_Magenta_Book.pd
[13] Government Social Research (2021) GSR Professional Guidance: Ethical assurance for Social and Behavioural Research in Government. GSR
[14] Osborne, S. (2006.) The New Public Governance? 1. Public Management Review 8 (3): 377–387.
[15] Government Social Research (2021) GSR Professional Guidance: Ethical assurance for Social and Behavioural Research in Government. GSR
[16] O’Keeffe, C. (2004). Object and Subject: The Challenges of Peer Research in Community Justice in British Journal of Community Justice Vol.3(1).
Buck, G,. Harriott, G., Ryan, K., Ryan, N., and Tomczak, P. (2020). All our justice: People with convictions and ‘participatory’ criminal justice, in The Routledge Handbook of Service User Involvement in Human Services Research and Education Edited By Hugh McLaughlin, Peter Beresford, Colin Cameron, Helen Casey, Joe Duffy
[17] O’Keeffe, C. (2003). Moving Mountains: Identifying and Addressing Barriers to Employment, Training and Education from the voices of women (ex) offenders. Sheffield: SHU Press.
[18] O’Keeffe, C. (2003) Moving Mountains: Identifying and Addressing Barriers to Employment, Training and Education from the voices of women (ex) offenders. Sheffield: SHU Press
