Report | Learning about evaluation with small cohorts

_{11 December 2023}

Summary

At the heart of impact evaluation is the need to establish a connection between cause and effect in order to explain how and why activities lead to changes in desired outcomes. While experimental and quasi-experimental evaluation methods can provide the higher education sector with the confidence required to state causal inference, it is not always possible to use these methods. 

Small and specialist providers, in particular, face difficulties when attempting to generate causal impact evidence, and even in larger institutions it can be difficult to use existing evaluation methods with smaller cohorts of students, such as those receiving highly targeted support or involved in specialist projects.

TASO’s ‘Impact evaluation with small cohorts’ project aimed to help higher education providers overcome methodological and theoretical challenges in evaluating the impact of interventions with small cohorts of participants, where traditional large-scale quantitative evaluation methods are not suitable. 

The project advisory team, from Manchester Metropolitan University’s Policy Evaluation and Research Unit, developed guidance on the use of eight evaluation approaches suitable for use in small-cohort contexts. This guidance was supplemented by hypothetical case studies showing how each method might be applied in a higher education evaluation context. 

Six project teams representing different higher education providers then tested four of these evaluation methodologies: Realist Evaluation, Contribution Analysis, Most Significant Change and Qualitative Comparative Analysis.

This report includes reflections from the project teams, including the challenges that arose during implementation of these methodologies; the benefits gained by small-cohort evaluation approaches; and the lessons learned. It also sets out a series of recommendations for higher education providers who want to evaluate the impact of widening participation and student success activities with small participant cohorts.

Read the report