Project | Learning about evaluation with small cohorts: pilots to test the methodologies

About the project

To evaluate impact of interventions, TASO promotes the use of rigorous methodologies. However, these traditional methods may not be suitable for interventions with small cohorts of participants. TASO evaluation guidance, impact evaluation with small cohorts, is designed to help higher education providers overcome these challenges.

The guidance sets out eight evaluation approaches suitable for use in small-cohort contexts. This project involved six higher education provider teams testing four of these evaluation methods: Realist Evaluation, Contribution Analysis, Most Significant Change and Qualitative Comparative Analysis. The six provider teams and their reports, case studies and Theory of Change documents are listed below.

The pilot project teams identified the challenges that arose during the implementation of these methodologies, the benefits produced by small-cohort evaluation approaches, and the lessons learned from the pilot projects. 

A series of recommendations were developed for those higher education providers looking to evaluate small-cohort interventions.

Reports

Webinar

Webinar launch: ‘Learning about evaluation with small cohorts’

Related outputs

City College Norwich

• Report: Contribution analysis – impact evaluation with small cohorts (PDF)
• Case study: Contribution analysis case study: Evaluation of the higher education tutorial supervisor role (PDF)
• Theory of change: Theory of Change for City College Norwich student support intervention: Higher education tutorial supervisor role (PDF)

Leeds Arts University

• Report: Evaluation with small cohorts: Contribution Analysis of a Leeds Arts University access intervention (PDF)
• Case study: Contribution Analysis case study: Evaluation of Creative Pathways Access Programme (PDF)
• Theory of change: Theory of Change for Leeds Arts University access intervention: Creative Pathways Programme (PDF)

Plymouth Marjon University

• Report: Using Transformative Evaluation (TE) in a Higher Education context: reflecting on the use of TE through the evaluation of the Student Colleagues scheme (PDF)
• Case study: Transformative Evaluation case study: Evaluation of Student Colleagues scheme (PDF)
• Theory of change: Theory of Change for Plymouth Marjon University employability intervention: Student Colleagues (SC) scheme (PDF)

University Centre Leeds

• Report: Gypsy, Roma, Traveller: community outreach programme (PDF)
• Case study: Realist Evaluation case study: Evaluation of Gypsy, Roma, Traveller Community Outreach Programme (PDF)
• Theory of change: Theory of Change for University Centre Leeds access intervention: Gypsy, Roma, Traveller community outreach programme (PDF)

University of Suffolk

• Report: Small ‘n’ Evaluation: Micro-placements: A Realist Evaluation (PDF)
• Case study: Realist Evaluation case study: Evaluation of micro-placements (PDF)
• Theory of change: Theory of Change for University of Suffolk employability intervention (PDF)

University of Leeds, Lifelong Learning Centre

• Report: A small-n study using Qualitative Comparative Analysis (QCA) to understand the combination of factors that contribute to a rise in self-reported confidence in participants on a non-accredited pre-entry programme (PDF)
• Case study: Qualitative Comparative Analysis case study: Evaluation of Jumpstart (PDF)
• Theory of change: Theory of Change for University of Leeds, Lifelong Learning Centre access intervention: Jumpstart  (PDF)

Background

At the heart of impact evaluation is the need to establish a connection between cause and effect in order to explain how and why activities lead to changes in desired outcomes. While traditional experimental and quasi-experimental methods can provide the higher education sector with the confidence required to state causal inference, it is not always possible to use these methods, particularly for small and specialist providers.

The higher education sector needs to use a wider range of impact evaluation methodologies. This project aimed to develop and test a range of evaluation designs that could be used to assess the impact of small-group interventions.