Learning about evaluation with small cohorts: pilots to test the methodologies

Learning about evaluation with small cohorts (pdf)

The impact evaluation with small cohorts project is designed to help higher education providers overcome methodological and theoretical challenges in evaluating the impact of interventions with small cohorts of participants, where traditional large-scale quantitative evaluation methods are not suitable.

An initial consultation phase revealed that respondents from across the sector face a range of challenges when evaluating small-cohort interventions.

The project advisory team developed guidance on the use of eight evaluation approaches suitable for use in small-cohort contexts.

This guidance was supplemented by hypothetical case studies showing how each methodology might be applied in a higher education (HE) evaluation context.

TASO report: Learning about evaluation with small cohorts

• Read the overarching report: Learning about evaluation with small cohorts (PDF)

Higher education provider project teams: testing the evaluation methodologies

Six project teams representing different higher education providers (HEPs) tested four evaluation methodologies: Realist Evaluation, Contribution Analysis, Most Significant Change (Transformative Evaluation) and Qualitative Comparative Analysis. The six HEPs and their reports, case studies and Theory of Change documents are listed below.

City College Norwich

• REPORT: Contribution analysis – impact evaluation with small cohorts (PDF)
• CASE STUDY: Contribution analysis case study: Evaluation of the higher education tutorial supervisor role (PDF)
• THEORY OF CHANGE: Theory of Change for City College Norwich student support intervention: Higher education tutorial supervisor role (PDF)

Leeds Arts University

• REPORT: Evaluation with small cohorts: Contribution Analysis of a Leeds Arts University access intervention (PDF)
• CASE STUDY: Contribution Analysis case study: Evaluation of Creative Pathways Access Programme (PDF)
• THEORY OF CHANGE: Theory of Change for Leeds Arts University access intervention: Creative Pathways Programme (PDF)

Plymouth Marjon University

• REPORT: Using Transformative Evaluation (TE) in a Higher Education context: reflecting on the use of TE through the evaluation of the Student Colleagues scheme (PDF)
• CASE STUDY: Transformative Evaluation case study: Evaluation of Student Colleagues scheme (PDF)
• THEORY OF CHANGE: Theory of Change for Plymouth Marjon University employability intervention: Student Colleagues (SC) scheme (PDF)

University Centre Leeds

• REPORT: Gypsy, Roma, Traveller: community outreach programme (PDF)
• CASE STUDY: Realist Evaluation case study: Evaluation of Gypsy, Roma, Traveller Community Outreach Programme (PDF)
• THEORY OF CHANGE: Theory of Change for University Centre Leeds access intervention: Gypsy, Roma, Traveller community outreach programme (PDF)

University of Suffolk

• REPORT: Small ‘n’ Evaluation: Micro-placements: A Realist Evaluation (PDF)
• CASE STUDY: Realist Evaluation case study: Evaluation of micro-placements (PDF)
• THEORY OF CHANGE: Theory of Change for University of Suffolk employability intervention (PDF)

University of Leeds, Lifelong Learning Centre

• REPORT: A small-n study using Qualitative Comparative Analysis (QCA) to understand the combination of factors that contribute to a rise in self-reported confidence in participants on a non-accredited pre-entry programme (PDF)
• CASE STUDY: Qualitative Comparative Analysis case study: Evaluation of Jumpstart (PDF)
• THEORY OF CHANGE: Theory of Change for University of Leeds, Lifelong Learning Centre access intervention: Jumpstart  (PDF)