ux-indonesia-8mikJ83LmSQ-unsplash

TASO has commissioned two separate organisations, National Education Opportunities Network (NEON) and Manchester Metropolitan University (MMU), to undertake the project, which aims to support the higher education (HE) sector in understanding how best to evaluate impact when dealing with smaller student cohorts.

At the heart of impact evaluation is the requirement to link cause and effect, to explain how and why activities lead to changes in outcomes of interest. While experimental and quasi-experimental evaluation methods can provide the sector with the confidence required to be able to state causal inference, it’s not always possible to use these methods. Small – or specialist – providers face issues when attempting to generate causal impact evidence, and even in larger institutions, existing methods can be difficult to use with smaller cohorts of students, for example, when evaluating highly targeted support.

This project has been split into three phases which will be managed by the respective organisations – the consultation phase, the methodology phase and the translate and test phase

 

  1. Evaluation with small cohorts: consultation phase and translation and test phase:

NEON has been awarded two distinct phases of this project – the consultation phase and the translation and test phase. They will also develop evaluation guidance to be disseminated across widening participation teams within the HE sector.

Activities that will be included within the scope of this project:

  • NEON will consult with relevant stakeholders to inform the project, ensuring that the content, format, and structure of the final report and guidance is fit for purpose. The stakeholders consulted will include: What Works Centres, the Association of Colleges, small and specialist providers, evaluators, academics and the TASO Advisory Groups.
  • NEON will pilot small-n evaluation methods with small and specialist HE providers, to include:
  • At least 3 – 5 small providers.
  • At least 3 – 5 larger providers working with small cohorts.
  • The pilot phase is to include a collaborative approach, pooling resources from separate providers to test collaborative working as a solution to the resource challenge often experienced when working with small cohorts.
  • NEON will test the feasibility of providing transferable methodological practice guidance to multiple specialist providers with a unique focus.
  • NEON will design and deliver practical guidance for HE providers, illustrating how to operationalise findings from the translation and testing phase.

“NEON is delighted to lead on this crucial project for TASO. It is vital that where there are smaller groups of learners, we work hard to understand how they can be supported to enter HE. This project will provide the tools that enable HE providers to make their work with these learners as effective as possible.”

Dr Graeme Atherton, Head, Centre for Levelling Up and Director, National Education Opportunities Network

 

2. Evaluation with small cohorts: methods guidance

MMU has been awarded the methodology phase of the project. They will review and select the most suitable evaluation designs and methods for impact evaluation with small cohorts.

Activities that will be included within the scope of this project:

  • MMU will conduct a review of the existing methods used to evaluate programmes/interventions with small cohorts.
  • The review will consider a range of methods used to evaluate small cohorts, including guidance on mixing methods and with a particular focus on methods that support causal inference where possible.
  • MMU will develop a methodology taxonomy and transfer knowledge from this methods guidance phase to NEON, who will be translating and testing the methods guidance.

“We are delighted to be working on this exciting project commissioned by TASO. Small-n impact evaluations have the potential to provide a rigorous approach to evaluating programmes that aim to improve access and outcomes in higher education, an area particularly close to our heart at Manchester Metropolitan University. We are bringing together experts in evaluation with an in-depth understanding of causation and impact and experts in higher education. Together we will produce a guide to small-n evaluation approaches set in the higher education context. We look forward to working with NEON who will implement it in the sector and believe that this project will help shape future evaluations in higher education.”

Dr Jessica Ozan, Research Associate, Policy Evaluation and Research, University Manchester Metropolitan University

 

“TASO is thrilled to be working with NEON and MMU on this project. The Higher Education sector needs to identify a broader selection of impact evaluation methodologies that are pragmatic enough to be adopted by a wide audience. This includes small and specialist providers, while maintaining the rigour that gives providers the confidence to make bold statements about what works. Therefore, TASO supports this project to address the difficult methodological and theoretical challenges faced by HE Providers evaluating impact when dealing with smaller student cohorts.”

Dr Omar Khan, Director of TASO

 

Research partners were selected after an open call for partners. All applications were assessed by TASO Research and Evaluation Team.

Final deliverables for the overall project will be released in April 2022.