New report: piloting methods to develop better evidence on student support

New report explores the effectiveness of less commonly used evaluation methods to develop more robust evidence and understand the impact of interventions.

Mental health and wellbeing

Previous work commissioned by TASO found gaps in the evidence on what works to improve the employment and employability outcomes for students from disadvantaged and underrepresented backgrounds, and what works to better the outcomes for disabled students.

The aim of the Efficacy Pilots project was to build Type 2 evidence for interventions that support these outcomes, and scope out the feasibility of Type 3 evaluation. Four higher education providers (HEPs) were matched with an independent evaluator to provide evaluation support and build internal evaluation expertise.

Lessons learnt about evaluation methods

Propensity Score Matching (PSM)

PSM is a quasi experimental method which matches people who engaged with an intervention (a treatment group) with a group of people who did not engage (a comparator group). The groups are matched based on how likely the people are to end up in the treatment group (their ‘propensity score’).
Three of the evaluations piloted PSM as a method for generating Type 3 evidence. These evaluations offer valuable case studies of how this method can be applied to evaluate the efficacy of interventions designed to improve outcomes for these groups of students.
The key strengths of these examples is that they go further than previous evaluations in taking into account pre-existing differences between students who do, and don’t, take part in interventions.
The key limitation of PSM is that it is only able to match students based on observable characteristics recorded in available datasets. Differences in unobservable characteristics – such as motivation, are unaccounted for in the match, meaning the method cannot entirely account for the potential selection bias in relation to participation in the intervention.

Contribution Analysis

Contribution Analysis explores whether impact can be attributed to an intervention through assessing the contribution that an intervention is making to observed results.
This analysis entails a series of steps to assemble and assess evidence against the intervention Theory of Change.
The University of Exeter Analysis Report provides a worked example of a Contribution Analysis. It demonstrates the framework which this method provides in triangulating different sorts of quantitative and qualitative evidence to assess whether an intervention’s Theory of Change holds true.
This method is well suited to unpicking the complexity of how interventions might be linked to outcomes, but this example has not provided evidence which constitutes Type 3 causal evidence.

Each of the Analysis Reports contains reflections on alternative evaluation methods which could be used to address the emerging issue from the pilots on the inability of the methods to account for selection bias.

Recommendations on evaluation methods for higher education providers:

Propensity score matching (PSM) offers some promise as an evaluation method which can help the sector move towards a better understanding of causal impact. However, sample size is a key consideration. Early estimations of sample sizes and preliminary calculations are needed to ensure the analysis will be informative for the intervention in question.
Explore methods discussed in the report to strengthen evaluation practice.
Invest in further evaluation capacity to facilitate stronger evaluation practice across their Access and Participation portfolio to ensure they are providing students with the best possible support.
Larger providers should seek assistance from academic colleagues if internal evaluation teams lack the capacity to deliver analysis of this sort.
Work to improve internal data linking and accessibility to facilitate analysis of institutional data using a range of methods.
Explore the ethical use of participant compensation in their evaluations to improve rates of data collection. TASO provides advice on this topic in our Research Ethics Guidance.

The University of Brighton and the University of Exeter worked with independent evaluator SQW. The University of Central Lancashire and the London School of Economics worked with an independent evaluator from the University of Cambridge.

Read the summary report: Piloting methods to develop better evidence on student support.