Evaluating the access to postgraduate research for students from Black, Asian, and minority ethnic groups

There are a range of racial inequalities experienced by students from black, Asian, and minority ethnic groups in higher education (HE). These inequalities are evident not only at the undergraduate level, but also at the postgraduate level, demonstrated by the lower representation of students from black, Asian, and minority ethnic groups in postgraduate research (PGR). The HE sector has developed various responses to address this gap, but so far these have not been effectively evaluated.

In response to this context, in October 2020, the Office for Students (OfS) and Research England (RE) launched a funding programme which provided nearly £8 million in funding to 13 projects to improve access and participation for black, Asian and minority ethnic students in PGR. The Evidence Development and Incubation Team (EDIT) at King’s College London, partnered with the Centre for Transforming Access and Student Outcomes in Higher Education (TASO) have been appointed as external evaluators to assess how effective the overall programme and projects are at achieving their aims and priorities.

This blog highlights the various interventions carried out by the projects, sets out our evaluation methodology, and outlines the progress made during the first year of the evaluation.

What interventions are the funded projects running?

The projects have been broadly funded to improve the following: access into research, the research environment for students from black, Asian, and minority ethnic groups, and their progression into a range of careers.

One of the first steps of the programme evaluation was analysing the interventions carried out by the projects to draw out any commonalities between them. Based on this analysis, we were able to highlight mentorship schemes, research opportunities, PGR training, and changes to the admissions systems as the main interventions identified by most projects.

However there are a few others with distinctive agendas such as the project led by the University of Cambridge and the University of Oxford which identified changes to the admissions process as its main focus. Also, the University of East London plans to develop a Master of Research (MRes) programme, while the University of Sheffield has launched a Centre for Equity and Inclusion. In terms of target groups, most projects broadly have black, Asian, and minority ethnic students as their population of interest. However, some others focus specifically on black students. e.g., the University of Surrey and the University of Essex. A comprehensive overview of activities by all 13 projects is detailed here.

The impact evaluation methodology

Given that the successful projects were selected through a competitive funding competition, an experimental approach was not possible as that would involve randomly allocating projects into an intervention group and a comparator group. We instead adopted a quasi-experimental approach to the evaluation which still allows us to observe the counterfactual i.e., what would have happened in the absence of the programme, without requiring randomisation.

We decided to use a matched difference-in-difference (DiD) approach to ensure that our comparator group included higher education providers (HEPs) that were as similar as possible to the HEPs involved in the programme (i.e., the intervention institutions) based on institutional characteristics like size, type, and mission group.

Baselining the outcome measures

A thorough review of the logic models developed by each project led us to identify five outcome measures that were common across all projects. These are the rates of progression, retention, completion, and post-PGR study for students from black, Asian, and minority ethnic groups, as well a measure of wellbeing, using data obtained from the Higher Education Statistics Agency (HESA) and the National Student Survey (NSS).

A condition of the DiD approach is that the trends in the identified outcome measures prior to the intervention are parallel for both the intervention and comparator groups. The idea being that if the pre-intervention trends are similar, we can expect that this would continue in the absence of the intervention. Therefore, any changes in outcome trends that occur after the intervention has been implemented can be safely attributed to the intervention.

Our analysis of pre-intervention trends (between 2017/18 and 2020/21) confirmed that all five outcome measures followed the same trends in the intervention and comparator groups. Following this, we have also established a baseline for each outcome measure to be used as a benchmark for analysing the impact of the programme over time.

The next twelve months…

In the next twelve months, the impact evaluation will focus on data collection and analysis of each outcome measure to estimate the impact of the programme in the first year, while the qualitative research will focus on administering student and staff surveys and conducting focus groups and interviews for select projects.

In summary

Understanding better how we can effectively tackle the PGR gap to enhance outcomes for students from black, Asian, and minority ethnic groups will serve to improve fairness and equality in the UK HE sector. There are also wider social and economic benefits: by tapping into the full talents of our society we improve the quality of research and innovation in the UK economy and contribute to our international success.

Download the annual evaluation report: Improving access and participation for black, Asian and minority ethnic groups in postgraduate research: Evaluation