Measuring impact along the path to raising attainment

Attainment-raising outreach is high on the agenda as higher education providers (HEPs) respond to latest guidance from the Office for Students (OfS). To help HEPs evaluate these interventions, we outline the best ways to understand if they are having the desired impact.

How do you measure the impact of attainment-raising interventions? 

Broadly, there are two ways in which we can measure attainment-related outcomes:

1. Directly – by observing grade outcomes.
2. Indirectly – by observing what are known as intermediate outcomes – measures for identifying progress along the path to your end goal. An intermediate outcome is the behaviour, attitude, knowledge, or skill that an intervention aims to influence and can act as proxy measures for attainment.

Measuring grade outcomes is meaningful, valuable and should be encouraged – it is also becoming easier with the recent launch of the Higher Education Access Tracker’s (HEAT) launch of their Key Stage 4 Attainment Tracking Dashboard.

However, only measuring grade outcomes would be limiting for a couple of reasons:

1. We often need results more quickly than we can access grade data.
2. There is nuance to the ‘active ingredients’ that may cause a change in a learners’ attainment and only measuring grades wouldn’t provide us with the full picture or allow us to improve the design of interventions to be more impactful.

How can measuring intermediate outcomes help? 

Measuring intermediate outcomes can help you observe the effect of interventions on students in the short-term.

We know that several key intermediate outcomes are associated with both progression to, and success at, HE. TASO’s recently published rapid evidence review provides a summary of 15 psychosocial intermediate outcomes, how they are measured and the extent to which they are associated with attainment and higher education access and success outcomes.

There following intermediate outcomes are positively associated with attainment:

• Academic self-efficacy
• Study strategies (a metacognitive outcome)
• Critical thinking / critical engagement
• Locus of control

There are also some outcomes that appear to indirectly influence attainment, primarily through improving an individuals’ belief in their capacity to achieve their goal – known as self-efficacy:

• Sense of belonging
• University expectations and knowledge
• Goal orientation

With robust, appropriate tools to measure these intermediate outcomes, we will be better able to evaluate our attainment-raising outreach activity.

How do you measure intermediate outcomes?  

Once you have identified which intermediate outcomes your attainment-raising outreach aims to impact, there are several resources that can help you identify the appropriate measurement tool:

TASO’s Widening Participation (WP) Questionnaire

• The TASO WP Questionnaire includes four scales suitable for measuring intermediate outcomes with learners in schools, sixth-forms, colleges, or young people not in education (Year 7 through to Year 13):
  
  – Academic self-efficacy
  
  – Study strategies (a metacognitive strategy)
  
  – Sense of belonging (prospective)
  
  – University expectations and knowledge

Existing questionnaires

The summary table in TASO’s rapid evidence review identifies pre-existing questionnaire scales for each intermediate outcome.

Education Endowment Foundation’s (EEF) SPECTRUM database

• The EEF’s SPECTRUM Database is a very useful systematic tool review with an accompanying online database of measures and an associated guide for their selection, deployment and interpretation. Unlike the TASO WP Questionnaire and rapid review, the SPECTRUM database has not been developed specifically for use within WP outreach. However, it covers a broad range of skills and attributes encapsulated within the field and is a valuable tool for WP evaluators.

How to measure impact without a validated tool? 

It is preferable, although not always possible, to use an existing, validated questionnaire scale. This is because the design process followed to produce these scales ensures the tool is valid (it measures the outcome it was designed to measure) and reliable (it consistently measures the correct outcome).

When you can’t access an ‘out of the box’ scale you can:

• Make modest changes to the wording and structure of existing scales to improve their usability in your context.
• Consider developing bespoke, validated, scales specifically for the constructs you are interested in. TASO has produced some simple guidance on how to do this and will be publishing a more detailed guide on how to design and validate your own scales in 2023.