On Thursday 21 September we will be publishing a new validated, multi-scale questionnaire to help the sector better evaluate its access and student success programmes and activities.

Much like the road to Amarillo, the path to validated questionnaire scales is a journey, and an important one at that. This particular journey started in January 2022 when TASO commissioned researchers from The Brilliant Club and the University of Cambridge to consult the higher education (HE) sector to understand the outcomes most salient in our work, to review existing scales and to design and validate a multi-scale questionnaire that can be used in the evaluation of access and student success programmes. The end of our journey is not far ahead, with the fully validated, multi-scale questionnaire coming your way on 21 September. For now, let us show you the pathways and avenues we’ve been exploring on this journey so far.

Intermediate outcomes 

It all begins with the recognition that measuring intermediate outcomes is a fundamental part of evaluating access and participation work as it provides us with early indicators of whether an intervention is effective. Intermediate outcomes include changes in behaviour, skills, and attitudes, and based on the research literature we have a good understanding of the outcomes that are strongly associated with school attainment, progression to HE, and success in HE. Measuring intermediate outcomes is recognised by the Office for Students as an appropriate and meaningful evaluation technique and TASO’s Mapping Outcomes and Activities Tool (MOAT) and Theory of Change resources provide support in how to identify and measure these outcomes.

Validation gap

It’s all very well knowing the intermediate outcomes that are associated with access and success in HE, however, if we don’t have the tools to meaningfully evaluate these outcomes within our interventions with our target students, we will never truly know the impact of the work that we do.

At present, there are a range of approaches being used to bridge this ‘validation gap’. For example, practitioners and evaluators are using their own questionnaires that they have created to measure their chosen intermediate outcome, or they are using items (the individual questions that make up a scale) based on existing questionnaires from the research literature. However, there are issues around the validity and reliability of using these scales, and therefore there has been movement in the sector towards a greater acknowledgement that using validated scales are a better way to measure these outcomes.

The purpose therefore for developing the ASQ is to help address the validation gap and provide the sector with a series of questionnaire scales that can be used with confidence when evaluating access and participation work. Critically, using consistent measures will also enable the sector to build up national data on intermediate outcomes, something that Higher Education Access Tracker (HEAT) have been supporting us with by embedding the questionnaire scales for their members to easily access.

We are delighted to see other researchers engaging in the validation process too – including the recently published Toolkit for Access and Participation Evaluation (TAPE) and also the on-going validation work by King’s College London.

Project update 

You may remember last year we published a draft version of the ASQ (previously named the widening participation questionnaire) to be piloted across the sector.

The draft version of the questionnaire was partially validated which meant that there was still one step in the validation process that needed to be completed before the final version of the questionnaire could be implemented as a validated tool to measure intermediate outcomes relevant to widening participation. This is why we needed to pilot the scale to collect and analyse data from the types of learners whom this scale will ultimately be used with.

Any changes we needed to make to the items following on from this analysis have now been made and the questionnaire that we’re publishing on 21 September will be fully-validated and ready for you to use.

Next steps

All you need to do now is sign up for the launch of the Access and Success Questionnaire (ASQ) on Thursday 21 September 2023. Sign up now.