Professor Billy Wong, Director of Research and Evaluation, University of Reading and Lydia Fletcher, Research and Evaluation Manager, University of Reading

23 April 2024

Stay on track: Using live data as the GPS for implementation and process evaluation

As we finalise new guidance to support the sector in conducting better and more consistent implementation and process evaluation (IPE), we hear from colleagues Billy Wong and Lydia Fletcher, at the University of Reading, on how IPE can help us understand the journey to change, not just the final destination and outcomes.

Planning the journey

If you regularly travel by bus, you will know that your journey time is rarely consistent. On a great day when everything goes to plan, the bus ride might only take 15 minutes. On a bad day, with heavy traffic or cancellations, the same route could take an hour or more – same destination, vastly different journeys.

Similarly, when planning our access and participation work in higher education, we are confident about where we want to be and the gaps we wish to close or eliminate, but the journey there is not always straightforward.

The complex terrain of higher education

To plan our journeys we are encouraged to think about our theories of change. A theory of change can help us consider the context and situation, drawing out the aims and objectives (or destinations) of our initiative and how we can get there using what tools and resources. It makes sense.

However, the devil is in the detail. Knowing the destination is a good start, but the complexity of the journey – the varying conditions and unexpected challenges – mirrors the dynamic landscape of higher education.

Even with a clear destination (for example, enrolment in higher education, closing the ethnicity degree awarding gap, positive graduate outcomes), planning the journey requires considerable work. Navigating this landscape involves more than just a linear route, it is about understanding the nuances and adapting to the changing educational terrain. Taking the bus can be straightforward if the bus turns up and proceeds as planned. But this straightforward route is rarely the case in the complex reality of access and participation work. Despite external forces causing delays or detours, we still aim to reach our destination (or objective) on time, or not too late.

Alternative routes

Our journey in higher education is not merely about reaching a destination – it’s about understanding and navigating the pathways that lead us there. The destination is important, of course, and impact evaluation often dominates the current evaluation discourse of access and participation work. But we also want to consider the main turns and stops, alternative routes or even different modes of transport. This means we need to be proactive rather than reactive, ready to adapt to unforeseen changes, much like a skilled and knowledgeable driver navigating a busy and congested town centre. Knowing the direction of travel is useful but it may not take us to the planned destination.

We need to be careful that our educational journey is not reliant on our assumptions about the route. It is of course useful to have local and experiential knowledge of shortcuts or specific routes, but we ought to be mindful that we do not travel without a real-time navigation of our progress. In other words, we don’t want to travel based on a vague idea of the direction, expecting to arrive at the right point, based on previous experiences or intuition alone.

Live data and decisions

Just as a GPS provides real-time updates and alternative routes, ongoing data collection in educational programmes ensures we are on the right path and allows for course corrections based on immediate feedback. In access and participation work, we must be prepared to adapt our strategies and methods.

To appreciate the journey, our emphasis should not be a sole focus on the distance or time but to consider the quality of the journey and the lessons learned along the way. We need to be open to different routes, understanding that sometimes we just need to drive down a longer but perhaps more scenic route to offer the most enriching and memorable experiences. So, whether that means incorporating new technologies or revising outdated curricula, our ability to pivot and embrace change is key.

While a theory of change can usefully provide a logical map to reach a particular destination (or objective), like any map, a theory of change – even the enhanced version – can only cover so much terrain and detail. It cannot foresee the weather and changing conditions of our journey, for which we need continuous and real-time guidance and data (akin to a GPS) to help us navigate our journey, especially through the fogs and mists.

Understanding the journey: Implementation and process evaluation

A reflective or iterative process evaluation can help to guide the journey , allowing us to understand what might be pushing us off course, especially for pilot or fluid programmes where changes and adaptations are anticipated.

Implementation and process evaluation involves gathering data on what is actually happening rather than just what is planned. For example, assessing how participants interacted or engaged with the activity, evaluating the environment, observing how practitioners delivered their activity, or determining the extent to which the activity deviated from the plan. Reviewing this implementation and process data regularly will help us to see if we are following the intended path. Gathering intermediate outcome data can then help us to build a picture of where we are along the route and identify any changes we might need to make to stay on track.

Evaluation is a journey and a destination

The journey matters as much as the destination and we need to ensure that our evaluation reflects this. By embracing a real-time evidence-based approach, we can ensure our journey is not only directed by logic and assumptions but is navigated by lived experience and evidence. This should offer us meaningful as well as measurable impacts, and ensure that we end up where we want to go, knowing how we got there.