Implementation and evaluation is more likely to be successful when the programme of work is thoroughly understood and the evaluation is planned in tandem. TASO’s new approach to commissioning is designed to do just that.
As outlined by our CEO Dr Omar Khan, in Six years on: What we’ve learned and where we’re going, TASO is developing a new commissioning model for our future evaluations. We’ve developed the new model in response to our experience of overseeing 14 randomised controlled trials, 13 quasi-experimental designs and nine theory-based evaluations.
From these trials we have learned that programme evaluation is most likely to be effective when the following ingredients are present:
- The issue the programme is addressing is understood and evidence-based
- The activities are defined in detail
- The causal pathways are are clearly articulated
- The outcome measures are collected and complete
- The implementation of the activities is considered in tandem with the development of an evaluation plan
Instead of evaluating each intervention after a brief inception period, we will carry out a thorough scoping of the proposal to help the institution shape it and make it evaluation ready.
TASO staff will run a series of workshops with providers to assess the feasibility of evaluating their programme of work, with the aim of taking the most feasible projects to evaluation.
It all begins with theory of change
Theory of change is the foundation of TASO’s approach to evaluation. And we have developed our thinking even further with our Enhanced Theory of Change model and our Theory of Change Builder tool – which provides the space to thoroughly scope out the programme of work and consider how it would be evaluated. But what will the result of this scoping look like in practice?
Understanding the issues
If the issue isn’t properly understood there’s a good chance the programme of work that is designed to resolve it isn’t ideal either. What is the evidence that the problem or context you’ve identified exists? If your institutional data finds an equality gap in first-year attainment between two student groups, do you dig further to thoroughly understand the context of the gap?; do you check if this is across all courses?; do you know if it started before enrolment? Understanding the context helps determine if the activities will plausibly lead to the desired outcomes.
The activities are defined in detail
It’s easier to develop strategies to address inequalities than to deliver them. It’s not uncommon for there to be a gap between specification and implementation. Perhaps the specification is not feasible to implement; maybe there is a gap in understanding between those who designed the programme and those on the ground; or there could be increased levels of staff turnover due to the financial situation in higher education. Even with successful implementation there is no guarantee of success if students are not engaging or the targeting has been ineffective and those who would most benefit are missed.
By having a detailed definition of the activities we will help providers to identify implementation risks and develop mitigations.
The causal pathways are are clearly articulated
Perhaps the most important part of a theory of change is the causal pathways. If they don’t clearly and plausibly link the activities and outcomes, and if assumptions have been missed, the chances of achieving successful outcomes are much reduced. At this point it may mean that the programme needs some redevelopment to ensure it achieves its aims.
The outcome measures are collected and complete
Many providers are still working towards centralised data collection systems. Sometimes attendance data is only collected in a subset of the institution. Or, the group of students you are interested in helping the most is not even recorded in the institutional administrative system – such as those who received contextual offers. We will look to see what information is available in the data, what is missing, and what we can realistically measure. This will help us focus on achievable goals and increase the chances of a successful evaluation.
The programme of work is developed in tandem with an evaluation plan
Successful quantitative causal evaluation is critically dependent on how a programme is rolled out to the institution. For example, if it is not possible to run a randomised controlled trial, then partial rollout to selected departments may enable a quasi-experimental design. This may lead to a tension between evaluation and implementation.
We will analyse outcome data prior to the planned implementation – for example, to establish the existence of parallel trends for the difference-in-differences method. This will help the team select more suitable departments for initial implementation, which in turn will improve the likelihood of robust evaluation.
What’s next
TASO will be implementing this new scoping model with the successful providers from our newest commissioning round. We will help providers test the feasibility of trialling aspects of their financial support schemes with the aim of taking the most feasibly evaluable to trial. Please join us.