Conducting meaningful evaluation: A case study using TASO’s Monitoring and Evaluation Framework
Defining the purpose of evaluation can be a tricky business (Wanzer, 2013; Mathison, 2008; Harvey, 2010) and in a world of outcomes, outputs and impact, outreach practitioners and evaluators are often attempting to achieve the same goals but using different approaches to get there.
For the Southern Universities Network (SUN), a Uni Connect partnership in the South Coast of England, evaluation has been approached using a collaborative lens with the help of TASO’s Monitoring and Evaluation Framework (MEF), ensuring evaluation activity is defined, meaningful and impactful to key stakeholders.
Here we outline an example of such an approach, illustrating how one-to-one student support aims to increase the likelihood of students applying to higher education (HE). This activity was delivered by SUN Progression Mentors who are employed by target SUN Further Education and Sixth Form Colleges to provide programmes of support for Uni Connect target learners.
Diagnose
We began by assessing the current body of literature and pre-existing knowledge that links mentoring to increased chances of progression to HE for under-represented groups. TASO’s evidence toolkit was a good place to start.
During this stage, we discovered that many practitioners’ understanding of ‘what works’ is held as tacit knowledge (Leguene, 2011). There was a frustration from practitioners that this knowledge wasn’t easily accessible and shared. We used this finding as the basis for our evaluation planning and Theory of Change development – to ensure that transparency and communication were at the heart of our evaluation methods and approach.
“The findings from the project have helped me to further structure my support for students to ensure they were beneficial to individuals. It’s also helped me to demonstrate to internal stakeholders, and the students themselves, how beneficial this targeted, bespoke support can be.”
SUN Progression Mentor, Highbury College
Plan
Developing our Theory of Change with practitioners provided a reflective space for us to identify and agree short, medium and long term objectives. We also discussed the pros and cons to various evaluation methods enabling us to choose methods that were fit for purpose.
The benefit of this collaborative approach was the timing of surveys and qualitative data collection worked alongside students’ assessment timetables ensuring higher response rates to surveys and focus groups. It also helped us identify touch points for briefings and communication with SUN staff based in target Further Education (FE) and Sixth-Form Colleges. This regular communication enabled us to adapt evaluation approaches quickly in relation to any unforeseen challenges such as Ofsted inspections or staff changes within colleges.
“We could draw upon the knowledge and close working relationships of SUN Progression Mentors with their students to co-produce evaluation resources, meaning that this was constructed in a way that encouraged target students to engage and respond.”
Further Education Project Leader, Southern Universities Network
Measure
The TASO MEF provided a guide for a consistent approach to analysing the quantitative data. It also provided transparency to our stakeholders in how we would be using the data collected as part of the project.
As the evaluation of outreach interventions remains a live conversation within the sector, following prescriptive methods has ensured a level of robustness for this project. By using sector standard resources, we can provide quality assurance in the way we are analysing, interpreting, and presenting the data. In turn, the project team has been able to confidently respond to questions about the validity of findings.
“The TASO framework provided a systematic evaluation process to follow. This transparency, combined with our ‘live’ updates to the wider team meant evaluation was a continuing process. Evaluation was embedded within the project.”
Naomi Clements, Senior Teaching Fellow in Academic Practice (Evaluation)
Reflect
Working collaboratively whilst following the TASO MEF has strengthened the link between practitioners and evaluators within the SUN partnership and enabled the team to identify clear next steps for the intervention and how best to evaluate within the FE context.
Practitioners feel more confident in evaluation and approaching future projects and the evaluation team now have invaluable insights into the challenges and opportunities of evaluating within an FE context.
“As a result of participating in the project, the SUN FE team has demonstrated increased confidence to embed evaluation practices into their project work. This has enabled a clearer focus on best practices in delivery for the wider SUN Progression Mentor team.”
Sinead Holmes, SUN Programme Manager
The TASO MEF has been a valuable resource for the SUN Uni Connect Partnership. It has supported evaluators and practitioners to work collaboratively to understand what works, why and for whom in the complex world of further education, outreach, and HE progression.