Decentralising APP Evaluation: Lessons in collaboration and governance

Why change was needed

Evaluating University of Staffordshire’s access and participation plan (APP) has strengthened collaboration, knowledge exchange and participatory approaches among practitioners and evaluators at the institution. This work has taken place amid sector-wide pressures, including financial instability, recruitment challenges and shifting policy headwinds. In 2025, the Office for Students (OfS) reported a third consecutive annual decline in sector finances, with concerning implications for institutional resources. Against this backdrop, building evaluation capacity and deepening collaboration with those delivering APP activities has been critical to our evaluation approach.

What we did

Setting the foundations

As a “wave 1” APP provider, we, the Inclusive Education and Impact team, developed our outcomes and targets in 2024 with our Business Intelligence and Management Reporting team. Using a risk-focused approach, we reviewed our performance across the OfS’ four lifecycle stages (access, continuation, attainment and progression) baselining indices of multiple deprivation (IMD), ethnicity, free school meal (FSM) status, disability and age gaps.

Table 1. University of Staffordshire APP 2023/24 – 2027/28 targets

This analysis informed the development of 16 APP activities organised across four strands:

• Access: outreach delivered with Higher Horizons to address barriers for FSM-eligible, Asian and Black learners.
• Transitions: pre-entry support to improve readiness for higher education, prioritising IMD quintile 1 and Asian students.
• Curriculum: inclusive pedagogy and employability initiatives to reduce on-course disparities, particularly for Asian and Black students.
• Student support: redesigned services covering financial, wellbeing and skills-based support.     

To ensure a shared understanding of success, we ran theory of change workshops using Advance HE’s ChangeBusters methodology. Bringing delivery teams together early created ownership, clarified outcomes and established realistic expectations around evaluation from the outset.

Co-designing evaluation

We applied the same collaborative principles to evaluation design. Working with activity leads, we developed 16 surveys, six focus‑group guides, four activity monitoring logs, and plans for Type 3 causal analyses.

Type 3 analyses enable us to assess causal impact. Activity-tracking logs record delivery fidelity, reach and intensity, while pre‑ and post‑activity surveys measure changes in knowledge, skills and confidence. Focus groups and interviews then help contextualise and validate those findings. Mixed methods triangulation increases the validity of evaluation findings and, practically, ensures we have a holistic and critical understanding of what is happening within our APP.

All evaluation tools were co-produced through workshops mapping outcomes to evaluation questions, followed by iterative refinement and planning for recruitment, data collection, analysis and reporting. Grounding evaluation around the expertise and experience of activity leads ensured there was clarity around expectations and deadlines despite growing demands on staff time.     

Governing and decentralising

Clear governance structures have been critical to sustaining APP evaluation during a period of financial pressure and organisational change. We developed project plans, activity trackers, reporting templates and data collection tools to support delivery teams. Surveys are managed centrally through Qualtrics to ensure GDPR compliance, while guidance and training are provided for qualitative research.

A structured governance model supports our decentralised evaluation. The three groups bring together key stakeholders across our team, activity leads, student representatives, senior leaders, academic schools, and professional services. Regular, standardised reporting through our Operational Group supports monitoring of delivery fidelity and risk, while strategic oversight by our Project Board assesses impact, ensures alignment with institutional priorities and facilitates evidence-led decision-making, such as adaptation, merging or discontinuation of activities. Annual and summative reports are reviewed by the Academic Board before publication.

Figure 1. University of Staffordshire APP reporting flow     

Building evaluation capacity

Activity leads are supported to evaluate their own programmes, with quality assurance provided by our team. We provide an asynchronous evaluation training module alongside standardised templates and guidance, enabling activity leads to take greater ownership of evaluation while maintaining consistency and quality.

Student involvement has also been key. Through our ‘Evaluation Assistants’ programme, learners with an interest in research support data collection and evaluation activities, while art students create comics that accessibly communicate evaluation findings to a range of audiences. Prioritising student co-creation means that students’ experiences are genuinely valued. It also creates opportunities for skills development and employability while empowering students to shape and decentralise the evaluation process.

Activity Spotlight: Step Up to HE

‘Step Up to HE’ is a transitions programme supporting students from marginalised backgrounds to prepare for academic study and life in higher education. Evaluation design was iterative and collaborative and included post‑module surveys and end‑of‑programme interviews assessing changes in higher education readiness, confidence and skills.

Fifty-five students participated across three cohorts in 2024/25. Delivery was largely stable, but evaluation proved challenging due to low engagement. In response, teams adapted by prioritising interviews over focus groups, boosting survey promotion and incorporating written feedback.

Although response rates were lower than anticipated, respondents reported improvements across all measured outcomes, with 96% of respondents reporting satisfaction and 90% completing a Level 3 certificate.

Figure 2. University of Staffordshire APP 2024/25 Transitions impact comic

What we learned     

Following APP approval, significant financial pressures and organisational change have rendered a fully centralised APP evaluation model unsustainable. Our experience decentralising evaluation activities has proven effective but challenging, with heavy reliance on strong governance, clear processes and ongoing skills development needed. 

Relationship-building, standardisation and meaningful student involvement from inception have helped build evaluation literacy, strengthen ownership among delivery teams and create a broader institutional evaluative culture. A further benefit of this necessary shift in approach is its potential to shape how University of Staffordshire measures institutional success in future.