How can you measure the impact of mentoring? A reflection on TASO’s mentoring report findings
Mentoring is a crucial component of many multi-intervention outreach programmes. These programmes aim to support students by giving them tailored advice and guidance, usually over several years, using a combination of interventions such as skills sessions and summer schools.
However, building the evidence on the impact of mentoring can be challenging as programmes differ so much in mode of delivery, number of sessions, or type of mentor. Consequently, there is a clear need to evaluate programmes locally. As part of TASO’s evaluation of multi-intervention outreach, a specific focus was placed on mentoring delivered through Brightside at two of the partner higher education providers. The resulting report provides valuable insights into how to creatively and effectively use the data collected from mentoring programmes.
Brightside has existed as an online mentoring charity for 20 years and along the way we’ve developed a rigorous approach to data analysis that allows us to continually improve our programmes. We have access to huge amounts of information, including engagement patterns, survey data, and every message exchanged between mentors and mentees. Analysing this data over the last 20 years has given us incredible insights, but there’s always more to learn and we’re grateful to TASO’s report for giving us new things to think about.
Our three key takeaways from the report:
Quantitative analysis is essential, but what is ‘good’ engagement?
Monitoring engagement throughout a mentoring programme can help identify patterns and tell us who is not messaging, allowing us to re-engage them through targeted communications or change aspects of delivery for future projects. But how do you decide what constitutes ‘good’ engagement? TASO’s mentoring report cautioned against only using the number of messages to evaluate engagement on a mentoring programme, as this measure didn’t always align with the number of days active. Since mentoring is personalised, there is no ‘bad’ way to engage – everyone does it differently – so there is no singular metric that can tell the whole story. It’s important to look at multiple measures, such as number of messages, days active, or word count to get a more holistic picture of engagement. Whilst we analyse engagement holistically in our reporting, we are still working on finding a benchmark for what ‘good’ engagement looks like.
Qualitative evidence supplements quantitative analysis
Looking at the content of messages can tell us if we need to embed different resources or topics into a programme so that we can structure the content around what students are most interested in. Messages were categorised in different ways during TASO’s project, looking at the nature of the messages and the associated topics, which helped flesh out the story of engagement. One particularly interesting finding looked at coding messages as requests, responses, or both, showing that mentees are more likely to respond to a message than request information, which has key implications for how mentoring may be effectively structured.
Mentors are key to mentoring!
Whilst this sounds obvious, the report suggests that mentoring programmes benefit from structure; providing mentors with topics to cover and following up to ensure they are meeting their commitments are crucial to ensuring mentees continue to meaningfully engage. Since mentees are more likely to respond to their mentor than reach out themselves, mentors need to make the effort to enable engagement. This finding is in-line with Brightside’s mentor guidance and training but reiterates the crucial point that mentees need to be actively encouraged to get the most out of mentoring.
So what next?
The publication of this report was timely for us as we are currently at the tail end of our outcomes review, which we undertook to bring our Theory of Change outcomes more in-line with current internal and external evidence. The report therefore provides us with a great opportunity to review how we evaluate and benchmark all of our quantitative engagement, impact, and quality data so that it tells us what we need to know.
We are also looking to renew our resources for partners in the next year so that it’s easier for our partner organisations to make sense of their own qualitative and quantitative data with our support. TASO’s report has shown us that we’re already doing a lot right in terms of data collection and analysis, but it’s also given us excellent examples of things we can do differently in the future.
We truly believe that mentoring has the power to change the lives of young people and it’s great to learn new ways of evidencing that impact.
Visit our website to learn more about our impact approach.
Download TASO’s summary report: Evaluating multi-intervention outreach and mentoring programmes.