27 January 2021

We need more evaluation to optimise support for CSC-experienced learners

Evaluation is about understanding why, how and if a programme is working as intended. Digging into the assumptions behind a programme is key to this process, so what happens when those assumptions don’t match reality?

Belonging in HE (post-entry)
Progression to HE
Wellbeing
Pre-entry to HE
Post-entry to HE
Care leavers

This week we publish a new report focused on learners with experience of children’s social care (CSC). We set out to understand what the existing evidence base could tell us about how to support this group into, and through, higher education. The report is rich with findings of the barriers faced by these learners and activities to support them, but one piece of research provides a particularly interesting case study for TASO.

Mooney et al. (2016) evaluated a popular and well-established book-gifting programme in Northern Ireland. The model was simple – children in foster care received monthly parcels of books with the aim of improving literacy skills. To test whether the programme was effective, the authors ran a randomised controlled trial (RCT), randomly varying whether or not children received the books. By running tests with both groups of children, they measured whether there were any differences in literacy skills, and from that measured the impact of the programme.

The authors found no evidence that the programme had any effect. It’s important to note that they were working with a small sample and so results should be handled with caution. But nonetheless, the findings do not provide even suggestive evidence that the programme was working. A disappointing result.

But the story doesn’t end here. The trial was accompanied by qualitative research. In this part of the study, the authors took a step back and asked about the design of the programme, why it might be expected to work, and how children felt about receiving the books.

They discovered it was underpinned by some key assumptions: first, that the children in foster care did not have easy access to books and would be ‘book hungry’ and second, that children would be inherently ‘book grateful’ and, upon receiving books, would naturally read them without extra encouragement from their carers.

The authors found these assumptions to be outdated and not supported by the facts. First and foremost, many of the children had easy access to books. And the extent to which children were inclined to engage with books hinged on their pre-existing interest and reading ability – so the books were less likely to help those children who needed it the most.

It’s not a surprise then that impact was lacking. In this case, the authors identified a clear missing link in terms of foster carers taking a more active role in helping children engage with the books (an element which was completely absent from the original programme). In the words of the authors “this popular, attractive and well-loved intervention [stood] to make greater impact…if underpinned by a clear theory of change”. In other words, the design needed to be based on a better understanding of the needs and motivations of the children.

I take two key messages away from this research. First, this is a nice case study of how two different methods gave us two parts of the picture – the trial told us that the programme wasn’t working as it should be and the qualitative research gave us clues on how to address this. This mixing of methods is something which TASO will try to bake into its evaluation work over the coming years.

But second, this study is a reminder that even the most popular, well-loved, and well-established, programmes can benefit from evaluation. As well as seeking to establish the impact of such programmes, evaluation should help interrogate the assumptions behind design and delivery, to identify whether and how we are best to support learners.

Our report highlights a hugely committed community of individuals across the higher education sector, schools, charities and local authorities, all striving to do the best they can by CSC-experienced learners. I would encourage all of those individuals to extend their commitment to reviewing the assumptions behind their work and developing more and better evaluation to support these efforts. Because, as demonstrated in this blog, even the most well-established programmes require a critical eye to ensure we are providing CSC-experienced learners with the support which they need, want and deserve.

Download the full report: Supporting access and student success for learners with experience of children’s social care