Evaluating the everyday

Evaluation, evaluation, evaluation. Yes, we’ve heard it all before. But remember: it’s about ‘getting on’, as well as just ‘getting in’; student success as well as student access. And it isn’t just about evaluating the specific access and participation interventions – the everyday stuff is at least as important in helping us to reach targets and close gaps. Most of what universities do for students is not neatly compartmentalised and targeted, nor is it run by access and participation teams. It’s the everyday.

What are your students up to?

Take teaching for a start – important in a university we’re sure you’ll agree. We don’t just mean the specific project on decolonising the curriculum, or the intervention to support students with disabilities, important though they may be. Keep it simple and step back a bit: how students engage with their education. At Aston University, we’ve been looking at this over the past few years using learning analytics data. Seeing what goes on even without us intervening. It can tell you a fair bit about both what students are doing and about how the university is working more generally.

Early engagement matters

For example, we have found that engagement in the first three weeks of the first year predicts subsequent activity and also attainment at the end of the first year. This matters because it shows how important the transition period from school or college to university can be. More specifically, it suggests that students need to know what they are supposed to be doing right from the start with all that ‘spare’ time they have. University is different from school and requires much more independent study – are you sure all your students know this or is it possible that some groups are more aware than others? It seems likely that at least some of your students are waiting to be told what to do. Ensuring that everyone is on the same page with some explicit guidance could pay dividends in helping to close those stubborn attainment gaps.

A group of buildings gathered around a library 

What about library book borrowing? Pre-pandemic at least this was something that students from disadvantaged backgrounds at Aston University did more than their peers. Are these students working harder? Perhaps – but maybe also not able to afford their own copy of a textbook or don’t have a suitable device to read the electronic copy. We can’t determine the reasons behind the behaviour using learning analytics, but if a university is ‘a group of buildings gathered around a library’ (Foote, 1993), then we need to be certain that our libraries are working the best they can for all of our students.

And along came COVID

Our data has also shown that whilst pre-pandemic ( in-person teaching and assessment) students from more disadvantaged backgrounds had higher overall engagement with their programme, during COVID (online teaching and assessment) they showed more comparable engagement to their peers. Interestingly, the main differences in engagement were for recorded lectures, with the number of recorded lectures watched decreasing during online teaching for students from disadvantaged backgrounds whilst synchronously delivered online lectures showed little difference in engagement.

This all matters because this drop in engagement for students from disadvantaged backgrounds was also associated with both a lowering of attainment and a widening attainment gap (Figure 1) similar to that found in TASO’s report into online teaching and learning in the time of COVID. This means that how you teach and who you teach both matter if we want to close attainment gaps. When we are considering what the ‘new normal’ should entail in terms of a student’s educational experience, we should also consider potential impacts of different delivery methods on different groups of students. Overall, we need to understand better what makes both a good online experience as well as a good on-campus experience.

[[OLD IMAGE]]
So remember – don’t just evaluate the interventions and projects, see what you can take from the everyday.