News22 February 2024
How can learning analytics – data systems that help understand student engagement and learning – be used to identify students who may be at risk of withdrawing from their studies, or failing their courses, and what interventions work to re-engage students in their studies?
To explore this question, TASO commissioned two universities – Nottingham Trent University and Sheffield Hallam University – to carry out randomised controlled trials of different interventions prompted by learning analytics systems.
At each university, students identified as at-risk of withdrawing from their studies or failing their courses on the trial were randomly assigned to one of two groups. In one group, at-risk students received an email alert with details of available support. In the other group, the email was also followed by a support phone call from a student adviser.
Following the trials, neither university found a measurable difference in post-intervention engagement rating between at-risk students who received an email followed by a support phone call and at-risk students who received only the email. Furthermore, neither university found any significant impact of the additional support phone call on the likelihood of a student generating additional at-risk alerts.
However, qualitative feedback indicated that students welcomed the interventions. And for some, the phone call was appreciated as a means of breaking down barriers between themselves and the institution and stimulating their re-engagement with learning.
Support is one of four foundations of belonging at university and students taking part in the trials positively welcomed the “personal” nature of the coaching phone call because it felt like “the university cared”.
Some recommendations from the report include:
- Future research should aim to replicate this work, but also to identify whether an email-only intervention is sufficient to help students re-engage with their learning, compared with no intervention.
- Interventions prompted by learning analytics systems must be evaluated with clear pre-specified endpoints to determine their effectiveness; these evaluations should be accompanied by a comprehensive Theory of Change that identifies the issue the intervention aims to solve, the assumptions underlying the intervention and the outcomes that will be measured.
- Developers of learning analytics systems should consider incorporating features that facilitate the evaluation of interventions, for example, in-system randomisation of at-risk students into different intervention groups, the ability to use different interventions with those groups, and the means of testing different methods of identifying at-risk students.
- Higher education providers should gather data to determine what support, if any, students access after receiving interventions, in order to understand whether the services are used, whether they are effective and whether particular groups of students are more or less likely to make use of them depending on the intervention they receive.
TASO next steps
There is currently little causal (Type 3) evidence to support the impact of this type of analytics-prompted intervention on student engagement or success, and the trials reported here are the first in a UK context.
TASO will continue to explore the potential for better evaluation of post-entry support, including interventions prompted by learning analytics systems, in our institutional data use project. We are working with a number of higher education providers to provide practical resources, guidance and evaluation examples for the sector.
Our research has found learning analytics is one of the most common approaches being used to address the ethnicity degree awarding gap. We have been running a project to generate Theories of Change for interventions designed to tackle this issue, which may include learning analytics and associated interventions.
Read: Using learning analytics to prompt student support interventions [PDF]