Evaluating the impact of different academic support schemes for first-year students

Project description and aim:

The university has a centrally managed and funded Student Academic Support scheme which promotes peer support of first-year undergraduate students by second- and third-year undergraduates. The scheme is not compulsory, and several departments across the university have their own approaches to peer academic support for first-year students that have emerged historically in the development of the university and its departments. The study was designed to assess the impact of the centrally managed scheme and two other departmental schemes on the academic development of first-year students.

This is a form of service evaluation, in that it is primarily concerned with improving the university’s support for its students, but the study’s conclusions and details of the data obtained will be circulated widely in the university and beyond. Should the study show one scheme to have a greater impact on student retention, progression or grades, this would directly affect the university’s policy. It was also clear that the evidence developed had potential implications for other universities and should be disseminated via conferences and publications.

Methods:

The study had a mixed-methods design and included three elements:

Statistical analysis of the retention, progression and overall grade for all first-year students in relation to their level of attendance and the academic support offered to students by their department and also examined in relation to IMD score for home address, gender and whether students commuted or lived in student accommodation
Audio-recorded focus groups (of a random sample of first-year students in each of the three types of support scheme) were held for:

(1) non-attendees at support sessions,

(2) occasional attendees (<33% attendance),

(3) regular attendees (33–66% attendance) and high attendees (>66%), with 9–11 students in each of the 12 focus groups

Interviews with second- and third-year students who were peer supporters on each of the three schemes.
The researchers also gathered background information about the schemes that were publicly available within the university, including minutes of planning meetings and any formal notes made by members of staff.

Key ethical considerations:

The three elements outlined above each brought ethical concerns.

Firstly, a central team could conduct the statistical analysis anonymously if attendance at academic support sessions were recorded directly in the same way as attendance at lectures and seminars. This required a change in the way data was collected but avoided the need to manually add attendance to the database (which would have required knowledge of students’ names). Further, it was realised that it could be possible to de-anonymise data collected for small course sizes. After investigation, it was concluded that de-anonymisation was unlikely to be possible in course cohorts greater than 50. A margin of error was built in, and courses of fewer than 60 students were excluded from parts of the analysis. It was judged that this part of the study carried a very low risk and, as it was anonymous, did not require ethical consent from students.

The second element – the focus groups – was considered to present a higher level of risk and multiple ethical challenges, including:

  • the need to identify students to contact them
  • the potential harm of discussing academic needs and support, especially with students who had not attended or attended very few sessions
  • the need to audio-record, transcribe and then quote students in reports
  • the lack of confidentiality given by a focus group compared with interviews or questionnaires.

The team decided that the student’s voice was necessary for the study and that collective discussion in groups, segmented by attendance level and the support scheme available, was also methodologically the most desirable way of gathering data. The team decided that course teams should put a short video on their virtual learning environment and show it in an all-course lecture. This video gave details of the study, explained that the evaluation team might be in touch via university email and confirmed that students would not be disadvantaged if they did not agree to take part in the study.

Course leaders were briefed on how to support unsure students. The study team contacted potential participants, and course teams did not know who had been invited or whether they had consented or refused. A random sample of students was approached with an information sheet and consent form, and given the day and time of the focus group. These meetings were scheduled over lunch when there were no lectures; lunch was provided, as were travel expenses for commuting students who were not on campus that day. Participants were also given a university travel mug and a voucher for a hot drink by way of a thank you.

Staff in central academic support had been briefed about the study and any student who was negatively affected by the focus group conversation was directed to their offices. Participants were assured of confidentiality from the study team and reminded of their right to withdraw at any point until the end of the focus group. In the focus group, participants gave themselves pseudonyms and referred to each other using these names during the session. The study team did not record the real names of participants. Transcriptions and notes were, therefore, pseudonymised at the point of data collection.

The third element of the study was deemed of lower risk. All second- and third-year students involved as peer supporters received from their scheme coordinators an information sheet outlining what was involved in the study. They were invited to email a member of the study team to arrange a mutually convenient time to meet on campus for an interview and were sent a consent form to complete. The interviews took place over a hot drink and vegan cake and lasted around 30 minutes, focusing on what those students thought the first-year students had gained from the support sessions. The interviewer was experienced and monitored for any indication of discomfort or distress. However, none was evident. Following the interview, the participant was sent a thank you email, a voucher for a hot drink on campus and a reminder they had a further 48 hrs to withdraw from the study.

The background documents used in the study were publicly available to all university members and had been redacted to remove any personal information. It was therefore decided that these documents could be used without further consent from their authors.

Scientific limitations and recommendations for future research:

The anonymity of participants’ personal information should be protected even when the main source of information is pre-collected data. Especially when a large statistical analysis is involved, de-anonymising samples can be difficult or impossible. An alternative approach should be considered if this is the case.

When identifying and approaching students in educational settings, the power dynamic between students and teachers can trigger ethical issues. For example, in this case, the course convenors were in a position to support and inform about the project, but were not required to approach or invite students. Thus, students did not feel they had to participate in the focus group because of the power relations between them and their teacher (e.g. for fear of losing marks if the course convenor were their examiner). Conversely, the course team did not know which students were involved in the project. The participants’ identities were further protected by not exposing their choices to non-investigators.

Download this case study as a PDF