Enhancing Quality Assurance in Medical Education Using Artificial Intelligence-Based Personalised Student Feedback System: ‘Sisu Athwala’

Abstract

Mentoring and feedback play crucial roles in medical education, addressing various issues including exam performance, study techniques, stress management and personal preferences and is an integral component of quality assurance in medical education. However, time constraints and limited human resources pose significant challenges in providing personalised guidance to students. This study aimed to explore the utility of an AI-based personalised feedback system in addressing the needs of medical students at the University of Peradeniya, Sri Lanka. The study was conducted at the Department of Pharmacology. Students provided self-evaluations including previous exam results, study techniques, and stress coping strategies. An AI system, powered by GPT-4 large language model using a retrieval augmented generation pipeline, engaged in conversations with students, systematically addressing each point. The AI's responses were grounded on a curated database of existing literature on feedback. Expert human student counsellors evaluated the system's performance across multiple domains, including its ability to address key points, provide insightful suggestions, offer sufficient details, personalise feedback, use varied language expressions and introduce novel perspectives. Evaluator responses were largely positive. All evaluators agreed that the system effectively addressed key points of students' strengths in study performance, clearly identified weaknesses and offered insightful and novel suggestions for improvement. The majority of evaluators (70-90%) agreed that the system provided clear guidance on exam preparation, offered sufficient detail to guide effective study techniques and personalised feedback to address individual needs. Eighty percent of evaluators agreed that the system utilised varied language and expressions effectively. The AI-based personalised feedback system- ‗Sisu Athwala‘ demonstrated promising results in providing comprehensive, tailored guidance to medical students. This approach shows potential in addressing the resource constraints faced by educational institutions while maintaining high-quality, personalised student support. Though not a replacement for human counsellors, this AI system could serve as a valuable complementary tool in medical education, enhancing the accessibility and consistency of student feedback and, also offers a scalable solution to enhance mentoring efforts, contributing to overall quality assurance in medical education.