Abstract:
Mentoring and feedback play crucial roles in medical education, addressing various issues including
exam performance, study techniques, stress management and personal preferences and is an integral
component of quality assurance in medical education. However, time constraints and limited human
resources pose significant challenges in providing personalised guidance to students. This study aimed
to explore the utility of an AI-based personalised feedback system in addressing the needs of medical
students at the University of Peradeniya, Sri Lanka. The study was conducted at the Department of
Pharmacology. Students provided self-evaluations including previous exam results, study techniques,
and stress coping strategies. An AI system, powered by GPT-4 large language model using a retrieval
augmented generation pipeline, engaged in conversations with students, systematically addressing each
point. The AI's responses were grounded on a curated database of existing literature on feedback.
Expert human student counsellors evaluated the system's performance across multiple domains,
including its ability to address key points, provide insightful suggestions, offer sufficient details,
personalise feedback, use varied language expressions and introduce novel perspectives. Evaluator
responses were largely positive. All evaluators agreed that the system effectively addressed key points
of students' strengths in study performance, clearly identified weaknesses and offered insightful and
novel suggestions for improvement. The majority of evaluators (70-90%) agreed that the system
provided clear guidance on exam preparation, offered sufficient detail to guide effective study
techniques and personalised feedback to address individual needs. Eighty percent of evaluators agreed
that the system utilised varied language and expressions effectively. The AI-based personalised
feedback system- ‗Sisu Athwala‘ demonstrated promising results in providing comprehensive, tailored
guidance to medical students. This approach shows potential in addressing the resource constraints
faced by educational institutions while maintaining high-quality, personalised student support. Though
not a replacement for human counsellors, this AI system could serve as a valuable complementary tool
in medical education, enhancing the accessibility and consistency of student feedback and, also offers
a scalable solution to enhance mentoring efforts, contributing to overall quality assurance in medical
education.
