Human vs Machine learning: rethinking student assessment to foster metacognition of pharmacological skills

Abstract

Introduction: Pharmacology students can recognize propranolol in a multiple choice of beta-adrenergic receptor antagonists (BRA). But can they integrate human physiology to predict the effects of BRA on the cardiovascular system? Can they perform or interpret pharmacological assays that quantify BRA effects? Are they aware of their true level of knowledge, or under the delusion that recognizing drug names (e.g., associative memory of the suffix -olol) is a skill that solves real-world pharmacology problems? The ability to self-assess, monitor, and improve via self-regulation are metacognitive skills that enhance academic learning and emotional intelligence. With the now widespread access to information, metacognitive skills are more valuable than simple associative memory, and should thus be fostered in students [1,2]. Objectives: Implement a learning environment that fosters the development of metacognitive skills, including an assessment strategy that values the demonstration of such skills. Methods: 2nd year Pharmaceutical Sciences students (2021/2022, n = 175) were surveyed for their motivation, initiative, and study strategies (Survey-1); and exposed to 10 weekly formative tests (ForTes) where they predicted their score (metacognitive monitoring and self-assessment). Students received weekly feedback on their ForTes and counselling on active study techniques. In the summative assessment (mid-term Exam), the accuracy of student score prediction was valued as bonus. Students were then surveyed (Survey-2) on their perception of ForTes, and their study adaptations according to counselling (metacognitive regulation). Anonymised data were analysed with R language to compute statistics and perform supervised machine learning. Results: Survey-1 showed that student self-perception of motivation, initiative and study were highly inter-related (internal consistency: IC = 86%; n = 144). The metacognitive ability of students to predict their grade started low (R2 = 0.13 and 0.10 for prediction vs. reality in ForTes 1 and 2), and then increased gradually (R2 = 0.22, 0.41, and 0.46 in ForTes 3, 4, and 5), with the observed maximum in ForTes 9 (R2 = 0.54). Also, the first three ForTes had low IC (44-52%), and then increased gradually to the observed maximum by ForTes 9 (IC = 81%). The Exam had high IC (88%), low standard error (5%), and a grade prediction vs. reality R2 of 0.40. Survey-2 showed that the majority of students consider that the ForTes were very useful (97%) and promoted: their motivation (90%); their regular study (74%); their use of active study techniques (80%); and improved their metacognitive skills (84%). A supervised machine learning (ML) model trained with a random sample of 70% of Exam scores (using as predictors the number of ForTes taken by the students and their average ForTes score) yielded an adjusted R2 of 0.67. Using the remaining 30% Exam scores as a test sample, the ML model predicted Exam scores (scale: 0-20 points) with an error under 2 points (root mean square error = 1.9). Conclusions: This strategy of regular formative tests with student score predictions, feedback, and counselling, increased student motivation and active learning. Moreover, increases in self-assessment accuracy indicate that students self-monitored, and self-regulated to increase their learning - as supported by increased internal consistency of the tests. Together with survey data, these results show that this strategy fostered student metacognitive skills. Further improvements in the ML models of human (student) learning, can potentially g