Translate this page into:
Evaluating objectivity in reflective writing assessments: A comparative study of Niemi’s and Moon’s approaches
*Corresponding author: Nonita Gangwani, Department of Physiology, University College of Medical Sciences, Delhi University, Delhi, India. ngangwani@ucms.ac.in
-
Received: ,
Accepted: ,
How to cite this article: Gangwani N, Khaliq F, Singh S. Evaluating objectivity in reflective writing assessments: A comparative study of Niemi’s and Moon’s approaches. Indian J Physiol Pharmacol. doi: 10.25259/IJPP_326_2024
Abstract
Objectives:
Reflective writing is a key component of competency-based medical education, promoting critical thinking, professionalism and the integration of knowledge into clinical practice. However, assessing reflective writing remains inherently subjective. This study aimed to evaluate student reflections from two MBBS batches using two different assessment approaches and to determine if there were differences in grading between the two methods.
Materials and Methods:
We conducted a retrospective audit of reflections from MBBS students in the 2021 and 2022 batches, focusing on their perspectives regarding the doctor-patient relationship, a crucial part of the ethics module. Following an instructional session on Rolfe’s model of reflection, conducted 1 week after the attitude, ethics and communication session, reflections were graded using Niemi’s and Moon’s approaches. Reflections categorised as ‘diffuse’ by Niemi and those receiving grades E or F by Moon were considered below standard.
Results:
In the 2021 batch (n = 156), 16.02% of reflections were categorised as ‘diffuse’ according to Niemi, with 55.76% receiving grades A and B, and 10.24% receiving grades E and F by Moon. In the 2022 batch (n = 146), 11.64% were classified as ‘diffuse’, 53.41% received grades A and B, and 4.78% received grades E and F. Analysis revealed significant overlap in grades using Moon’s approach. No statistically significant difference was found between the assessments of the two batches using Niemi’s approach, suggesting greater reliability of this method.
Conclusion:
Our study supports the use of Niemi’s approach for a more objective evaluation of reflections. The findings highlight the need for refining assessment criteria and minimising interrater variability to ensure more accurate and equitable evaluations.
Keywords
Reflective writing
Objective assessment
Competency-based medical education
AETCOM
Medical students
INTRODUCTION
The current educational trend of competency-based learning places significant emphasis on gathering multi-source feedback and fostering reflective practice among learners.[1] Reflection involves pausing and retrospectively examining events to derive valuable insights.[2] It encompasses self-reflection, as well as the examination of others’ experiences. Through reflection, teachers and students engage in a process of introspection, contemplating the details and reasons behind a particular event, while also considering potential improvements. Reflection is widely regarded as a fundamental aspect of learning, as it promotes learner autonomy, self-directed learning and the development of lifelong learning skills.[3]
Attitude, ethics and communication (AETCOM) module is an important part of the new competency-based medical education (CBME) curriculum and is divided into all the phases of the total MBBS duration.[4] Teaching AETCOM involves establishing professionalism, guidelines, rational decision-making principles and proper communication between doctor and patient to establish a healthy relationship. However, it also demands creativity and recognition of the emotional aspects inherent in our decision-making. As educators, we must strike a balance between providing structure and allowing space for introspection.
The incorporation of reflective processes is essential for refining attitudes, cultivating virtues and instilling values. Without reflection, these aspects cannot be adequately developed. Educators who aspire to go beyond simply transmitting subject matter recognise the importance of teaching reflection. They believe that by creating supportive learning environments, they can gain a deeper understanding of their students and the intricacies of the learning process.[5]
A potential approach to ethics education involves utilising film clips as a model, which promotes a more comprehensive understanding. Using films, particularly short clips, to initiate and frame discussions, participants can engage in reflective exercises that focus on emotions. This practice helps individuals enhance their reflective abilities and foster the necessary attitudes for creating more humanistic and ethically-minded physicians.
A significant component of the CBME curriculum is the logbook, which is a verified record of the progression of the learner documenting the acquisition of the requisite knowledge, skills, attitude and competencies. Through it, we can ensure that the learner is actively involved in acquiring all the key competencies. A page or a separate logbook is also dedicated to AETCOM Competencies where the formative assessment is in the form of reflection that has to be graded by the faculty and they have to determine the level of achievement as satisfactory (meets expectations) or not satisfactory (does not meet the expectation). Meets the expectation will mean that there is successful completion of the activity and hence the attainment of the competency.[6] To know about the success of any implementation of an educational module, it is important to evaluate the learnings at the end of that programme.[3]
But the major question arises how to assess the reflections? How do we say that they meet the expectation or they don’t? Both qualitative and quantitative methods for analysing reflections such as close reading, thematic analyses and the use of a rubric to evaluate the level of reflection have been suggested.[7] The qualitative analysis provides an in-depth understanding of social reality subjectively and scientifically and an understanding of deeper meanings of written words. However, the major disadvantage is researcher bias and time-consuming.[8]
Association for Medical Education in Europe (AMEE) guide no. 44 suggests two approaches to grade the reflections easily.[9] The first approach by Niemi is based on the observed stages of professional development[10] and the second by Moon is a more pragmatic approach based on practical considerations.[11] Niemi’s approach is categorised into committed, emotional, objective and diffuse reporting of reflections. Moon’s approach is categorised into six grades from grade A to grade F, wherein there is reporting of a change or a commitment to the change in grade A and a poor description of the event in grade F. We went ahead with both to be able to understand ourselves which one is better to follow only one of the approaches in future batches.
Our goal was to enhance the objectivity of evaluating students’ reflections, with a specific focus on minimising interrater variability. Hence, the present study was planned to assess the reflections using both approaches amongst two batches of MBBS students and to see if there is any difference in the grading between the two.
MATERIALS AND METHODS
This study on writing and assessment of reflections was conducted in the Department of Physiology, Medical College. The ethical approval was taken from the Institute’s Ethical Committee-Human Research vide letter number IECHR-2022-54-6-R1.
Ethics module
In the 1st year, the five ethics modules of AETCOM are divided into anatomy, physiology, and biochemistry departments. In physiology, we have to teach three important aspects that are, what it means to be a doctor, what it means to be a patient, and the doctor-patient relationship. There are specifically 7–8 h dedicated to the teaching of each of these competencies.
Assessment of the learnings and recording them in the logbook
The most important aspect of the new CBME curriculum is to stress the learnings as a very important prerequisite to progress further on the course and to maintain the record of this in the logbook. Since the AETCOM module is divided into all the phases, it has a dedicated page to itself in the logbook of each year. The criteria for the successful completion of the activity are writing and assessment of reflections, clearly stating whether they meet the expectation or not. If they do not meet the expectation, then the student is supposed to repeat the task. Since satisfactory completion is an essential pre-requisite for eligibility in the 1st professional examinations so at that time, reflections were evaluated subjectively by the faculty, and those reflections thought to be poorly written were asked to be repeated. This gave rise to the need for objectively evaluating reflections.
Study plan
This retrospective audit was designed to be easily able to grade the reflections so that the criteria of meeting or not meeting the expectation can be completed in the logbook and accordingly final decision is made as completed, remedial or repeat. The reflections from two cohorts of Phase One MBBS 2021 and 2022 students were included for evaluation. The reflections were written on the doctor-patient relationship from the AETCOM module which was taught using Trigger films (TFs) as a teaching-learning method. TFs are short movie clips of 2–3 min duration and depict the concept, we want to teach interestingly. The methodology followed is depicted in Figure 1.

- Methodology followed. AETCOM: Attitude, ethics and communication.
The CBME guidelines for the logbook also mandate reflections as an assessment tool for AETCOM. Since it’s a mandatory part of the curriculum, all the students of these two batches were eligible and had to write reflections in their logbooks. The students who did not attend the session were excluded because they wrote the reflections later on from peer learning.
The students of the 1st-year MBBS 2021 cohort underwent two sessions, each lasting 2 h, between July and August 2021. These sessions, spaced a month apart, were dedicated to exposing them to various scenarios depicting the doctor-patient relationship through TFs. Post each session, a brief discussion ensued, dissecting the highlighted aspects of the clips. As per the schedule of lectures, a week later, the students were instructed on composing reflections based on the Rolfes model,[12] followed by the practical application of crafting reflections in their logbooks. A parallel approach was adopted for the 1st-year MBBS 2022 batch, with sessions occurring from February to April 2022. The reflections were subsequently assessed objectively to reduce interrater variability.
Writing of reflections
As is depicted in the flow chart, 1 week after conducting the AETCOM session, another session was held where the students were taught how to write reflections using Rolfe’s model. This was in the first half of the session, and in the second half of that session, they wrote the reflections in their logbooks. It was a supervised session and was conducted by the author FK. The other two authors and junior residents were also present during the session to supervise the students. The structured model of reflection that we taught our students is Rolfe’s model of reflection because it is simple and easy to understand.[12] It consists of three prompts, What? So what? And now what? The ‘what’ aspect of the model is a description-seeking question where the students have to write about what happened and who was involved in that session. The ‘so what’ aspect enquires about their thoughts and their feelings about a situation or experience that they’ve had, what was the most important aspect, and how was it similar or different from others? It allows them to bring some depth into their experience. The last aspect of the model ‘now what’ mainly talks about what they have learned and how they will apply this in the future or what change has brought in them.
Assessment of reflections
The reflections on the TF sessions were collected from the students and manually classified by a single researcher (NG) into the grades based on both approaches. According to Niemi’s approach[10] that we have chosen, the reflections were classified under 4 headings namely, committed, emotional, objective and diffuse. To classify a reflection as committed, the author looked for lines in reflections where students had written that in the future they would change their attitude/have better learned how to act in a particular situation. For an emotional reflection, NG looked for lines like enjoyed the session/gripping experience/felt sad while watching a clip. In an objective reflection, the students had only reported the sequence of the session or a few movie clips. In diffuse reporting, the student had described a few things in a random sequence or incompletely.
For categorisation of reflections by the Moon’s approach,[11] Grade A was given to reflections where students said that they experienced a change or how they will improve their response in future AETCOM sessions or similar situations when interacting/treating a patient. For Grade B, we looked for comments like this session went well compared to others, whether they learned better or not. For Grade C, students described the session and identified the various key points in different clips and how they felt. For Grade D, students had described the session with the main theme of each clip but could not explain its importance. For Grade E, there was just a description of the session with/without some repetitions. For Grade F, the session was described poorly.
As per the AMEE guide 44, the hierarchical models give an idea about the depth of reflection. Hence, we have considered the lowest grade which is ‘diffuse’ as per Niemi’s approach and ‘grades E and F’ as per Moon’s approach to be below expectation.
Data analysis
After grading was done for all the reflections, the data were entered into an Excel sheet and frequency distribution for all the grades was calculated. A chi-square test was performed to assess any difference between the two batches for both types of approach. The aim behind comparing the two batches of students was to assess the reliability of the approach of assessment. And since statistically no significant difference was obtained in Niemi’s approach, it is more reliable.
RESULTS
In both batches, the total number of students was 170 and we obtained responses from 156 students in the batch 2021 and 146 in the batch 2022, giving a response rate of 91.76% and 85.88%, respectively. This was because some of the students were absent during the reflective writing session, so they wrote reflections in their log book after that which was not considered for evaluation in this study. Figure 2 shows the frequency distribution of grading of reflection in both batches according to Niemi’s approach. On comparing the categories of reflections of the two batches, none of the responses came out to be significantly different between the two batches. Figure 3 shows the frequency distribution of grades of reflection in both batches according to the Moon’s approach. And on comparing the grades of reflections between the two batches, only grade B is significantly different with batch 2021 receiving more grade B compared to batch 2022. Table 1 shows some examples from students’ reflections to better explain Niemi’s approach. Table 2 shows some quotes from the reflections written by students to explain Moon’s approach. When we analysed the reflections, some of them were quite overlapping between different grades according to Moon’s approach, for example, if we categorise the following quote.

- Comparison of frequency distribution of categories of reflection in both batches according to Niemi’s research.

- Comparison of frequency distribution of grades of reflection in both batches according to Moon’s approach *P<0.05.
Few quotes from the student's reflections suggesting ‘committed reflection’ are |
|
Few quotes from the student's reflections suggesting ‘emotional reflection’ are |
|
Few quotes from the student's reflections suggesting ‘objective reflection’ are |
|
Few quotes from the student’s reflections suggesting ‘diffuse reflection’ are – |
|
AETCOM: Attitude, ethics and communication
Moon’s (2004) criteria for grading reflections | Examples of student reflections |
---|---|
Grade A – Experiencing an event(s) has changed, or confirmed, how you experience an event(s). | I had seen these movies before also but I realised the unethical aspects shown in them were pretty normalised. Now, when I go to practice I will remember these points I got to know a lot about medical ethics after watching these videos, and it made me feel how I would react in such situations. I would also make note of value points in such future sessions. |
Grade B – Involves judgment – what went well, or less well and why. | These short films help me to learn fast and retain Since audio and visual memory stays for way longer duration and it was easier to understand as well I really enjoyed the sessions, they were useful in adding to my knowledge and personality. But could have been better by including role plays also so that we could have been more activelyinvolved. |
Grade C – Describing an event – recognising how it affects your feelings, attitudes and beliefs and/or questioning what has been learnt and comparing it to previous experience. | Movie clips covered various aspects like how to take informed consent, confidentialityand privacy. This information helps us to learn proper ways to treat the patient and make their situation easier. Enables us to learn how to avoid crossing professional boundaries. |
Grade D – Describing an event – recognising that something is important but not explaining why. | Through different videos I could understand how to break bad news, How to take informed consent, How to show empathy |
Grade E – Describing an event – repeating the details of an event without offering any interpretation. | The session was helpful. We were taught various important aspects through clips like breaking bad news and communication. It will help in future. |
Grade F – Describing an event – poor description of an event. | We were taught Dr patient relationships through small movie clips which were fun and the quiz was thrilling. |
‘The emotions and the real-life situations made a deep impact on my psyche,’ it can be categorised into grade B as it involves judgment, or grade C as it mainly describes the emotional impact, whereas, as per Niemi’s approach, it will be an ‘emotional reflection’. Such experiences with many quotes indicated that Niemi’s approach is more effective than Moon’s.
DISCUSSION
Assessing reflections in a competency-based curriculum often involves a subjective process. Our study, however, aimed to evaluate reflections more objectively by utilising two distinct approaches. Our findings indicated that Niemi’s approach proved more effective than Moon’s, primarily because it allows for conclusive categorisation of certain reflections. This categorisation can reduce interrater variability and has the potential to serve as a training tool for educators and faculty members. Our objective was to identify reflections in the logbook that fall short of expectations. Students whose reflections do not meet the expected standards can then be notified and guided to repeat the activity under supervision, helping them align with the desired criteria. Specifically, reflections categorised as ‘diffuse’ according to Niemi’s approach and those rated ‘grades E and F’ in Moon’s approach were deemed below expectations, necessitating further guidance and improvement.
These findings align with previous research highlighting the variability in reflective writing assessments. Most earlier studies have focused on qualitative analysis. For instance, Venkataramani and Sadanandan (2018) conducted a qualitative analysis of reflective writing by 66 3rd-year paediatric residents, classifying the reports into four themes: issues related to children and caregivers, learning, motivation and miscellaneous concerns.[13] They found that reflection helped students become more aware of ward activities and address troubling issues by discussing them with mentors, who, in turn, gained insights into the students’ perspectives and concerns. Similarly, Ann-Christin Haffling’s analysis of final-year medical students’ reflections in their portfolios emphasised affective issues, particularly self-awareness of feelings, attitudes and concerns.[14]
Similar to our study, Cohn et al. (2009) conducted a retrospective audit and examined what reflective writing reveals about students’ personal and professional values, identifying narrative typologies used in ethical dilemmas and assessing the usefulness of reflective writing in informing ethics/professionalism curricula.[15] They also compared two approaches for the assessment of reflections and found similar results in both of them. Stark et al. (2006) used guided reflection related to professionalism with 1st-year students, while Castleberry et al. (2016) evaluated a teaching certificate program by analysing the content and extent of participants’ reflections, noting that themes such as strengths, weaknesses, assessment and feedback were commonly discussed.[16,17] In contrast, our study employed an open method, without restricting participants with questions or prompts.
Despite the diversity of methods, there remains a lack of best practices for evaluating reflections. The overall reliability of Niemi’s approach supports recommendations for more standardised assessment tools in reflective writing, as suggested by Makarem et al., 2020.[18] With the increasing use of reflective writing to promote and assess reflection, the need for a valid and reliable evaluative tool that can assess students’ levels of reflection is becoming more urgent. Tools like Niemi’s can enhance assessment reliability and reduce interrater variability.
We believe that the AETCOM teaching approach is vital in engaging students and improving the quality of reflections. We used TFs-short movie clips designed to illustrate concepts in an engaging way as catalysts to spark reflective processes, eliciting emotional engagement from learners and fostering deeper learning experiences. This method empowers educators to cultivate reflective practice among future doctors, motivating them to critically evaluate their experiences and sustain ongoing learning and personal growth. Most participants found the reflections very enlightening and impactful, allowing them to appreciate patients’ feelings and develop empathy. The absence of a statistically significant difference between the two batches in our study suggests that Niemi’s approach is highly reliable.[10] However, contrary results were observed in a study by Tikare and Dhundasi, where the percentage of irrelevant reflective writings was significantly higher for Group I compared to Group II, suggesting the initial method of categorising reflections objectively was unreliable, though they justified this by noting that Group I was the pilot group.[3] Later, they qualitatively analysed the relevant reflections using predefined codes.
It is crucial to evaluate the content taught to know how well the students can understand whatever you are trying to convey and enhance their learning.[3] For this different modalities can be used like collecting and assessing the reflections. Considering the prevalent use of reflective writings in the field of anatomy and physiology, employing a natural language processing method for analysis has the potential to offer a valuable reservoir of fresh insights into students’ experiences and capabilities that were previously unexplored.[19] Due to the ease of use and effectiveness of assessment scales, the assessment of a huge number of reflections which was previously regarded to be difficult, has now become comfortable.[20]
Our findings contribute to the growing body of evidence supporting the use of structured and objective assessment frameworks in reflective writing. Future research could further refine these approaches and investigate their effectiveness across different educational contexts and learner populations.
Limitations
One of the limitations of this study is the diverse range of student backgrounds, each possessing unique capabilities. This diversity presents a challenge that must be acknowledged and addressed in the interpretation of the study’s findings. Notably, students exhibit varying levels of proficiency and interest in writing, introducing potential disparities in their responses and engagement with the subject matter.
Future scope
The insights gained from this study will inform future training for faculty and contribute to improving reflection-based assessments in medical education.
CONCLUSION
By implementing two grading approaches, the study identified areas where reflections fell short of expected standards and revealed variations in grading consistency. Based on our study findings, we would suggest Niemi’s approach for objectively evaluating the reflections. The findings underscore the importance of refining assessment criteria and reducing interrater variability to ensure fair and accurate evaluations.
Acknowledgment
The work was presented as a poster at the TNS Congress 2023 in AIIMS New Delhi and received a letter of appreciation for the same.
What this study adds?
This study provides an objective method for educators to quantify reflections and determine satisfactory completion for logbook entry, ensuring fair and reliable assessment. Implementing these objective measures enhances the consistency and equity of competency-based assessments, benefiting both educators and students.
Ethical approval
The research/study was approved by the Institutional Review Board at the University College of Medical Sciences, number IECHR-2022-54-6-R1, dated August 10, 2022.
Declaration of patient consent
Patient’s consent was not required as there are no patients in this study.
Conflicts of interest
There are no conflicts of interest.
Use of artificial intelligence (AI)-assisted technology for manuscript preparation
The authors confirm that there was no use of artificial intelligence (AI)-assisted technology for assisting in the writing or editing of the manuscript and no images were manipulated using AI.
Financial support and sponsorship:
Nil.
References
- Competency-based undergraduate curriculum for the Indian Medical Graduate. Available from: https://www.nmc.org.in/information-desk/forcolleges/ug-curriculum [Last accessed on 2023 Nov 10]
- [Google Scholar]
- The effect of reflective learning e-journals on reading comprehension and communication in language learning. Comput Educ. 2013;71:124-32.
- [CrossRef] [Google Scholar]
- Reflective writing: Experience of first professional medical students with Attitude ethics and communication module. South East Asian J Med Educ. 2021;15:41-9.
- [CrossRef] [Google Scholar]
- Attitude, ethics & communication (AETCOM) competencies for the Indian medical graduate. Available from: https://www.nmc.org.in/wpcontent/uploads/2020/01/AETCOM_book.pdf [Last accessed on 2023 Nov 10]
- [Google Scholar]
- Reflection in/and writing: Pedagogy and practice in medical education. Acad Med. 2012;87:603-9.
- [CrossRef] [PubMed] [Google Scholar]
- Logbook. Available from: https://www.nmc.org.in/MCIRest/open/getDocument?path=/Documents/Public/Portal/LatestNews/Logbook%20Guidelines_17.01.2020.pdf [Last accessed on 2023 Nov 14]
- [Google Scholar]
- Beyond the margins: Reflective writing and development of reflective capacity in medical education. J Gen Intern Med. 2010;25:746-9.
- [CrossRef] [PubMed] [Google Scholar]
- Factors confounding the assessment of reflection: A critical review. BMC Med Educ. 2011;11:104.
- [CrossRef] [PubMed] [Google Scholar]
- The use of reflection in medical education: AMEE Guide No. 44. Med Teach. 2009;8:685-95.
- [CrossRef] [PubMed] [Google Scholar]
- Medical students' professional identity: Self-reflection during the pre-clinical years. Med Educ. 1997;31:408-15.
- [CrossRef] [PubMed] [Google Scholar]
- A handbook of reflective and experiential learning: Theory and practice In: Abingdon. [E-book]. London and New York: Routledge Falmer; 2004.
- [Google Scholar]
- Critical reflection for nursing and the helping professions: A user's guide London: Palgrave MacMillan; 2001.
- [Google Scholar]
- Analysis of reflective writing of 3rd year medical students during the pediatric clerkship. CHRISMED J Health Res. 2018;5:286-9.
- [CrossRef] [Google Scholar]
- Students reflections in a portfolio pilot: Highlighting professional issues. Med Teach. 2010;32:e53240.
- [CrossRef] [PubMed] [Google Scholar]
- Interpreting values conflicts experienced by obstetricsgynecology clerkship students using reflective writing. Acad Med. 2009;84:587-96.
- [CrossRef] [PubMed] [Google Scholar]
- Discovering professionalism through guided reflection. Med Teach. 2006;28:e2531.
- [CrossRef] [PubMed] [Google Scholar]
- Qualitative analysis of written reflections during a teaching certificate program. Am J Pharm Educ. 2016;80:10.
- [CrossRef] [PubMed] [Google Scholar]
- Grading reflective essays: The construct validity and reliability of a newly developed Tool-GRE-9. BMC Med Educ. 2020;20(1):331.
- [CrossRef] [PubMed] [Google Scholar]
- The use of artificial intelligence to detect students' sentiments and emotions in gross anatomy reflections. Anat Sci Educ. 2023;17(5):954-966.
- [CrossRef] [PubMed] [Google Scholar]
- Reflection: Can we assess it? Should we assess it? Assess Eval High Educ. 1996;21:12130.
- [CrossRef] [Google Scholar]