Indian Journal of Physiology and Pharmacology
Medical Education / Original Article
Semi-Structured Assessment of Practical Examination : Need of Hour
Arun Goel1, Rajesh Kathrotia1, Jayvardhan Singh2 and Prashant Patil1*
1Department of Physiology,
*Corresponding author :
Background: To evaluate the learning outcomes, objective methods of assessment play an important role. With limited resources in medical colleges, it is difficult to implement objective methods like objectively structured practical examination (OSPE). This study is an effort to amalgamate objectivity in conventional type of assessment.
Methods: 33 students of first MBBS were assessed independently by 3 examiners during practical examination. One examiner awarded marks in conventional manner while two other examiners used semi-structured assessment taking into consideration the affective, cognitive and psychomotor domain of the students. Linear mixed effects analysis of the relationship between the outcome score and methods of assessment were performed using statistical software R. Two models were constructed: first, a null-model without any predictor and examiners as random effect and second, a full-model with method of assessment as fixed effects and examiners as random effects.
Results: Intra-class-correlation (ICC) was 0.24 in null model and 0.01 in full model. The decrease in variance due to examiners between the models can be accounted to different methods of assessment. Pearson’s correlation coefficient between the two semi-structured assessments was 0.754 as compared between conventional and semi-structured assessment, which was 0.488 and 0.466.
Conclusion: Linear mixed effects analysis of the relationship between the outcome score and methods of assessment shows the significant effect of types of assessment on the outcome score of the students. Semi-structured assessment is better in terms of validity, reliability, feasibility and applicability.
Method of assessment is an important part of medical curriculum as learning of students depend greatly on that. Assessment methods are analysed objectively in terms of its validity, reliability, feasibility, acceptability & educational impact (1,2). A properly designed assessment method has many benefits. It meets the learning objectives, maintains standard of students, help in giving feedback to students as well as teachers and train the students to deal with real life situations (3).
While considering the practical examination, there are different patterns of assessment that can vary from subjective traditional method to more objective OSPE (objectively structured practical examination) method. Both the patterns have their own merits and limitations although properly conducted OSPE seems to have an edge in terms of objectivity and uniformity (4, 5).
Establishment of department of medical education and imparting training to teachers by various teachers-training programmes have helped in improving the assessment methods (6). It has tried to objectify the assessment of examination and bring uniformity in pattern of examination. At the same time, it is also important to be aware of the resources available and their optimum use with minimal effect on the quality of assessment.
The combination of conventional practical examination and OSPE may improve the validity of the examination (7). With this background, we structured the conventional manner of rewarding marks to the students for an experiment so that an examiner is bound to test the students less subjectively and more objectively. The total marks of an experiment or assignment were categorized into pre-defined 3 or 4 parts giving them appropriate weightage and thus testing most of the skills or domain intended to test. Conventional assessment is then compared with semi-structured assessment method.
It was an exploratory study. Ethical clearance was taken for the study.
Subjects were first year MBBS students of batch of 33 students. Subjects were asked to go to each examiner who then assessed them independently in practical examination. Out of three examiners, two (Examiners 1 & 2) awarded marks in pre-defined objective structured manner (semi-structured) and one (Examiner 3) awarded marks in conventional manner without any predefined distribution of marks. Each practical exercise was of 15 marks. In semi-structured format, distribution of marks was done taking into consideration the affective, cognitive and psychomotor domain of the students. Table 1 depicts the sample marksheet utilized for awarding marks in semi-structured format.
We performed a linear mixed effects analysis of the relationship between the outcome score and methods of assessment using R (8) and lme4 (9). Two models were constructed: first, a null-model without any predictor and examiners as the random effect and second, a full-model with method of assessment as fixed effects and examiners as random effects. Intra-class-correlation (ICC) was calculated from the variance components of both the models. We obtained P-values by likelihood ratio tests of the full model with the fixed-effect of assessment methods versus the null-model without the fixed-effect. Additionally, Pearson’s correlation coefficient was calculated between the marks given by different examiners.
In the null-model, ICC was found to be 0.24, which is the proportion of variance in the outcome score that is attributable to examiner level factors. It can be assumed here that part of this variance may be due to the different methods of assessment. For that, we also calculated the ICC from the full-model, which came out to be 0.01. Thus, the decrease in variance due to examiners between the models can be accounted to different methods of assessment.
We also calculated the R - squared from the variance component of both the models and it was found that 96% of the variance was explained by use of different methods of assessment. P-value was also statistically significant (<0.05) for the fixed effect of the methods of assessment.
Pearson’s correlation coefficient between the two semi-structured assessment was 0.754 as compared between conventional and semi-structured assessment which was found to be 0.488 and 0.466.
Conventional subjective pattern of assessment is in place since a very long time. It generally depends on the experience and knowledge of examiners. While on one hand, it may not test all the objective traits or skills (cognitive, psychomotor, affect, behavioural etc.) of the students, on the other hand, it may give an undue weightage to one or the other skill. It can be very unfair or demoralizing to some of the students as students are unique in one or other domain (10).
Secondly, it also affects the uniformity of assessment thus compromising with the quality of examination. Examiners may give more emphasis on final result of examination rather than the ability of student to perform the procedure or examination. There is more thrust on global performance rather than demonstration of individual competencies. Students does not get proper feedback about his performance. Thus, lack of objectivity does not meet the educational objectives and purpose of assessment is not fulfilled (11, 12).
In view of these limitations, there were attempts on
incorporating some objectivity in the examination (13,
14). This has resulted in the introduction of Objective
structured practical examination (OSPE). OSPE was
first conceptualised in form of Objective structured
clinical examination (OSCE) and later extended to
practical examination in 1979 by Harden and
Gleesson (15, 16). OSPE is superior in many ways
to conventional pattern of examination. It tries to incorporate all the possible domains of the student
assessment by dividing an experiment to various
steps and giving weightage at each station. It ensures
objectivity in the assessment to the maximum
possible extent thus overcoming the flaws in the
highly subjective pattern of examination (17). OSPE
pattern of examination is utilized in various medical
disciplines for formative and summative assessment
(18). But this pattern of examination is not also
completely flaw-less and amenable to unsafe
practices. There is minimum examiner – student
interaction so that examiners are now mere observers
and just mechanically giving the marks to the
students. It has been observed that steps designed
in objective check-lists are such that they may not
address the important information of a procedure or
skill that should be essential for all students to
qualify the examination. As this method of
Figure-1 depicts that marks scored in semi-structured format are more in agreement with each other than with conventional assessment. It is also reflected in values of correlation coefficient between the marks given by different examiners (0.754 versus 0.488& 0.466).
It can also be inferenced from figure-2 that there is tendency for giving marks in narrow range to all the students in conventional assessment. Thus, semi-structured tool of assessment appears to be more reliable than conventional tool of assessment.
In terms of acceptability, semi- structured format was more acceptable to students as it provides an opportunity to the student that if he or she is weak in one domain, performance in one or other domain can compensate for that. This approach also addresses the attitude and communication skills of the students.These are very important in the realworld scenario when students will have to interact with the staff, patients and their attendants once they qualify the final examination. It also ensures the evaluation of student for all the important aspects of the examination related to the experiment and its applied importance and thus minimizing the chances of highly subjective type of assessment. At the same time, this method of assessment is also feasible in a sense that it does not require additional number of examiners and can be accepted and adopted effectively in available amount of resources.
A complete crossed design between examiners and different methods of assessment could be a better study design for comparing the two methods but that may not be feasible in this context.
We suggest utilizing the merits of both the subjective and objective pattern of examination and thus adopting a middle way of assessment so that it will be more convenient for adoption in majority of the medical schools keeping in view of their resources. Medical council of India has already relaxed the norms for faculty and staff requirements for medical colleges to all time low (20). This has resulted in drastic reduction of faculties in various medical colleges to such an extent that OSPE pattern of examination is practically impossible in them. This leads to non-uniformity in assessment of students across the country.
The semi-structured pattern of assessment
should be tested and refined further to devise a
method of assessment that does not compromise
with the basic modalities of assessment tool like
validity, reliability, feasibility and acceptability. It
should also be convenient to all the personnel related
to student assessment so that it can be widely
adopted in all the medical colleges across the