Inter-examiner reliability of the clinical parts of MRCPsych part II examinations

AIMS AND METHOD

The aim of the study was to investigate the interrater reliabilityof the clinical component of the MRCPsych part II examinations,namely the individual patient assessment and the patient managementproblems. In the study period, there were 1546 candidates and773 pairs of examiners. Kappa scores for pairs of examinersin both these assessments were calculated.

RESULTS

The kappa scores for exact numerical agreement between the pairsof examiners in both individual patient assessment and patientmanagement problems were only moderate (0.4-0.5). However,the kappa scores for agreement between pairs of examiners forthe reclassified pass and fail categories were very good (0.8).

CLINICAL IMPLICATIONS

The poor reliability of thetraditional long case and oral examinationsin general is one of the most potent arguments against theiruse. Our finding suggests that the College clinical examinationsare at least not problematic from this point of view, particularlyif global pass or fail judgements rather than discrete scoresare applied.

Introduction

TOP

Introduction

There are undoubted changes facing the practice of medicine.These changes embrace the shift away from the care of the individualto concerns about the health of the community, from cure ofdisease to preservation of health, from episodic care to continuousand comprehensive care, and from in-patient care to communityor home treatment (Jones et al, 2001). These changes are accompaniedby changes in medical curricula characterised by the integrationof basic sciences and clinical medicine in undergraduate medicine,the adoption of social, psychological and humane disciplinesby medicine and the move from didactic teaching to the promotionof active learning. Inevitably the assessment methods thathave traditionally been used in determining clinical competence in undergraduate and postgraduate medicine are also undergoingrevision. In postgraduate medicine, these changes are overseenby the Postgraduate Medical Education and Training Board andare allied to the changes in the structure of training programmesoverseen by Modernising Medical Careers. In this context thereare bound to be far-reaching changes to the College examinations.

The MRCPsych part II clinical examinations currently comprisethe individual patient assessment (IPA), also known as thetraditional long case, and the patient management problems(PMP), which is also known as the structured oral examination.The traditional long case has been established for over 150years as a method of examining clinical skills in medicine.In recent years its value as an assessment method has comeunder great scrutiny. The strengths of the traditional longcase include its obvious face validity, as it evaluates theperformance of a doctor in an encounter with real patients wherebyinformation is gathered and treatment plans are developed under realistic conditions. The task for the candidate is to takea history, to structure the clinical problem, synthesise thefindings and formulate an appropriate management plan. Formany clinicians these skills are fundamental to the practiceof medicine and the authenticity of the challenge for the candidateis an intuitively correct method of assessing clinical competence. Despite these obvious strengths, the traditional long casehas inherent problems. The clinical challenges posed to candidatesin the long case are not identical, equal or even similar incomplexity. Furthermore, it is assumed that performance onone particular type of case is predictive of performance onother types of cases, when most clinicians know that they donot necessarily perform uniformly across all patient problemtypes. In addition, there is concern that examiner behaviouris not reliable (the problems of interrater and intra-raterreliability; Norcini, 2002). In short, there is a conflictbetween validity and reliability.

The problems that arise with the individual patient assessmentalso pertain to the clinical oral examination, the patientmanagement problems. In unstructured viva voce examinationscandidates are liable to be asked whatever questions the examinerchooses and there is the risk that the examiner will concentrateon their pet interests. Furthermore, there is evidence thatstructured viva voce examinations are more reliable than unstructuredexaminations (Tutton & Glasgow, 2005) and the College hasmade the necessary changes to accommodate this. The concernsabout clinical oral examinations are also pertinent to PhDvivas (Jackson & Tinkler, 2001; Morley et al, 2002) andto job selection interviews (Wiesner & Cronshaw, 1988;McDaniel et al, 1994).

In this study we investigate the level of agreement betweenthe two examiners in both component parts of the MRCPsych clinicalexaminations. This is a measure of the inter-rater reliability.

Method

TOP

Method

Data from four sittings of the MRCPsych examinations (Spring2003, Autumn 2003, Spring 2004 and Autumn 2004) were availablefor analysis. In both the individual patient assessment andpatient management problems, examiners are required to awardan independent and individual mark from 1 to 10 (1 being verypoor and 10 being excellent) before entering into the discussionthat precedes the award of the final jointly awarded mark.It is the individually awarded marks that are the subject ofthis study. Scores 1-4 designate failure and 5-10 a pass.

Data were analysed using SPSS version 12 for Windows. In thisstudy the kappa statistic ( ${kappa}$ ) was used as the measure of strengthof agreement between pairs of examiners. The generally acceptedstandards of strength of agreement of ${kappa}$ are: <0 poor, 0.1-0.2slight, 0.21-0.40 fair, 0.41-0.60 moderate, 0.61-0.80 substantial,0.81-1.0 almost perfect (Landis & Koch, 1977).

Results

TOP

Results

In the study period, there were 1546 candidates and 773 pairsof examiners. Data were not available for all candidates whosat the examinations during the study period; only for 1537candidates who sat the individual patient assessment and 1518who sat the patient management problems. The ${kappa}$ for exact agreementbetween the pairs of examiners for the individual patient assessmentfor all scores was 0.513 (P < 0.0001), for the failing candidatesonly was 0.462 (P < 0.0001) and for the passing candidateswas 0.485 (P < 0.0001). When the scores were reclassifiedinto pass and fail categories, ${kappa}$ for agreement was 0.794 (P< 0.0001). The ${kappa}$ for exact agreement between the pair of examinersfor the patient management problems for all scores was 0.515 (P < 0.0001), for failing candidates only it was 0.562 (P < 0.0001) and for the passing candidates it was 0.475 (P< 0.0001). The ${kappa}$ for agreement between examiners in the patientmanagement problems when the scores were reclassified intopass and fail categories was 0.802 (P < 0.0001).

Discussion

TOP

Discussion

The kappa scores for exact numerical agreement between the pairsof examiners in both individual patient assessment and patientmanagement problems were only moderate. However, the kappascores for agreement between pairs of examiners for the reclassifiedpass and fail categories were substantial. We believe thatthis is the first report of the measure of agreement betweenpairs of examiners in a clinical examination in postgraduate psychiatry comprising the traditional long case (individualpatient assessment) and the structured oral examination (patientmanagement problems). The finding of substantial agreementbetween pairs of examiners on the reclassified pass or failcategories suggests that global pass or fail judgements maybe more robust than the award of discrete marks. However, itis important to note that the lower kappa statistic in thecomparison of individual marks is understandable and predictablebecause lower reliability is expected when there are a highernumber of rating options. Notably when a decision is made onpass or fail ratings, there are only two categories, so agreementis statistically more likely. This probably accounts for thehigher kappa statistic for pass or fail judgements. There arereports of high interrater reliability in undergraduate clinicalexaminations (Schwiebert & Davis, 1993; Morgan et al, 2001; Wass & Jolly, 2001). However, there are also reports oflow reliability, particularly with global performance ratingsas opposed to checklists (Daelmans et al, 2005). In postgraduatemedical examinations, there are very few published findings.In anaesthesia, Devitt et al (1997) reported high interrater reliability between pairs of examiners observing performanceon an anaesthesia simulator test.

The reliability of the traditional long case and oral examinationsin general is one of the most potent arguments against theiruse. Our finding suggests that the College clinical examinationsare at least not problematic from this point of view. Norcini (2002) has argued that there are at least three ways to improvethe reproducibility of scores awarded by examiners in the traditionallong case: employing a statistical model to remove differenceamong them; training examiners; or increasing the number of examiners. It is likely that the close agreement between examinersin the College examinations is due to the training provided.New examiners receive initial training before examining andall examiners are required to attend the annual board of examinersmeeting where refresher training takes place. However, Norcini(2002) argues that any improvements in reproducibility of scores,that is, in the reliability of the traditional long case, willonly be modest and that the largest effect is likely to bedue to increasing the number of examiners.

Several authors have proposed modifications to the long caseto make it fit for purpose. These discussions have developedbecause of anxiety that the objective structured clinical examination(OSCE) assesses breadth of skill but at the expense of depth(Wass & van der Vleuten, 2004). The OSCE is able to assessmany competencies but because of the format, very limited timeis available for the assessment of these competencies. In theMRCPsych part I OSCE there are currently twelve 7-min ‘stations’.This illustrates the problem well; there is extensive coverageof novel clinical areas such as information-giving to patients,carers, doctors and other healthcare professionals, yet, theactual time allocated to the evaluation of these competencesis arguably limited. Furthermore, in psychiatry, there is therisk that the OSCE promotes a disjointed acquisition of clinicalskills and that the capacity to integrate a case in all itsfullness may be lost in the process. In real-life situations, the discrete clinical competencies are deployed in order toserve the interest of an individual patient, and to do thissatisfactorily the various aspects of the case need to be integratedinto a meaningful whole. The proposals to improve the intercasereliability of the long case include:

increasing the number of patient encounters that candidateshave, perhaps up to ten (Wass et al, 2001)
increasing thenumber of examiners
observing the clinical encounter as wellas the candidate’s presentation.

McKinley et al (2005) suggest an innovative solution to theproblem of increasing the number of patient encounters; they advocate sequential testing. This involves all students beingdirectly observed in four consultations by a different pairof examiners for each case. Each consultation lasts 30 min.Those considered to be unlikely to fail are excused furthertesting; the rest, approximately a quarter of the class, are observed consulting with four more patients by another fourpairs of examiners. In this system, failing candidates areexamined on eight cases by eight pairs of examiners.

These proposals are resource intensive and probably impractical.In the current MRCPsych part II, the time required for eachcandidate to examine ten patients will amount to at least 10h. In an examination dealing with approximately 1000 candidatesannually, this will be an impossible task. The same problempertains to any extension in the number of examiners or the introduction of wholly observed clinical examinations. Theissue is whether these proposals will produce significant ormerely marginal improvements, and ultimately whether they willbe cost-effective.

There is evidence that the desire to create assessment methodsthat rely on standardised and objectified tasks in a controlledenvironment is returning full circle to the assessment of candidatesin the real world of patients and the workplace (van der Vleuten & Schuwirth, 2005).The concern about the variance introduced by real patientsand the emphasis on the desirability of standardised patients has lessened with the use of the Mini-Clinical Evaluation Exercise(mini-CEX) in workplace-based assessments, with limited observationsof candidate encounters with real patients (Norcini et al, 1995).However, it is doubtful that the mini-CEX can be successfullyapplied to psychiatry without modification. What is now alsoclear is that the reliability of clinical examinations is notdependent solely on objectification or standardisation, butalso on careful sampling across clinical content domains whichneeds substantial hours of testing time (Petrusa, 2002). The reliability estimates for the long case depending on hoursof testing are reported as 0.60 for 1 h, 0.75 for 2 h, 0.86for 4 h and 0.90 for 8 h. These estimates are comparable formultiple choice question papers, oral examination and OSCE(van der Vleuten & Schuwirth, 2005).

It is clear that the proposals to improve the traditional longcase are unlikely to be efficient or cheap. However, the energygoing into the process suggests an awakening to the potentialrisks of relying merely on tests of competence such as theOSCE. At present it is uncertain how far workplace-based assessmentsof clinical performance using instruments such as the mini-CEXcan adequately replace the traditional long case and oral examination.Our findings show that there is a good measure of agreement between pairs of examiners in these examinations, particularlyfor global pass or fail judgements. In this transitional period,as assessments of clinical competence and performance evolve,whatever programme of assessments is developed and adopted,the value of the traditional long case and structured oralexamination need to be carefully considered. It is probablytrue to say that the unique contribution of the long case inparticular is unlikely to be surpassed by simulated patientsor standardised and objectified assessments.

References

TOP