© 2007 The Royal College of Psychiatrists
Education & Training |
Foundation programme assessment tools in psychiatry
*Division of Mental Health, St George’s, University of London, Cranmer Terrace, London SW17 0RE, email: a.naeem{at}sgul.ac.uk
Division of Mental Health, St George’s, University of London
Charing Cross Higher Specialist Training Scheme, London
 TOP Introduction The assessment toolsDiscussionConclusionsReferences |
|---|
In line with Modernising Medical Careers (Department of Health, 2003), the foundation year programme aims to bridge the gap between undergraduate and specialist training. Psychiatry posts have been incorporated into the second year of this programme, with satisfactory progress of doctors being monitored via a range of workplace-based assessment tools. Learning that occurs in the context of the daily workplace is more likely to be relevant and reinforced, leading to better practice (Davis et al, 1995).
This paper provides an overview for consultants, specialist registrars (SpRs) and staff grade/associate specialists, all of whom may be approached to assess foundation year 2 trainees using these competency-based assessments. Examples of psychiatric settings in which the range of workplace-based assessment tools can be used and a critical review of their usefulness are considered.
|
TOPIntroductionThe assessment toolsDiscussionConclusionsReferences |
|---|
There are four tools that assessors may be asked to complete by foundation year 2 psychiatric trainees:
- mini-Clinical Evaluation Exercise (mini-CEX)
- case-based discussion (CbD)
- mini-Peer Assessment Tool (mini-PAT)
- direct observation of procedural skills (DOPS)
Although the trainee holds responsibility for deciding the time of the assessments, where they occur, and selecting their assessors (Davies et al, 2005), they have to complete a minimum number of each during foundation year 2. General instructions for using these tools are available at the Modernising Medical Careers website (http://www.mmc.nhs.uk/pages/assessment)
| Box 1. Examples of foundation year 2 mini-Clinical Evaluation Exercise
(mini-CEX) in psychiatry
|
mini-Clinical Evaluation Exercise (mini-CEX)
This is a 15-min ‘snapshot’ assessment of an observed clinical
encounter, adapted from the American mini-CEX, which was originally designed
as a replacement for the traditional long case format. It is adaptable to a
range of clinical scenarios, producing roughly comparable scores over
examiners and settings (Norcini et
al, 1997), and has been shown to be a feasible and reliable
evaluation tool (Durning et al,
2002). Each mini-CEX should be followed by a 5-min instant
feedback by the assessor. Although the difficulty of each mini-CEX is
dependent on the patient’s complexity, assessors tend to take this into
account by overcompensating for patient difficulty
(Norcini et al,
2003).
Strengths
Being similar to the MRCPsych objective structured clinical examination
(OSCE) format, certain areas lend themselves well to being assessed with this
tool (Box 1). Scenarios should
be incorporated into daily clinical practice settings such as ward reviews or
out-patient clinics. There is scope for other professionals (for example,
liaison psychiatric nurses) to act as assessors, while the opportunity for
instant feedback can help trainees refine their interview techniques and
diagnostic skills by encouraging reflective practice.
Physical examination skills with a relevance to psychiatry can also be assessed. This should encourage assessors to keep their skills up to date, a concern highlighted by Garden (2005).
Weaknesses
Only small aspects of the psychiatric consultation process can be assessed,
and there may be a greater variance with psychiatric patients (in terms of
rapport, willingness to discuss issues or volunteering information). Concerns
have been raised that assessors tend to form limited general impressions of
trainees based on their assessment of only one or two objectives (for example
clinical skills, professionalism). This may be exaggerated in psychiatry,
where verbal communication skills play a key role.
| Box 2. Example of case-based discussion questions in psychiatry Assessment of alcohol misuse for detoxification
|
Case-based discussion (CbD)
By focusing on a trainee’s case note records, this allows the
assessment of the trainee’s clinical decision-making, reasoning and
application of medical knowledge with actual patients
(Brown & Doshi, 2006). It
is based upon the concept of ‘chart stimulated recall’, used in
the USA and Canada. As each CbD lasts only 15 min, only one aspect of the case
(rather than the entire case) should be explored (see
Box 2).
Strengths
‘Holistic’ aspects of the treatment of common mental illnesses
can be discussed, in relation to what the trainee actually did. It can also be
useful to explore the trainee’s ‘hierarchy’ of drug
management (for example using antidepressants or antipsychotics), and whether
it is consistent with the current evidence base. Issues of capacity and
consent may be better looked at within a psychiatric context, as can
differential diagnoses and underlying aetiological factors.
Case-based discussion can enhance the skills of doctors who may enter other specialties (for example general practitioners, gastroenterologists and endocrinologists who come across comorbid alcohol misuse or complications of obesity), and can encourage good record-keeping.
Weaknesses
The duration is not long enough to explore complex patient’s
problems, and there can be a danger of the exercise turning into a
mini-long-case viva, particularly with assessors who are more familiar with
that format. Psychiatric notes are more comprehensive than those of other
specialties, so it is often necessary to adapt the questioning for certain
cases.
mini-Peer Assessment Tool (mini-PAT)
This is a form of multi-source feedback, and has been adapted from the
Sheffield Peer Review Assessment Tool
(Archer et al, 2005).
It is based on the concept of 360° assessment, in which a trainee seeks
feedback about their performance at work from a variety of colleagues,
highlighting areas of strength and those in need of improvement
(King, 2002). It is a useful
way of assessing generic skills (such as communication, team working, teaching
and reliability), which indirectly measure performance
(Hays et al, 2002).
In UK pilots, it has been found to be practical and acceptable to senior house
officers in hospital settings (Whitehouse
et al, 2002). A variation of this tool (the Team
Assessment of Behaviour) is used in some regions.
Strengths
The principles of 360° appraisal are supportive to interprofessional
team development (McLellan et al,
2005). It can help foundation year 2 trainees to develop a
holistic approach to patient care by playing a part in multiprofessional ward
reviews and care programme approach (CPA) meetings.
By assessing aspects of the doctor-patient relationship, this tool assesses the qualities which overlap with attributes of a good psychiatrist (Bhugra & Holsgrove, 2005). Ward reviews or CPA meetings can allow trainees to display their communication skills with patients and carers (for example explaining diagnoses or treatments) in front of other health professionals, some of whom could be selected to complete the mini-PAT forms. Out-patient psychiatric clinic letters also allow a trainee’s written communication skills to be assessed, as copies are usually sent to other members of the multidisciplinary team. The use of several assessors in the mini-PAT process leaves the tool less open to bias.
Weaknesses
In the context of the shift system of psychiatric on-call work, and the
short duration of each post, how many peer colleagues are in a position to
accurately comment on a trainee’s performance? Evans et al
(2004) have commented on this
concern in other specialties. It can also encourage unduly positive feedback
in the ‘space for comments’ section, owing to concerns that
trainees may recognise anonymous comments highlighting poor performance. The
most valid source of ratings for ‘humanistic’ dimensions are
patients (Church, 1997),
particularly in psychiatry, but they are surprisingly excluded from the
assessment process.
Direct observation of procedural skills (DOPS)
This assesses trainees’ practical skills, in a range of
pre-determined tasks with a patient. Each DOPS should last no longer than 15
min, followed by 5 min of feedback.
Tasks such as venepuncture (for clozapine blood monitoring or for plasma lithium levels), performing an electrocardiogram (for example prior to considering antipsychotic treatment) or giving electroconvulsive therapy can be readily assessed, placing emphasis on the trainee’s communication skills (for example obtaining valid consent and explaining the need for the test) in performing the task. Aside from these examples, this tool currently has limited applicability in foundation year 2 psychiatry posts.
|
TOPIntroductionThe assessment toolsDiscussionConclusionsReferences |
|---|
There is no nationally agreed undergraduate curriculum in psychiatry to act as a baseline template for foundation year 2 competency assessments, with medical schools having considerable autonomy in how they teach and examine psychiatry. The ‘ten essential shared capabilities’ (Hope, 2004) are an attempt to reach a consensus on what competencies should be core for the entire mental health workforce. This can act as a parallel framework for applying workplace-based assessment tools for foundation year 2 psychiatric trainees.
It is essential that trainees get a fair assessment of their competence, by pitching the tools at an appropriate level, in the context of more generic career development; this requires adequate training of all potential assessors. It would be reasonable to expect a foundation year 2 psychiatric trainee to perform a safe and effective suicide risk assessment, or be able to describe their routine approach to the assessment of pyrexia or confusional state within a psychiatric context. However, it may not be appropriate to expect a detailed knowledge or experience of cognitive-behavioural therapy.
Although foundation year 2 trainees may have more medically oriented approaches to the presentation of common mental health problems (for example assessment and management of substance misuse), the assessment tools can provide an opportunity to reinforce the importance of personal, family, social and cultural factors.
Feedback from these tools should encourage trainees to create self-directed learning plans. However, there are concerns as to whether this actually happens in reality (Norman et al, 2004). Criticisms also exist regarding the vagueness of the scoring systems and the absence of independent assessors, as most assessors are known to the trainee (Rose, 2006).
Exposure to foundation year 2 psychiatry posts offers an opportunity to boost recruitment into our specialty, but evidence suggests that undergraduate experience may be a more positive determining factor (Goldacre et al, 2005). Therefore, these assessment tools should be used as an opportunity to develop more advanced psychiatric competencies in trainees who may enter other specialties.
Workplace-based assessment tools for specialist training
Although the foundation year 2 assessment tools will also be used to assess
doctors in specialist training, some of the formats have been adapted by the
Royal College of Psychiatrists (for example the mini-CEX has been revised to
become the mini-Assessed Clinical Encounter or mini-ACE). Additional tools are
also currently being piloted, including the Assessment of Clinical Expertise
(ACE), case and journal club presentations, and a patient satisfaction
questionnaire (details are available from the Royal College of
Psychiatrists’ website:
http://www.rcpsych.ac.uk/training/specialtytrainingassess.aspx).
Assessors are likely to have to use different sets of workplace-based
assessment tools for trainees in their foundation year 2 or specialist
training years 1-5.
|
TOPIntroductionThe assessment toolsDiscussionConclusionsReferences |
|---|
Foundation year 2 assessment tools provide structured, standardised methods of monitoring doctors’ progress, against a broad range of core competencies. Brown & Bhugra (2005) have highlighted the need for such tools to assess clinical and non-clinical competencies. By grounding these assessments within a relevant workplace context, it is hoped that trainees will retain the skills they learn. As the character Arthur said in Patrick White’s The Solid Mandala: ‘I forget what I was taught. I only remember what I’ve learnt.’
We are grateful to Dr Joan Rutherford (Consultant Psychiatrist, South West London and St George’s Mental Health NHS Trust) for reading and commenting on the final draft of this paper.
|
TOPIntroductionThe assessment toolsDiscussionConclusionsReferences |
|---|
- ARCHER, J. C., NORCINI, J. & DAVIES, H. A. (2005)
Use of SPRAT for peer review of paediatricians intraining.
BMJ, 330, 1251
-1253.
[Abstract/Free Full Text] - BHUGRA, D. & HOLSGROVE, G. (2005) Patient-centred
psychiatry. Training and assessment: the way forward? Psychiatric
Bulletin, 29, 49
-52.
[Free Full Text] - BROWN, N. & BHUGRA, D. (2005) Modernising medical
careers, the Foundation Programme and psychiatry. Psychiatric
Bulletin, 29, 204
-206.
[Free Full Text] - BROWN, N. & DOSHI, M. (2006) Assessing
professional and clinical competence: the way forward. Advances in
Psychiatric Treatment, 12, 81
-89.
[Abstract/Free Full Text] - CHURCH, A. (1997) Do you see what I see? An exploration of congruence in ratings from multiple perspectives. Journal of Applied Social Psychology, 27, 983 -1020.[CrossRef]
- DAVIES, H., ARCHER, J., HEARD, S., et al (2005) Assessment tools for foundation programmes - a practical guide. BMJ Careers, 330, 195 -196.
- DAVIS, D. A., THOMPSON, M. A., OXMAN, A. D., et al
(1995) Changing physician performance: a systematic review of the
effect of continuing medical education strategies. Journal of the
American Medical Association, 274, 700
-705.
[Abstract/Free Full Text] - DEPARTMENT OF HEALTH (2003) Modernising Medical Careers. Department of Health.
- DURNING, S. J., CATIAN, L. J., MARKERT, R. J., et al (2002) Assessing the reliability and validity of the mini-clinical evaluation exercise for internal medicine residency training. Academic Medicine, 77, 900 -904.[Medline]
- EVANS, R., ELWYN, G. & EDWARDS, A. (2004) Review
of peer assessments of physicians. BMJ,
328, 1240
-1242.
[Abstract/Free Full Text] - GARDEN, G. (2005) Physical examination in psychiatric
practice. Advances in Psychiatric Treatment,
11, 142
-149.
[Abstract/Free Full Text] - GOLDACRE, M. J., TURNER, G., FAZEL, S., et al
(2005) Career choices for psychiatry: national surveys of
graduates of 1974-2000 from UK medical schools. British Journal of
Psychiatry, 186, 158
-164.
[Abstract/Free Full Text] - HAYS, R. B., DAVIES, H. A., BEARD, J. D., et al (2002) Selecting performance assessment methods for experienced physicians. Medical Education, 36, 910 -917.[CrossRef][Medline]
- HOPE, R. (2004) The Ten Essential Shared Capabilities - A Framework for the Whole of the Mental Health Workforce. Department of Health.
- KING, J. (2002) 360 degree appraisal. BMJ Career Focus, 324, s195 .
- McLELLAN, H., BATEMAN, H. & BAILEY, P. (2005) The place of 360 degree appraisal within a team approach to professional development. Journal of Interprofessional Care, 19, 137 -148.[CrossRef][Medline]
- NORCINI, J. J., BLANK, L. L., ARNOLD, G. K., et al (1997) Examiner differences in the mini-CEX. Advances in Health Sciences Education: Theory and Practice, 1, 27-33.
- NORCINI, J. J., BLANK, L. L., DUFFY, D., et al
(2003) The Mini-CEX: a method for assessing clinical skills.
Annals of Internal Medicine,
138, 476
-481.
[Abstract/Free Full Text] - NORMAN, G. R., SHANNON, S. I. & MARRIN, M. L.
(2004) The need for needs assessment in continuous medical
education. BMJ, 328, 999
-1001.
[Free Full Text] - ROSE, N. (2006) Hazards ahead? Invited commentary on:
Assessing professional and clinical competence. Advances in
Psychiatric Treatment, 12, 89
-91.
[Abstract/Free Full Text] - WHITE, P. (1969) The Solid Mandala. Penguin.
- WHITEHOUSE, A., WALZMAN, M. & WALL, D. (2002) Pilot study of 360 degree appraisal of personal skills to inform record of in-training assessments for senior house officers. Hospital Medicine, 63, 172 -175.[Medline]
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||