Hostname: page-component-7c8c6479df-r7xzm Total loading time: 0 Render date: 2024-03-27T08:07:08.778Z Has data issue: false hasContentIssue false

Systematic review of workplace-based assessments in psychiatry: surgical dissection and recommendations for improvement

Published online by Cambridge University Press:  02 January 2018

Kanchan Sugand
Affiliation:
St George's ST4–6 Psychiatric Training Scheme, London
Swapnil Palod
Affiliation:
St George's ST4–6 Psychiatric Training Scheme, London
Kalu Olua
Affiliation:
St George's ST4–6 Psychiatric Training Scheme, London
Satyajit Saha
Affiliation:
St George's ST4–6 Psychiatric Training Scheme, London
Asim Naeem*
Affiliation:
South West London & St George's Mental Health NHS Trust, London
Samina Matin
Affiliation:
South West London & St George's Mental Health NHS Trust, London
Mary Howlett
Affiliation:
South West London & St George's Mental Health NHS Trust, London
*
Asim Naeem (anaeem@sgul.ac.uk)
Rights & Permissions [Opens in a new window]

Summary

Recent surveys have highlighted widespread criticisms of the use of workplace-based assessments (WPBAs) in psychiatric training. We describe our systematic review of psychiatric WPBAs, including a ‘surgical dissection’ of their format and process. From our review, we identified seven overarching WPBA themes, and have drawn on these to make further recommendations to strengthen the wider acceptability of WPBAs in psychiatric settings. We hope this will encourage further debate on ways of improving these tools, rather than them becoming side-lined as ‘top-down’ tick-box exercises.

Type
Education & Training
Creative Commons
Creative Common License - CCCreative Common License - BY
This is an Open Access article, distributed under the terms of the Creative Commons Attribution (CC-BY) license (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted re-use, distribution, and reproduction in any medium, provided the original work is properly cited.
Copyright
Copyright © Royal College of Psychiatrists, 2011

The past 5 years have seen major changes to UK psychiatric training, with a move towards competency-based curricula. Many of these changes have occurred within the context of wider national reforms to postgraduate medical education, in response to Unfinished Business, 1 Tomorrow's Doctors, 2 Modernising Medical Careers, 3 The Gold Guide to Postgraduate Specialist Training, 4 The Core Curriculum Guide, 5 and the establishment of the Postgraduate Medical Education and Training Board (PMETB). By aiming to explore all of Miller's pyramid assessment of competencies, Reference Miller6 including what the trainee actually does in daily clinical practice, workplace-based assessments (WPBAs) have a central role in monitoring the progress of trainees. They also offer a number of other advantages compared with traditional methods of trainee assessment, including: the opportunity to observe real-life ‘long case’ assessments of patients; Reference Fitch, Malik, Lelliott, Bhugra and Andiappa7 time for immediate trainee debriefing, highlighting their strengths and weaknesses; opportunities for sampling a wide range of clinical scenarios; assessments of a trainee's teaching skills to peers in journal clubs and case presentations; and encouragement of formative learning. Reference Miller and Archer8

However, recent surveys have highlighted widespread problems and negative attitudes among trainees and trainers towards the current format of WPBAs. Reference Pathan and Salter9-Reference Babu, Htike and Cleak11 Between a half and two-thirds of trainees feel that WPBAs do not accurately reflect their progress, have questionable validity and reliability, and have no real beneficial effects on supervision, training, clinical practice and confidence. Reference Menon, Winston and Sullivan10 Although the College's core training WPBA guide 12 provides greater guidance in mapping scenarios against levels of performance, wider concerns about these tools remain.

Within the context of the current evidence base, drivers to change within postgraduate medical education, and likely future WPBA changes, how can we improve WPBAs to make them more meaningful within psychiatric settings? In order to address this question, we set about to: systematically review the existing evidence relating to WPBAs; record the salient WPBA themes identified from articles meeting the inclusion criteria for our review, including any points made by the authors of those papers; and draw on our systematic review findings to make recommendations for change, to improve WPBAs in psychiatry.

Method

We systematically reviewed the literature with an objective to search for and review the evidence base of WPBAs, primarily focusing on their use within psychiatric settings. We initially searched the four electronic bibliographic databases MEDLINE, EMBASE, CINAHL and PsycINFO (all from 1981 to 2010). Our search terms (English language only) in the title and abstract were: “workplace-based assessment”; “360 degree feedback” OR “multisource feedback” OR “mini peer assessment tool” OR “mini PAT” OR “mini assessed clinical encounter” OR “mini ACE” OR “case based discussion” OR “assessment of clinical expertise”; and “psychiatry”. Results from these searches were combined with AND and duplicate searches removed. Using this search strategy we found 332 articles, which were independently screened by two authors (K.S. and S.P.) on the basis of title, abstract and (where necessary) full texts, and those articles clearly not relevant excluded (e.g. those where WPBAs only occurs as a citation, but without further expansion). We evaluated the remaining articles for quality using our modified version of the Best Evidence Medical Education (BEME) guide Reference Buckley, Coleman, Davidson, Khan, Zamora and Malick13 (Box 1), which was modifed to take into account the relatively few published quantitative comparative studies evaluating WPBAs in psychiatry. Only those articles which met five or more of the quality indicators, from our modified guide, were included, with any discrepancies in the modified BEME score reviewed by one of the consultant authors (A.N.). Using this method for assessing quality, 14 articles satisfied our inclusion criteria: 12 full text articles, including 1 invited commentary Reference Oyebode14 in response to 2 useful WPBA surveys; Reference Menon, Winston and Sullivan10,Reference Babu, Htike and Cleak11 and 2 published correspondences, 1 with valid statistical information from a regional WPBA survey, Reference Pathan and Salter9 and 1 by the Dean of the Royal College of Psychiatrists Reference Howard and Brittlebank15 in response to a regional WPBA survey. Reference Menon, Winston and Sullivan10

In order to widen our search for further psychiatry-specific papers we also conducted focused database searches of the Royal College of Psychiatrist journals The Psychiatrist (previously Psychiatric Bulletin), Advances in Psychiatric Treatment and the British Journal of Psychiatry (all from January 2000 to June 2010). We used the same search strategy (but excluding ‘psychiatry’ as a search term) as per our electronic database review, with the search being applied in all fields and for all types of published articles within the paper journals. We found 243 articles, which were independently screened as above (including removal of duplicate articles identified from our previous search), and those articles clearly not relevant were excluded. After applying our inclusion criteria (modified BEME score ⩾5), we found three additional articles: two full text articles and one published correspondence Reference Sikdar16 highlighting important issues in response to a regional WPBA survey. Reference Menon, Winston and Sullivan10

An additional 6 articles were found to meet our inclusion criteria (modified BEME score ⩾5), by searching the references lists of the 17 articles so far included in our systematic review. To be fully inclusive of all national guidelines relating to WPBAs and postgraduate training, we also conducted an electronic website search (all from 2000 to 2010) of: the Royal College of Psychiatrists; PMETB; Department of Health; Modernising Medical Careers; Medical Education England; and the General Medical Council (GMC). We found a further ten articles that met our inclusion criteria using this final search method.

Box 1 Modified version of the Best Evidence Medical Education guide

  1. Psychiatry-specific articles

  2. Peer-reviewed (or equivalent) publications

  3. Published national guidelines

  4. Study subject group appropriate for the study being carried out

  5. Studies with data collection methods that were reliable and valid for the research question and context

  6. Data-set complete, with an acceptable questionnaire response rate

  7. Appropriate methods of analysis (statistical or other)

  8. Article recommendations reproducible and applicable for workplace-based assessments in psychiatry

  9. Article data or discussions justify a clear set of conclusions

  10. UK published articles

Adapted from Buckley et al. Reference Buckley, Coleman, Davidson, Khan, Zamora and Malick13

Overall, 33 articles met our inclusion criteria for this review (Box 2). Two pairs of authors independently extracted information relating to WPBAs from these articles, using a predesigned and piloted template. This included recording: any salient WPBA themes highlighted in each article; any findings (from formal studies, surveys or other methods) and recommendations from the articles’ authors; and any areas of concern for using WPBAs in psychiatric settings. The accuracy of information extraction was double-checked by the corresponding pair of authors, with inconsistencies resolved by review by one of the consultant authors.

Results

Seven overarching themes relating to WPBAs were identified from the articles included in our systematic review, with some articles covering more than one theme.

Assessment of psychiatric competencies

Workplace-based assessments were not specifically designed to assess postgraduate psychiatric competencies in the first place. Reference Fitch, Malik, Lelliott, Bhugra and Andiappa7,Reference Bhattacharya, Maier, Bhugra and Warner18 Therefore, concerns expressed by trainees and trainers Reference Pathan and Salter9-Reference Babu, Htike and Cleak11 should hardly be surprising. Psychiatry remains unique in placing greater emphasis on working in partnership, promoting a longer-term ‘recovery model’, negotiating meaningful goals that are achievable by the patient, promoting safety and positive risk-taking and challenging inequalities. Demonstration of a trainee's ability to devise diagnostic formulations, alongside awareness of differentials and classification systems, remain important core curricula skills. Reference Bhattacharya, Maier, Bhugra and Warner18 Likert-type assessment scales fail to capture the complexities of these competencies, consistent with evidence that a checklist approach is not appropriate for assessing higher-level trainees. Reference Hodges, Regehr, McNaughton, Tiberius and Hanson19 This can lead to complex psychiatric skills, that take time to master, being reduced to isolated box-ticking competencies. Reference Menon, Winston and Sullivan10 Assessors

Box 2 Systematic review of psychiatric workplace-based assessments: articles included in our review

  1. Full text studies/surveys: Archer et al (2010); Reference Archer, McGraw and Davies17 Babu et al (2009); Reference Babu, Htike and Cleak11 Bhattacharya et al (2010); Reference Bhattacharya, Maier, Bhugra and Warner18 Hodges et al (1999); Reference Hodges, Regehr, McNaughton, Tiberius and Hanson19 Julyan (2009); Reference Julyan20 Menon et al (2009); Reference Menon, Winston and Sullivan10 Searle (2008); Reference Searle21 Wilkinson et al (2008) Reference Wilkinson, Crossley, Wragg, Mills, Cowan and Wade22

  2. Full text commentaries: Oyebode (2009) Reference Oyebode14

  3. Full text review articles: Abdulla (2008); Reference Abdulla23 Brown & Cooke (2009); Reference Brown and Cooke24 Crossley et al (2002); Reference Crossley, Humphris and Jolly25 Faruqui & Ikkos (2007); Reference Faruqui and Ikkos26 Fitch et al (2008); Reference Fitch, Malik, Lelliott, Bhugra and Andiappa7 Holsgrove et al (2009); Reference Holsgrove, Malik and Bhugra27 McIlwrick et al (2006); Reference McIlwrick, Nair and Montgomery28 Miller (1990); Reference Miller6 Naeem et al (2007); Reference Naeem, Kent and Vijayakrishnan29 Ten Cate (2006); Reference Ten Cate30 Thompson (2009) Reference Thompson31

  4. Published correspondence: Howard & Brittlebank (2010); Reference Howard and Brittlebank15 Pathan & Salter (2008); Reference Pathan and Salter9 Sikdar (2010) Reference Sikdar16

  5. National guidelines/papers: Department of Health (2005); 32 Department of Health et al (2007); 4 Modernising Medical Careers (2007); 3 Oakley (2009); Reference Oakley33 Royal College of Psychiatrists (2009); 5 Royal College of Psychiatrists (2010); 12 Royal College of Psychiatrists (2010); 34 Postgraduate Medical Education and Training Board (2007); 35 Postgraduate Medical Education and Training Board (2009); 36 Temple (2010) Reference Temple37

are less likely to correctly administer a tool that significantly limits their freedom to employ their professional judgement. Reference Holsgrove, Malik and Bhugra27

Although the current WPBA forms do allow for some ‘free text’ comments by assessors, they remain secondary to lists of Likert-scale scores, limiting the opportunity for trainees to gain specific advice on how to improve their performance. Workplace-based assessments must provide relevant feedback to trainees, 4 which is a powerful instrument in furthering their personal and professional development. Reference Brown and Cooke24 However, even the best-intended feedback may be unhelpful if it is not descriptive or specific enough. Reference McIlwrick, Nair and Montgomery28

Assessing performance against an agreed exit standard for all trainees, including those who have just started in their post, remains problematic. Some assessors judge where trainees should be at their current stage of training, whereas others score against the ‘end-point’ of their training (which is what these tools were originally intended to do). 36 Although WPBA assessor training has the potential to improve standardisation, wider concerns exist if assessors and trainees regard this part of the system to be inherently unfair. Reference Babu, Htike and Cleak11,Reference Oyebode14 How many other professions assess their workforce according to what they should be able to do at the end of the year? Medical trainees are generally competitive, and view low scores as ‘failure’. 36

Core v. higher specialist training assessments

The European Working Time Directive has resulted in concerns about the impact of restricting the range of clinical experiences trainees are exposed to during their training. Reference Oakley33 Some aspects of the medical curriculum require more rigorous assessment than others, such that a trainee needs to be assessed carrying it out a number of times. 36 Within psychiatry, these include risk assessments, ability to formulate complex diagnostic situations and explaining specific treatments. Concerns continue to be raised by the low success rate of trainees at the MRCPsych Clinical Assessment of Skills and Competencies (CASC) examinations, Reference Sikdar16 where all subspecialty scenarios are assessed. There is a lack of established standards for various training grades, Reference Menon, Winston and Sullivan10 especially for those in higher (ST4-6) specialist training.

Formative v. summative assessments

Workplace-based assessments should act as supportive learning tools for trainees, primarily designed to assess their readiness for progression to summative tests. 36 However, this is where the paradox lies for both trainers and trainees. If WPBAs are primarily tools for giving formative feedback, 5 why are they given significant weight during the Annual Review of Competence Progression (ARCP)? Non-fulfilment of WPBA requirements can result in trainees being failed at their ARCP. Reference Menon, Winston and Sullivan10

Alternatively, given that WPBAs assess up to the highest level of Miller's pyramid of competencies, Reference Miller6 how can ARCP panels accurately gauge a trainee's progress in the clinical workplace without WPBAs contributing evidence to inform summative decisions? 5 By making a clear divide between ‘formative’ and ‘summative’ assessment roles, we can end up under- or overestimating the role of WPBAs in monitoring the progress of trainees.

The role of non-medical assessors

Compared with other medical specialties, psychiatry places greater importance on multidisciplinary team working. Although assessments by non-medical members of teams remain vital to providing a broad psychosocial perspective, devolving trainee assessment to generic mental health workers carries significant concerns. Reference Menon, Winston and Sullivan10 Trainees perceive non-medical assessors to have significantly less knowledge of WPBAs, and be less likely to assess them accurately. Reference Menon, Winston and Sullivan10 Limited evidence exists for using assessors who are not senior medical clinicians to assess experienced postgraduate trainees. Reference Searle21 Assessors need to be able to demonstrate their competence in using WPBAs in psychiatry, including their ability to give feedback. 3,4

360-degree feedback assessments

The concept of 360-degree feedback originated within industrial organisations who wished to improve the leadership qualities of their workforce via self-awareness and positive behaviour change. The mini-Peer Assessment Tool (mini-PAT), a shortened version of the Sheffield Peer Review Assessment Tool (SPRAT), was introduced by PMETB as one of its multisource feedback tools. Whereas some feasibility data are available for the SPRAT, the mini-PAT currently lacks robust data regarding its validity and reliability. Reference Abdulla23 Concerns exist that the validity of multisource feedback tools can be limited by systematic bias from the wide range of assessors, resulting in problems with unregulated self-selection of assessors by trainees. Reference Archer, McGraw and Davies17 For core trainees, in the context of their predominant shift system of on-call work and shorter posts, how many team colleagues are in a position to accurately comment on their performance? Reference Naeem, Kent and Vijayakrishnan29 It can be difficult for team members to confidentially mention their concerns without being easily recognised.

Quality assurance

Standardising WPBA judgements by trainers remains problematic. Reference Pathan and Salter9,Reference Babu, Htike and Cleak11 Even the Assessed Clinical Encounter (ACE), one of the tools that trainees find to be most useful, Reference Babu, Htike and Cleak11 has weak interrater reliability. Reference Searle21 Although assessor training can reduce the potential ‘hawk’ or ‘dove’ effects of individual assessors, there are wider problems. These include the fears of trainers damaging their trainer-trainee relationship by giving low scores, Reference Brown and Cooke24 the ‘acquaintance effect’ where trainees who are well-known to the rater are scored higher, Reference Searle21 and that more senior staff tend to give lower but more accurate ratings compared with less senior staff. Reference Wilkinson, Crossley, Wragg, Mills, Cowan and Wade22 We have all come across the core trainee with a portfolio of perfect WPBA scores, baffled by their failure to pass the MRCPsych CASC exam. Reference Howard and Brittlebank15,Reference Sikdar16

Consultant supervision

The ‘1 hour per week’ supervision has been the cornerstone of the delivery of direct teaching of psychiatric trainees by their supervisors. For supervisors, the time required to deliver this is estimated to be 0.25 programmed activities per week per trainee, which should be incorporated into their job plans. 34 Supervision and WPBAs can have complementary functions, but the increased emphasis on assessment of competencies may displace other educational needs that would have previously been covered within the ‘protected hour’. Reference Julyan20 Quality of supervision remains the most important factor in determining overall trainee satisfaction. Reference Faruqui and Ikkos26

Discussion

In this systematic review, we identified and examined the available current evidence relating to WPBAs and their use within psychiatry. Seven overarching WPBA themes were identified from the reviewed literature, which we have grouped into three areas (Box 3) that pose the greatest challenge in terms of wider acceptability of these tools within psychiatric settings. Our discussion draws on the review findings covering these three grouped areas, making further recommendations for change to help improve the use of WPBAs within psychiatric training.

Box 3 Psychiatric workplace-based assessments (WPBAs): grouping the seven overarching themes identified from our systematic review into areas for improvement

Overarching themes from our systematic review Areas for improvement
(1) Assessment of psychiatric competencies Structure and format of WPBAs
(2) Core v. higher specialist training WPBAs and psychiatric training
(3) Formative v. summative assessments
(4) The role of the non-medical assessor
(5) 360-degree feedback assessments
(6) Quality assurance Quality assurance and consultant supervision
(7) Consultant supervision

Structure and format of WPBAs in psychiatry

Our review identified widespread concerns regarding the use of multiple Likert-type assessments to assess complex psychiatric skills. The alternative, global marking schemes, can increase the ‘halo effect’, however, they allow expert assessors to ‘weight’ the components of the task in a more situation-specific way. Reference Thompson31 There is emerging evidence that narrative information enriches the assessment process, 35,36 and can improve the aspiration towards ‘excellence’ rather than just ‘competence’. This would also allow supervisors to factor in components such as their ‘professional trust’ of the trainee. Reference Fitch, Malik, Lelliott, Bhugra and Andiappa7,Reference Ten Cate30 Some specialties have already designed their own specific WPBA tools (for example Assessment of Psychotherapy Expertise). The concept of a marking system where scores progressively improve with time is

not adopted in most medical schools across the world, and remains problematic. Reference Babu, Htike and Cleak11,Reference Oyebode14 Our recommendations for change include the following.

  1. 1 We support plans to replace the Likert-scale scoring system with a single global marking scheme, as employed in the MRCPsych CASC examinations. However, allowing greater ‘free text’ space, rather than MRCPsych tick-box lists of ‘areas for development’, would encourage assessors to provide more personalised and context-specific feedback to trainees. Examples would include commenting on trainees’ awareness of resource management and National Institute for Health and Clinical Excellence guidelines, their ‘higher order’ thinking in assessing diagnostic uncertainties, and their skills at imparting therapeutic optimism to patients.

  2. 2 Trainees would be better informed of their progress if they were predominantly assessed according to their current level of training, with the global exit standard score appearing at the bottom of the WPBA form. This would reduce the risks of trainees clustering their assessments at the end of their post, when their scores may be highest.

WPBAs and psychiatric training

Acquiring competencies in one branch of psychiatry may not generalise to acquiring skills in another context. Reference Holsgrove, Malik and Bhugra27 For example, a trainee may be able to elicit a history of dementia in a generic setting, but may have more difficulty doing so within an intellectual disability context. This problem may be exacerbated by the impact of the European Working Time Directive Reference Oakley33 and New Ways of Working, 32 and reflected by low MRCPsych CASC pass rates.

Although WPBAs have an important formative assessment role for trainees, they should retain a summative role for ARCP purposes, alongside other portfolio documentation. This dual formative-summative role should be made clear to trainees at the outset. Reference Crossley, Humphris and Jolly25

Our systematic review has highlighted the extremely limited evidence base for using non-medical assessors, particularly for more experienced trainees. This is consistent with our findings that non-medical assessors score trainees more generously, limiting the usefulness of WPBA tools.

Predictive validity studies of multisource feedback tools, correlating them with clinical and examination performance, are necessary to establish their longer-term credibility. Reference Holsgrove, Malik and Bhugra27 We have found that mini-PATs have consistently low response rates from multidisciplinary team members. Allowing core trainees to nominate assessors from previous posts within the last year may help to resolve some of these difficulties. However, we question the validity of the mini-PAT in being able to pick up core trainees in difficulty, as they are more likely to be identified by their supervisors through direct observation, or via formal or informal feedback from those working with them. Our recommendations for change include the following:

  1. 1 Training programme directors should ensure that all core trainees continue to get opportunities to do a post in neurodevelopmental psychiatry, to allow them to complete two to three direct clinical contact WPBAs in this field prior to entering the MRCPsych CASC.

  2. 2 The need to spread out WPBAs should be explicitly incorporated into a trainee's personal development plan at the beginning of each post.

  3. 3 The use of non-medical assessors should be restricted to the multisource feedback components of WPBAs. In examining a patient for consideration of selective serotonin reuptake inhibitor (SSRI) antidepressant treatment, can a non-medical assessor accurately assess a trainee's ability to assess for significant medical aspects of the history, enquiring about gastrointestinal bleeding, coagulopathies or liver disease? Are they able to assess for a trainee's ‘enhanced’ competencies, by rewarding a trainee who enquires about chronic pain disorders and over-the-counter medications taken by the patient, as SSRIs can greatly increase the risk of gastrointestinal bleeding if regularly taken with non-steroidal anti-inflammatory drugs.

  4. 4 As future GMC revalidation is likely to include some form of direct patient feedback, it would be beneficial to link multisource feedback assessments with validated tools that evaluate patient satisfaction.

Quality assurance and consultant supervision

Our review identified the ongoing concerns of trainees undergoing summative WPBAs predominantly by their own clinical supervisor. The low MRCPsych CASC pass rates would reflect these concerns. Quality assurance of WPBA scoring by assessors could be improved by linking the current core training WPBA guide 12 with video examples on the Royal College of Psychiatrists Assessments online website (https://training.rcpsych.ac.uk/), or via a College training DVD. However, to improve the overall validity of WPBAs, there needs to be some form of ‘external’ assessment within each post.

Although WPBAs should be incorporated into daily clinical practice, their implementation has placed considerable demands on trainees and supervisors. It remains unclear as to how much, if any, of the supervision ‘hour’ is devoted to completing WPBAs, and if there is individual variation among supervisors. Medical Education England has emphasised the need for commissioner levers to be strengthened to incentivise training. Reference Temple37 Our recommendations for change include the following:

  1. 1 It should be mandatory for each trainee to have one to two ‘external’ direct clinical contact WPBAs in each post. Within our training scheme, we have introduced a system of ‘paired’ supervisors, where core trainees have a number of assessments by a consultant psychiatrist who is not directly supervising them. Alternatively, ‘external’ assessor pools can be built up from neighbouring training schemes in a reciprocal way, although this has resource implications.

  2. 2 Time for additional training responsibilities, such as WPBA assessments, must be maintained within consultants’ job plans.

Conclusions

Within the context of changes to the delivery of postgraduate medical education, WPBAs play a central role in the assessment of trainees. Although many of the principles underpinning them were sound, there have been significant limitations in their effectiveness when applied to routine clinical practice. We hope our systematic review and recommendations offer practical ways of improving the structure and administration of these tools so that they become more meaningful longer-term assessments. It also offers opportunities for widening the currently limited evidence base on the use of WPBAs in psychiatry, via piloting of some of our recommendations before any general implementation. Areas for College research could include comparative studies looking at the correlation of original and amended WPBA outcomes with MRCPsych CASC scores (including those in subspecialty stations) for core trainees, and with the recently validated psychiatric patient satisfaction (PatSat) scale for all trainees. Reference Hansen, Vincent, Harris, David, Surafudheen and Kingdon38 As competency-based curricula and their associated assessments develop internationally, there would also be scope for other countries to pilot original and amended UK WPBA formats before any general implementation.

Limitations

We used a modified BEME guide for selecting eligible articles, as we found few published comparative descriptive or observational studies of the impact of WPBAs. We set a relatively low threshold number for our modified BEME guide, to ensure we considered the broadest range of currently published evidence. As we did not include e-letters, unpublished articles or informal evidence from conferences, there is a risk of publication bias. There could also be a risk of selective outcome recording (reporting bias), although we tried to minimise this by using ‘hierarchical’ internal consultant peer review of the information extracted from the articles included in our review. We did not assess for bias in each included article, and not all papers meeting the inclusion criteria in our review were used in constructing our recommendations of improving WPBAs. In designing our recommendations, we did not survey psychiatric trainees, but did incorporate triangulation methodology, utilising the experiences of training programme directors involved in the ARCP process. As the use of WPBAs is still in its infancy, it may be too early to establish clear conclusions as to their efficacy. There may also be future changes to their implementation due to wider financial constraints.

Footnotes

Declaration of interest

A.N., S.M. and M.H. are MRCPsych CASC examiners.

References

1 Department of Health. Unfinished Business – Proposals for Reform of the Senior House Officer Grade. Department of Health, 2002.Google Scholar
2 General Medical Council. Tomorrow's Doctors. GMC, 2002.Google Scholar
3 Modernising Medical Careers. About Modernising Medical Careers: The Principles of MMC. MMC, 2007.Google Scholar
4 Department of Health, Department of Health Social Services and Public Safety, National Health Service. The Gold Guide: A Guide to Postgraduate Specialty Training in the UK. Modernising Medical Careers, 2007.Google Scholar
5 Royal College of Psychiatrists. A Competency Based Curriculum for Specialist Training in Psychiatry – Core Module. Royal College of Psychiatrists, 2009.Google Scholar
6 Miller, GE. The assessment of clinical skills/performance. Acad Med 1990; 65: S637.CrossRefGoogle Scholar
7 Fitch, C, Malik, A, Lelliott, P, Bhugra, D, Andiappa, M. Assessing psychiatric competencies: what does the literature tell us about methods of workplace-based assessment? Adv Psychiatr Treat 2008; 14: 122–30.Google Scholar
8 Miller, A, Archer, J. Impact of workplace based assessment on doctors' education and performance: a systematic review. BMJ 2010; 341: c5064.Google Scholar
9 Pathan, T, Salter, M. Attitudes to workplace-based assessment. Psychiatr Bull 2008; 32: 359.Google Scholar
10 Menon, S, Winston, M, Sullivan, G. Workplace-based assessment: survey of psychiatric trainees in Wales. Psychiatr Bull 2009; 33: 468–74.CrossRefGoogle Scholar
11 Babu, KS, Htike, MM, Cleak, VE. Workplace-based assessments in Wessex: the first 6 months. Psychiatr Bull 2009; 33: 474–8.CrossRefGoogle Scholar
12 Royal College of Psychiatrists. WPBA Guide for Core Psychiatry Training. Royal College of Psychiatrists, 2010 (http://www.rcpsych.ac.uk/pdf/WPBA%20Guide%20for%20Core%20Psychiatry%20Training%20(June%202010).pdf).Google Scholar
13 Buckley, S, Coleman, J, Davidson, I, Khan, KS, Zamora, J, Malick, S, et al. The educational effects of portfolios on undergraduate student review. BEME Guide No 11. Med Teach 2009; 31: 282–98.Google Scholar
14 Oyebode, F. Competence or excellence? Invited commentary on … Workplace-based assessments in Wessex and Wales. Psychiatr Bull 2009; 33: 478–9.Google Scholar
15 Howard, R, Brittlebank, A. The future of workplace-based assessments for core trainees. Psychiatrist 2010; 34: 114–5.Google Scholar
16 Sikdar, S. WPBA or CASC/OSCE: where is it going wrong? Psychiatrist 2010; 34: 72–3.CrossRefGoogle Scholar
17 Archer, J, McGraw, M, Davies, H. Assuring validity of multisource feedback in a national programme. Arch Dis Child 2010; 95: 330–5.Google Scholar
18 Bhattacharya, R, Maier, M, Bhugra, D, Warner, J. Curriculum for workplace-based assessments: a Delphi study. Psychiatrist 2010; 34: 204–7.Google Scholar
19 Hodges, B, Regehr, G, McNaughton, N, Tiberius, A, Hanson, M. OSCE checklists do not capture increasing levels of expertise. Acad Med 1999; 74: 1129–34.Google Scholar
20 Julyan, TE. Educational supervision and the impact of workplace-based assessments: a survey of psychiatry trainees and their supervisors. Med Educ 2009; 9: 51.Google Scholar
21 Searle, GF. Is CEX good for psychiatry? An evaluation of workplace-based assessment. Psychiatr Bull 2008; 32: 271–3.Google Scholar
22 Wilkinson, J, Crossley, J, Wragg, A, Mills, P, Cowan, G, Wade, W. Implementing workplace-based assessment across the medical specialties in the United Kingdom. Med Educ 2008; 42: 364–73.Google Scholar
23 Abdulla, A. A critical analysis of mini peer assessment tools. J R Soc Med 2008; 101: 22–6.Google Scholar
24 Brown, N, Cooke, L. Giving effective feedback to psychiatric trainees. Adv Psychiatr Treat 2009; 15: 123–8.Google Scholar
25 Crossley, J, Humphris, G, Jolly, B. Assessing health professionals. Med Educ 2002; 36: 800–4.Google Scholar
26 Faruqui, RA, Ikkos, G. Poorly performing supervisors and trainers of trainee doctors. Psychiatr Bull 2007; 31: 148–52.Google Scholar
27 Holsgrove, G, Malik, A, Bhugra, D. The postgraduate curriculum and assessment programme in psychiatry: the underlying principles. Adv Psychiatr Treat 2009; 15: 114–22.CrossRefGoogle Scholar
28 McIlwrick, J, Nair, B, Montgomery, G. ‘How am I doing?’: many problems but few solutions related to feedback delivery in undergraduate psychiatry education. Acad Psychiatry 2006; 30: 130–5.Google Scholar
29 Naeem, A, Kent, A, Vijayakrishnan, A. Foundation programme assessment tools in psychiatry. Psychiatr Bull 2007; 31: 427–30.Google Scholar
30 Ten Cate, O. Trust, competence, and the supervisor's role in postgraduate training. BMJ 2006; 333: 748–51.Google Scholar
31 Thompson, CM. Will the CASC stand the test? A review and critical evaluation of the new MRCPsych clinical examination. Psychiatr Bull 2009; 33: 145–8.Google Scholar
32 Department of Health. New Ways of Working for Psychiatrists: Enhancing Effective, Person-Centred Services Through New Ways of Working in Multidisciplinary, Multiagency Contexts. Department of Health, 2005.Google Scholar
33 Oakley, C. The Impact of the European Working Time Directive in Psychiatry. Royal College of Psychiatrists, 2009.Google Scholar
34 Royal College of Psychiatrists. Specialist Training in Psychiatry (OP69). Royal College of Psychiatrists, 2010.Google Scholar
35 Postgraduate Medical Education and Training Board. Developing and Maintaining an Assessment System – A PMETB Guide to Good Practice. PMETB, 2007.Google Scholar
36 Postgraduate Medical Education and Training Board. Workplace Based Assessment (WPBA). A Guide for Implementation. PMETB, 2009.Google Scholar
37 Temple, J. Time for training. A Review of the Impact of the European Working Time Directive on the Quality of Training. Medical Education England, 2010.Google Scholar
38 Hansen, LK, Vincent, S, Harris, S, David, E, Surafudheen, S, Kingdon, D. A patient satisfaction rating scale for psychiatric service users. Psychiatrist 2010; 34: 485–8.Google Scholar
Figure 0

Box 3 Psychiatric workplace-based assessments (WPBAs): grouping the seven overarching themes identified from our systematic review into areas for improvement

Submit a response

eLetters

No eLetters have been published for this article.