Hostname: page-component-7c8c6479df-hgkh8 Total loading time: 0 Render date: 2024-03-28T10:30:48.127Z Has data issue: false hasContentIssue false

Colleague and patient appraisal of consultant psychiatrists and the effects of patient detention on appraisal scores

Published online by Cambridge University Press:  02 January 2018

Miranda Heneghan
Affiliation:
Royal College of Psychiatrists' Centre for Quality Improvement, London, UK
Robert Chaplin*
Affiliation:
Royal College of Psychiatrists' Centre for Quality Improvement, London, UK
*
Correspondence to Robert Chaplin (rob.chaplin@rcpsych.ac.uk)
Rights & Permissions [Opens in a new window]

Abstract

Aims and method

This paper aims to review colleague and patient feedback from the 10-year period of the operation of the Royal College of Psychiatrists' 360-degree appraisal system, specifically: (1) examine the overall distribution of ratings; (2) examine the effect of working primarily with detained patients on patient feedback, represented by forensic psychiatrists; and (3) look for a relationship between colleague and patient ratings.

Results

Data were analysed for 977 participating psychiatrists. Both colleagues and patients rated psychiatrists overall with high scores. Less than 1% were identified as low scorers, although there was no relationship between those identified by colleagues or patients. Colleague and patient feedback scores varied little between subspecialties including forensic consultants.

Clinical implications

Psychiatrists in all subspecialties obtained high scores from colleagues and staff. Working with detained patients appeared to have little effect on patient ratings.

Type
Original Papers
Creative Commons
Creative Common License - CCCreative Common License - BY
This is an open-access article published by the Royal College of Psychiatrists and distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
Copyright
Copyright © 2016 The Authors

To receive revalidation from the General Medical Council (GMC), 1 all doctors practising in the UK have to participate in annual appraisal. This has to include patient and colleague feedback (360-degree appraisal) at least every 5 years. Doctors of different specialties participate in various systems of appraisal involving colleague and patient feedback, including those organised by the Royal Colleges of general practitioners, physicians, surgeons and radiologists (under discussion but not yet finalised), and the GMC has produced staff and patient feedback tools. 2 The Royal College of Psychiatrists has developed a system for collecting and analysing data from colleague and patient feedback and have validated questionnaires included in ACP 360, Reference Lelliott, Williams, Mears, Andiappan, Owen and Reading3 which has been shown to provide a reliable assessment.

Patient feedback ratings of a consultant psychiatrist's performance may be influenced by patient or consultant characteristics, or issues arising from the therapeutic alliance. Psychiatrists are unique among doctors as many of them treat patients who are detained and treated against their will. This study analyses the results of patient and colleague feedback received by consultant psychiatrists working in different subspecialties and specifically forensic psychiatrists as they work primarily with detained patients. It is hypothesised that consultants working in forensic psychiatry will have similar colleague ratings but inferior patient ratings of their performance when compared with consultant psychiatrists in other subspecialties.

Method

Data were collected from the ACP 360 system administered by the Royal College of Psychiatrists' Centre for Quality Improvement (CCQI) received between September 2005 and December 2014. To obtain colleague feedback, consultant psychiatrists were asked to supply the email contacts of a range of 15 colleagues from their line manager, consultant psychiatrists, other clinical staff, junior doctors and secretarial, administration and management colleagues. Questionnaires were completed online on 46 items covering a variety of domains of consultant performance, including communication, availability, emotional intelligence, decision-making and relationships with patients and carers, relationships with psychiatrist peers and external agencies. A minimum of 25 patients who the consultant had had significant or recent contact with were sent a standard letter asking them to return a postal questionnaire, rating the consultant on 15 key areas of performance. Each item was rated on a scale of 1 (very low), 2 (low), 3 (moderately low), 4 (moderately high), 5 (high), 6 (excellent). Psychiatrists were grouped as low performing if their overall score was less than or equal to 3.

Data were analysed according to staff feedback and patient feedback separately by the types of subspecialty. To examine the effects of working with detained patients on the quality of their rating, feedback received from forensic psychiatrists was then compared with the other categories of subspecialties combined, referred to as ‘all other subspecialties’. This was because forensic psychiatrists work with detained patients and the other subspecialties only work with a minority of detained patients. Therefore, forensic psychiatrists were presumed to work with detained patients and all the other categories combined were presumed to work with patients who are not detained. It was hypothesised that forensic psychiatrists would receive lower ratings than their colleagues from patients but similar ratings from their colleagues. Data were entered onto the SPSS (version 21) database and statistical analysis was performed with non-parametric Mann–Whitney tests owing to the data being not normally distributed.

The ACP 360 system does not collect any further information about the participating psychiatrists such as demographics, so further analysis of the psychiatrists' individual characteristics and their influence on patient and colleague feedback is not possible.

Results

Between September 2005 and December 2014, there have been 27 826 colleague and 20 543 patient ratings of 977 consultant psychiatrists. Although some psychiatrists will have participated in more than one round of feedback, this study is unable to identify how many have done so.

The distributions of overall scores rated by colleagues and patients are presented in Fig. 1 and Fig. 2 respectively. These ratings are clearly skewed towards the higher ratings and yield a mean overall score of 5.10 (s.d. = 0.293) from colleagues and 5.21 (s.d. = 0.362) from patients, where a maximum possible score was 6.0 for both colleagues and patients. Low performing psychiatrists were those whose mean scores were 4 or below (a score of below the highest scoring range). According to colleague feedback, 8 (0.8%) consultants scored an overall rating of below 4, whereas patient feedback identified 7 (0.7%) different consultants scoring below 4. There was no correlation between colleague and patient feedback (r = −0.21, P = 0.522) (Fig. 3).

Fig. 1 Overall colleague feedback scores for all subspecialties.

Fig. 2 Overall patient feedback scores for all subspecialties.

Fig. 3 Correlation between colleague and patient feedback.

Overall, there were 3891 returns of colleague feedback and 1765 patient returns from 133 forensic consultants (Table 1). Forensic consultants received an overall mean colleague assessment score of 5.04, which was slightly less and significantly lower than the overall score of ‘all other subspecialties’ of 5.11 (Mann–Whitney U-test, P = 0.02). The overall mean score for patient ratings of forensic psychiatrists was 5.23, not significantly different than that for ‘all other subspecialties’ (Mann–Whitney, P = 0.305).

Table 1 Colleague and patient rating scores for subspecialty of consultant psychiatrist

Consultants
n
Colleague scores Patient scores
Specialty Mean (range 1–6)
Addictions 40 5.13 5.29
Child and adolescent psychiatry 6 5.15 5.16
Eating disorders 3 5.17 4.46
Forensic 133 5.04 5.24
General and community 501 5.11 5.20
Intellectual disability 2 4.72 5.23
Liaison 16 5.30 5.02
Old age 217 5.10 5.22
Perinatal 6 4.85 5.26
Psychotherapy 23 5.11 5.29
Rehabilitation and social 30 5.05 5.31
All except forensic 844 5.11 a 5.21 b
Total 977 5.11 5.21

a. Mann–Whitney P = 0.02 higher than forensic.

b. Mann–Whitney P = 0.30.

Discussion

This study has shown that both colleagues and patients rate consultant psychiatrists highly, and that there is little variation in ratings of psychiatrists by either colleagues or patients between psychiatrists working in different subspecialties. High feedback rating scores from staff and patients are not a new finding. With regard to psychiatrists, the original pilot of the ACP 360 Reference Lelliott, Williams, Mears, Andiappan, Owen and Reading3 yielded similar findings.

Multisource feedback tools in general obtain results from colleagues and patients that are highly skewed towards favourable assessments of performance of doctors' practising in a range of specialties. Reference Wright, Richards, Hill, Roberts, Norman and Greco4 This study obtained multisource feedback on over 1000 practising doctors from a range of specialties including acute hospital care, mental health and primary care organisations, however, differences in rating scores achieved by doctors working in different specialties were not reported. Participating doctors were asked to provide lists of consecutive patients seen, so as to minimise the bias of obtaining responses from patients thought likely to respond positively. A study of multisource feedback on the performance of surgeons showed similar high ratings by patients (also consecutively selected) and colleagues, higher than the participants' self-ratings. Reference Violato, Lockyer and Fidler5 However, these studies relied on the participants to provide their own lists of patients. To be more rigorous about the elimination of selection biases, Ramsey et al Reference Ramsey, Wenrich, Carline, Inui, Larson and LoGerfo6 compared the results of staff respondents self-selected by the practising physicians in the USA with those randomly selected by their clinical directors and found no evidence of response bias.

This study identified a small number of psychiatrists rated as low performers: 0.8% by colleagues and 0.7% by patients. This suggests that the instrument is able to detect some individuals whose performance may be poor, although these are not the same individuals poorly rated by both patients and colleagues, a finding which differs from the positive correlation reported. Reference Lelliott, Williams, Mears, Andiappan, Owen and Reading3

With regard to the hypothesis that there would be no difference in rating of colleagues and lower ratings by patients of psychiatrists who worked primarily with detained patients (in this study represented by forensic consultants), there was no evidence to support this. This hypothesis was developed after staff working in the ACP 360 service started receiving feedback from psychiatrists believing that patient feedback would result in a negative assessment owing to having a poor therapeutic alliance resulting from working with detained in-patients. The relationship between detained status, the experience of coercion and the therapeutic alliance is complex and none of these factors have been studied specifically with regard to appraisal of a consultant psychiatrist's performance.

The effects of patients being detained under the Mental Health Act 1983 may have a potentially negative impact on the evaluation. For example, patients admitted to crisis houses rather than to acute mental health wards reported better therapeutic alliance and higher satisfaction with treatment. Reference Baker7 Further, there was a relationship between patients who perceived coercion (related to both informal and detained in-patient admissions) and a negative evaluation of the therapeutic alliance, suggesting the relationship between detention and poor therapeutic alliance is more complex than the legal situation of the patient. Reference Sheehan and Burns8 However, a study of in-patient admissions Reference Roche, Madigan, Lyne, Feeney and O'Donoghue9 found that although detention in hospital predicted a poor therapeutic alliance, the perceived level of coercion on admission did not predict a negative therapeutic alliance. Finally, in a forensic in-patient setting, Donnelly et al Reference Donnelly, Lynch, Devlin, Naughton, Gibbons and Mohan10 found it possible for patients to reliably rate interpersonal trust and working alliance. These studies suggest that psychiatrists working with in-patients or detained patients may have a more negative rating from their patients.

The failure to support our hypothesis of forensic psychiatrists receiving lower patient ratings may come from a number of reasons. First, forensic psychiatrists may have long-term working relationships with patients (seeing the ‘usual’ doctor predicts positive ratings) Reference Campbell and Wright11 and they may be perceived as serving a positive function, for example, rescuing a patient from prison. Second, detention in the community is now common and involves other specialties of psychiatrist, particularly general and community, so other specialties may be exercising coercion. It is of note that Campbell et al Reference Campbell, Richards, Dickens, Greco, Narayanan and Brearly12 found that patients, but not colleagues, scored doctors from a mental health trust lower than doctors from acute and primary care trusts. This study when viewed alongside our study suggests that there may be factors general to all psychiatrists that influence patients' ratings that do not differ particularly across subspecialties.

It is known that patients seeing their ‘usual doctor’ are more likely to give positive feedback. Reference Campbell and Wright11 There is less likelihood that low rates of forensic psychiatrists obtaining patient feedback are due to the same reasons, as longer-term therapeutic relationships are more likely in forensic in-patient settings. Other factors may include patients being unable to complete the feedback owing to lack of capacity. This may apply to psychiatrists working in forensic psychiatry and in psychiatric rehabilitation, although those working in intellectual disability distributed modified questionnaires which are not analysed here.

Strengths and limitations

The ACP 360 is an instrument with robust psychometric properties which has been extensively field-tested. This study involved a large and representative group of psychiatrists from a variety of different subspecialties. However, owing to reasons of confidentiality, no further demographic data (e.g. age, gender, place of employment) were collected from the participants, therefore not permitting further analysis of psychiatrists' characteristics on performance. Furthermore, the categories of subspecialty are broadly defined, for example that of ‘general and community’ psychiatry could include a wide variety of posts including community mental health teams, acute in-patient admission wards and psychiatric intensive care units. Thus there may be variation in patient and indeed colleague rating within this broad category. However, given the narrowness of the patients' ratings and the probable high number of consultants working exclusively with in-patients contained within this group, a higher number of patients rating their consultants at the lower end of the scale would have been expected if detention status was to predicate lower patient ratings.

Clinical implications

This study has two important implications. First, it suggests that the ACP 360 colleague and patient yield results skewed towards the higher ratings of performance in line with those received by doctors practising in other areas of medicine. Second, the appraisal tool does not wrongly discriminate against the performance of forensic psychiatrists who work almost exclusively with detained patients. It also is likely that patients treated by psychiatrists from other subspecialties may rate their psychiatrists in a manner which is not related to their detention. This study is not able to confirm this suggestion and further work into patient ratings of psychiatrists by individuals who have experienced detention or coercion is necessary.

The 360-degree appraisal is a helpful means of assessing psychiatrists' performance when used alongside other measures and psychiatrists need not fear negative ratings from patients when they work in services with high numbers of detained patients.

Footnotes

Declaration of interest

M.H. and R.C. work at the Royal College of Psychiatrists which manages ACP 360.

References

1 General Medical Council. How revalidation works. GMC, 2014. Available at http://www.gmc-uk.org/doctors/revalidation/9612.asp (accessed 14 September 2015).Google Scholar
2 General Medical Council. Colleague and patient questionnaires. GMC, 2014. Available at http://www.gmc-uk.org/doctors/revalidation/colleague_patient_feedback_resources.asp (accessed 14 September 2015).Google Scholar
3 Lelliott, P, Williams, R, Mears, A, Andiappan, M, Owen, H, Reading, P, et al. Questionnaires for a 360-degree assessment of consultant psychiatrists: development and psychometric properties. Br J Psychiatry 2008; 193: 156–60.Google Scholar
4 Wright, C, Richards, SH, Hill, JJ, Roberts, MJ, Norman, GR, Greco, M, et al. Multisource feedback in evaluating the performance of doctors: the example of the UK General Medical Council Patient and Colleague Questionnaires. Acad Med 2012; 87: 1668–78.CrossRefGoogle ScholarPubMed
5 Violato, C, Lockyer, J, Fidler, H. Multisource feedback: a method of assessing surgical practice. BMJ 2003; 326: 546–8.Google Scholar
6 Ramsey, PG, Wenrich, MD, Carline, JD, Inui, TS, Larson, EB, LoGerfo, JP. Use of peer ratings to evaluate physician performance. JAMA 1993; 269: 1655–60.Google Scholar
7 Baker, J. Be nice to patients and they will get better? Therapeutic alliance and service user satisfaction. The Mental Elf, 2014; 16 September.Google Scholar
8 Sheehan, KA, Burns, T. Perceived coercion and the therapeutic relationship: a neglected association? Psychiatr Serv 2011; 62: 471–6.Google Scholar
9 Roche, E, Madigan, K, Lyne, JP, Feeney, L, O'Donoghue, B. The therapeutic relationship after psychiatric admission. J Nerv Ment Dis 2014; 202: 186–92.Google Scholar
10 Donnelly, V, Lynch, A, Devlin, C, Naughton, L, Gibbons, O, Mohan, DJ, et al. Therapeutic alliance in forensic mental health: coercion, consent and recovery. Ir J Psychol Med 2011; 28: 21–8.CrossRefGoogle ScholarPubMed
11 Campbell, J, Wright, C. GMC Multi-Source Feedback Questionnaires: Interpreting and Handling Multi-Source Feedback Results: Guidance for Appraisers. General Medical Council, 2012.Google Scholar
12 Campbell, J, Richards, S, Dickens, A, Greco, M, Narayanan, A, Brearly, S. Assessing the professional performance of UK doctors: an evaluation of the utility of the General medical Council patient and colleague questionnaires. Qual Saf Health Care 2008; 17: 187–93.Google Scholar
Figure 0

Fig. 1 Overall colleague feedback scores for all subspecialties.

Figure 1

Fig. 2 Overall patient feedback scores for all subspecialties.

Figure 2

Fig. 3 Correlation between colleague and patient feedback.

Figure 3

Table 1 Colleague and patient rating scores for subspecialty of consultant psychiatrist

Submit a response

eLetters

No eLetters have been published for this article.