Hostname: page-component-7c8c6479df-27gpq Total loading time: 0 Render date: 2024-03-27T14:57:51.199Z Has data issue: false hasContentIssue false

Evaluating the quality of educational materials about schizophrenia

Published online by Cambridge University Press:  02 January 2018

J. Scott
Affiliation:
Department of Psychological Medicine, Academic Centre, Gartnavel Royal Hospital, Greater Glasgow Primary Care NHS Trust, Glasgow, G12 0XH
Rights & Permissions [Opens in a new window]

Abstract

Aims and Method

To identify and evaluate the reliability and quality of educational materials provided to individuals with schizophrenia and their carers. Materials used by mental health professionals working in community and in-patient settings were collated. Two independent raters used the ‘Discern’ questionnaire to assess the publications.

Results

Fifteen documents were identified, but only 11 were suitable for evaluation. Interrater reliability of ratings using the Discern tool was highly significant. No educational package scored maximum marks, but four scored in the good quality range. About a fifth of the materials in widespread use were assessed as of poor quality.

Clinical Implications

The use of a rating instrument to assess the quality of educational publications appears to be a reliable and acceptable way to identify the strengths and weaknesses of widely available materials. Staff can use these data to assess the quality of their preferred materials against other publications and make an informed selection.

Type
Original Papers
Creative Commons
Creative Common License - CCCreative Common License - BY
This is an Open Access article, distributed under the terms of the Creative Commons Attribution (CC-BY) license (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted re-use, distribution, and reproduction in any medium, provided the original work is properly cited.
Copyright
Copyright © Royal College of Psychiatrists, 2002

Leaflets and other information packages are regarded as integral to educational strategies designed to promote physical health (Reference CoulterCoulter, 1998). These packages provide patients with information to support their participation in treatment decisions and in the management of their health problems (Reference CoulterCoulter, 1997). Much work goes into the production of educational materials. However, to be effective they must contain comprehensible, relevant and accurate information. Unfortunately, few patient information materials currently meet these standards and many leaflets are inaccurate or misleading (Reference CoulterCoulter, 1998).

This study evaluates the quality and usefulness of educational materials about schizophrenia that were regularly used by mental health professionals working at a large primary care trust in Scotland. This evaluation was important because psychoeducation is clearly recommended as part of the basic psychosocial management of schizophrenia (Scottish Intercollegiate Guidelines Network, 1998). Also, a baseline audit of trust practice suggested a need for greater rigour in the provision of information and education to patients, families and carers.

Various tools have been advocated to enhance the quality of health information (Reference Entwistle, Watt and DavisEntwistle et al, 1996; Reference Jadad and GagliardiJadad & Gagliardi, 1998; Reference Silberg, Lundberg and MusacchioSilberg et al, 1997). The NHS Direct website identifies the Discern questionnaire as the first standardised index of the quality of consumer health information (Reference CharnockCharnock, 1998). Discern has undergone detailed development and review and particularly assesses the quality of written information about treatment choices. However, there are no publications regarding its use in the assessment of educational materials about mental health topics. This study had the following objectives:

  1. (a) To evaluate the quality and usefulness of educational materials about schizophrenia used regularly by staff at the Greater Glasgow Primary Care Trust.

  2. (b) To document the strengths and weaknesses of all the packages reviewed.

  3. (c) To comment on the acceptability and reliability of the Discern tool in assessing mental health information.

Method

The study began with a stocktake exercise to collate materials that were currently used by mental health professionals working in community and in-patient settings. Locality teams were contacted and asked for information on and copies of educational materials being used within their service. A list of the educational materials provided was compiled and sent back to the same staff seeking confirmation that the data-set was complete.

Each piece of educational material was initially assessed to determine if it included comments on treatment choices and whether it was widely available to staff in other trusts. Two independent raters then assessed each publication using the Discern questionnaire. This comprises 16 questions rated on a 1-5 scale. The questionnaire is divided into three sections. Section one (eight questions) concerned reliability, section two (seven questions) measured the quality of information on treatment choices and section three (one question) was an overall rating based on the preceding 15 questions. The scores for each section and the overall rating ranged from 1-5. Ratings using Discern are classified as good, fair or poor according to the following criteria:

Good — score = 4 or 5. The publication rated highly on the majority of questions. A high overall score indicates the publication is a useful and appropriate source of information about treatment choices.

Fair — score = 3. The publication rated high and low on a similar number of questions, or most scores fell in the mid-range. A moderate overall rating suggests the material is a useful source of information on treatment choices but has some limitations. Additional supporting information would be required.

Poor — score = 1 or 2. The publication rated low on most questions. A low overall quality rating indicates the publication is poor quality — it has serious shortcomings and is not a useful or appropriate source of information about treatment choices. It is unlikely to be of benefit and should not be used.

Interrater reliability was recorded and a κ score calculated. All scored publications were summarised for strengths and weaknesses. The overall findings and ratings were compiled into a table and fed back to the clinical teams.

Results

Fifteen publications were identified as being in regular use in the trust for patients with schizophrenia, their families and carers. Four were excluded from further assessment. Two publications were linked specifically to pharmacological products and were not considered suitable for generic use. Two other packages were excluded as they were used as part of a local psychoeducation project on schizophrenia, were undergoing review and were not widely available outside of the trust. Eleven publications were independently scrutinised using the Discern tool. Interrater reliability for overall rating was very good, with a κ of 0.87 (P<0.0001). A review of ratings of each Discern question showed the largest difference in ratings (only two marks on the item rating scale) occurred on only one occasion.

As shown in Table 1 none of the publications achieved either a maximum score or a minimum score, indicating that every publication had at least some good aspects but also some limitations. Only two documents showed disparities in the ratings of section one (reliability of information) and section two (quality of information on treatment choices) of the Discern questionnaire. The first (Reference Wilkinson and KendrickWilkinson & Kendrick, 1996) scored 3.3 for reliability and 5.0 for quality of information. The second (PatientWise, 1997) scored 1.9 and 3.2 on sections one and two, respectively. Based on the Discern ratings, four publications currently in use could be recommended with confidence and two recommended for withdrawal. The remaining five publications got a moderate or fair score, indicating some benefits if used in conjunction with other educational materials.

Table 1. Quality of educational materials about schizophrenia

Title of publication Overall rating Quality of publication
Caring and Coping (National Schizophrenia Fellowship, 1996 (NSF)) 4 Good
A Carer's Guide to Schizophrenia (Wilkinson & Kendrick, 1995) 4 Good
Schizophrenia: The Forgotten Illness (SANE, 1993) 4 Good
Coping with Schizophrenia (Prelapse, 1995) 4 Good
Schizophrenia: Notes for Relatives and Friends (Reference Leff, Berkowitz and Eberlein-FriesLeff et al, 1991) 3 Fair
Schizophrenia (NSF, 1998) 3 Fair
Schizophrenia (NSF & Royal College of Psychiatrists, 1998) 3 Fair
Schizophrenia: Changing Minds (Royal College of Psychiatrists, 2000) 3 Fair
Talking about Schizophrenia (Health Education Board for Scotland, 1994) 3 Fair
Understanding Schizophrenia: A Guide for Sufferers and Carers (Lilly Psychiatry, 1997) 2 Poor
Medical and Health Information for Patients: Schizophrenia (PatientWise, 1997) 2 Poor

Interestingly, there were three weaknesses common to all documents: most failed to provide details of any external appraisal of the information presented, sources of information were not adequately described or dated and areas of uncertainty were not clearly acknowledged. Other recurring weaknesses were failure to provide references, inadequate descriptions of the risks of treatment and failure to consider the ‘no treatment’ option. Last, the date of publication was only printed on four of the documents.

Discussion

This exercise to evaluate the quality and usefulness of the main schizophrenia educational material in the trust showed that most of the material was at least of fair quality. Although the overall score of each publication does not provide any great insight into the strengths and weaknesses of a publication, the Discern questionnaire can provide a greater level of detail if item scores are reviewed.

Having used Discern, the documents that scored most highly shared the following characteristics:

  1. the documents started with clear aims on what the publication was about, what it was to cover and who it was for

  2. information was provided that was specific and relevant to its target audience

  3. sources of information and dates of publications were clear

  4. details of other sources of appropriate information on the topic were provided

  5. the mechanism of how treatment actually works, and its risks and benefits were detailed

  6. information was objective and uncertainties were acknowledged

  7. the consequences of not having treatment were detailed

  8. choices in treatment were highlighted

  9. the idea of shared decision-making was promoted.

The Discern questionnaire was relatively easy to use and interrater reliability for assessments was good. The learning time was also acceptable, taking about 1-2 hours initially to become familiar with the rationale and layout of the document. However, rating each document took 1-3 hours as the reader has to become very familiar with the publication being rated to ensure each Discern question was rated as directed. There are also one or two limitations of the Discern document. First, the developers clearly state that it cannot be used to assess the scientific quality or accuracy of the evidence on which a publication is based — this means information has to be checked against other sources. Second, Discern is described as a stand-alone tool, allowing judgement of a publication without specialist knowledge of the topic under review. Having used the tool, this is something that the project group would not entirely agree with. For example, knowledge on the possible mechanisms of action of certain treatments is required in order to assess if a document described this fully. The same applies to judgements of the range, benefits and risks of treatments. However, these limitations aside, this is a robust tool that can be used to evaluate medical and psychiatric educational materials.

Finally, it is important to note that a major advantage of using the Discern questionnaire and then providing detailed feedback on ratings of reliability and quality of information, is that staff and service users were able to make informed choices about which educational materials to access. This approach avoids the problems that can arise if staff perceive practice guidelines as too prescriptive. To ensure that assessment of the quality of publications used is maintained in the future, one member of staff from each locality team has been trained in the use of Discern so that they can evaluate the quality of new educational materials and provide feedback to their team members.

Qualitative details on the strengths and weaknesses of each document reviewed are available from the first author.

References

Charnock, D. (1998) The DISCERN Handbook: Quality Criteria for Consumer Health Information: User Guide and Training Resource. Oxon: Radcliffe.Google Scholar
Coulter, A. (1997) Partnerships with patients: the pros and cons of shared clinical decision-making. Journal of Health Services Research and Policy, 2, 112121.CrossRefGoogle Scholar
Coulter, A. (1998) Evidence based patient information is important, so there needs to be a national strategy to ensure it. BMJ, 317, 225226.CrossRefGoogle Scholar
Entwistle, V., Watt, I., Davis, H.J., et al (1996) Developing information materials to present the findings of technology assessments to consumers. International Journal of Technology Assessment in Health Care, 8, 425437.Google Scholar
Health Education Board for Scotland (1994) Talking About Schizophrenia. Edinburgh: HEBS.Google Scholar
Jadad, A. & Gagliardi, A. (1998) Rating health information on the internet: navigating to knowledge or to Babel? JAMA, 279, 611614.CrossRefGoogle ScholarPubMed
Leff, J., Berkowitz, R., Eberlein-Fries, R. J., et al (1991) Schizophrenia: Notes for Relatives and Friends. Kingston-upon-Thames: NSF.Google Scholar
Lilly Psychiatry (1997) Understanding Schizophrenia: A Guide for Sufferers and Carers. London: Lilly/NSF.Google Scholar
National Schizophrenia Fellowship (1996) Caring and Coping. Kingston-upon-Thames: NSF/G.P.L..Google Scholar
National Schizophrenia Fellowship (1998) Schizophrenia. Kingston-upon-Thames: NSF.Google Scholar
National Schizophrenia Fellowship & ROYAL COLLEGE OF PSYCHIATRISTS (1998) Schizophrenia. (Help is at hand leaflet) London: RCPsych/NSF. (http://www.rcpsych.ac.uk/info.help.schiz)Google Scholar
Patientwise (1997) Medical and Health Information for Patients: Schizophrenia. Chichester: John Wiley & Sons Ltd.Google Scholar
Prelapse (1995) Coping with Schizophrenia. London: Prelapse.Google Scholar
Royal College of Psychiatrists (2000) Schizophrenia (Changing Minds). London: Royal College of Psychiatrists. (http://www.rcpsych.ac.uk/campaigns/cminds/leaflets/sch/sch.htm)Google Scholar
SANE (1993) Schrizophrenia: The Forgotten Illness. London: Sane Publications.Google Scholar
Scottish Intercollegiate Guidelines Network (1998) Psychosocial Interventions in the Management of Schizophrenia. Edinburgh: Scottish Executive.Google Scholar
Silberg, W., Lundberg, G. & Musacchio, R. (1997) Assessing, controlling and assuring the quality of medical information on the Internet. JAMA, 277, 12441245.CrossRefGoogle ScholarPubMed
Wilkinson, G. & Kendrick, T. (1996) A Carer's Guide to Schizophrenia. London: Royal Society of Medicine Press.Google Scholar
Figure 0

Table 1. Quality of educational materials about schizophrenia

Submit a response

eLetters

No eLetters have been published for this article.