Skip to main page content

• HOME
• Current
• Archive
• Feedback
• Subscribe
• Help

Psychiatric Bulletin (2006) 30: 31-34. doi: 10.1192/pb.30.1.31
© 2006 The Royal College of Psychiatrists

This Article Full Text (PDF) Submit an eLetter Alert me when this article is cited Alert me when eLetters are posted Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via CrossRef Citing Articles via Google Scholar Citing Articles via Scopus Google Scholar Articles by Mukherjee, R. A. S. Articles by Hollins, S. Search for Related Content PubMed Articles by Mukherjee, R. A. S. Articles by Hollins, S.

Psychiatric Bulletin (2006) 30: 31-34
© 2006 The Royal College of Psychiatrists

Education & Training

Evaluating qualitative papers in a multidisciplinary evidence-based journal club: a pilot study

Raja A. S. Mukherjee, Specialist Registrar, Honorary Lecturer

Department of Mental Health, Learning Disability, Division of Mental Health, Social and Developmental Psychiatry, St George’s, University of London, Tooting, London SW17 0RE, e-mail: rmukherj{at}sgul.ac.uk

Katherine Owen, Research Assistant

St George’s, University of London

Sheila Hollins, Head of Department

St George’s, University of London

Declaration of interest

None.

TOP Introduction Method Results Discussion Appendix References

 
Journal clubs have traditionally been important means by which clinicians, academics and trainees appraise research that relates to their field. In recent years the evidence-based style has become more prevalent, allowing knowledge gleaned from research to be applied to clinical situations. Currently, the main approach of evidence-based journal clubs is quantitative. This paper describes the evaluation of a modified journal club format, developed in an academic department of a medical school and combining the appraisal of both qualitative and quantitative papers.

The place of evidence-based journal clubs is now established. Since the first series of articles from McMaster University in Canada, when a set of guidelines for evidence-based journal clubs was suggested, there has been a growing literature on the subject (McMaster University, 1981). Subsequently, numerous established international journals have all published their own versions of these guidelines. Gilbody (1996) suggested the format which has been adopted in many psychiatric journal clubs. Warner & King (1997) reported that once implemented in this format, 88% of participants improved their critical appraisal skills and 100% felt it was an appropriate use of academic programme time. Geddes (1998) suggested that the critical appraisal skills of clinicians were, for many, at best rusty. He also highlighted the importance placed on this skill by the College, which introduced the critical review paper as part of the core skills tested in the MRCPsych examination. Dhar & O’Brien (2001) highlight the usefulness of this approach for trainees in preparation for postgraduate examinations. Owen et al (1995) point to the usefulness of this approach in improving research practice and Geddes & Harrison (1997) suggest that adopting this approach improves clinical practice.

Although the argument that qualitative research has an important role is becoming more widely accepted, there is little evidence of an increasing ability to evaluate the quality of qualitative research. There is an ongoing debate about how this can be done and what kind of criteria should be used. In 1998, the National Health Service commissioned a review of the literature and their report highlighted criteria that could be used to differentiate between the quality of qualitative papers (Murphy et al, 1998). Pope & Mays (1999) summarise some of these points and offer further guidelines.

The idea to change the journal club format within the Department of Mental Health (Learning Disability) from an older style to a more evidence-based approach had been suggested for some time. However, the expectation that this would be based on quantitative research was not acceptable because of the unique mix of medical and social science skills within the group. Each theoretical perspective was felt to have equal validity and therefore required equal consideration in the journal club. It was proposed to establish a journal club, which would present and assess both qualitative and quantitative research papers on the same subject at the same session. It was envisaged that critical appraisal skills in both quantitative and qualitative research among all members, regardless of their discipline, would develop. As the critical appraisal of qualitative papers has not been previously described, this component of the process was evaluated.

TOP Introduction Method Results Discussion Appendix References

 
Journal clubs are held regularly as part of the regular postgraduate academic meetings of the Department of Mental Health, Learning Disability. They are attended by doctors, psychologists, clinical and social science researchers, and multidisciplinary community team members, with a variable attendance averaging about ten individuals. It was decided to pilot the joint (i.e. qualitative and quantitative) process of assessment with a view to extending it to a wider audience if successful.

The structure of our new journal club was modelled loosely on that described by Gilbody (1996). A different presenter was chosen for each session, a psychiatric specialist registrar for the quantitative section of the session and social science researcher for the qualitative section. The structure was modified based on comments received throughout the period of study.

Lists of questions used in quantitative appraisal are available from numerous sources (Greenhalgh, 1997; Sackett et al, 1999). These were summarised and provided to the facilitators. For the qualitative papers a series of questions based upon the framework suggested by Pope & Mays (1999) were developed (see Appendix). A glossary of terms, collected from available literature, was also provided to both groups (available from the authors).

Two short questionnaires composed of a mixture of open answer boxes as well as some 5-point Likert scales were developed. The first questionnaire was used to obtain baseline opinions. The second, distributed after four sessions over a 6-month period, was used to assess any change in confidence when appraising qualitative papers, and participants’ enjoyment and the perceived usefulness of the new format.

The collected data were analysed using Stata, version 7, for Windows. Non-parametric statistics were used to assess the change in confidence in those completing the journal clubs based on the null hypothesis that there would be no change.

TOP Introduction Method Results Discussion Appendix References

 
Table 1 provides a breakdown of those attending both the initial and final journal club for the pilot study. Table 2 shows how confidence levels changed during the course of the study and gives a breakdown of the perceived educational value, enjoyment and usefulness of this new format. In all areas, seven or more of those attending reported positive statements.

View this table: [in this window] [in a new window]

Table 1. Breakdown of those attending the pilot study

View this table: [in this window] [in a new window]

Table 2. Confidence levels of nine participants in assessing qualitative papers before and after the pilot study based on a 5-point Likert scale

The majority of people at the start of the pilot study showed little confidence in appraising qualitative papers. Those who were more confident tended to be researchers who were already familiar with qualitative research methods. Owing to the small number in the study, the non-parametric Wilcoxon matched-pairs sign-rank test was used. A significant level of change in confidence was noted (z=2.535, P=0.012). The largest changes were seen in those with the least initial experience of reading and appraising qualitative papers.

TOP Introduction Method Results Discussion Appendix References

 
Despite increasing recognition of the value of the evidence-based journal club, there has been little, if any, appraisal of qualitative journal papers in a medical institution. However, there is an increasing body of evidence to show that qualitative research is considered to be as equally important as quantitative research. Rosser (1999) argues that only the quality and the relevance of evidence blended with the context and values of the patient will achieve the benefit of medical evidence for patients. Greenhalgh & Hurwitz (1999) highlight the importance of narrative, particularly its role in an evidence-based world. They emphasise the importance of listening and understanding the patients’ views and suggest that the process of taking a history can be comparable to methods of qualitative research. They further argue that it is important not to ignore the relevance of qualitative research, as it often seeks a deeper truth and aims to understand the significance of phenomena (Greenhalgh, 1997; Greenhalgh & Hurwitz, 1999). Malterud (2001) argues that qualitative enquiry could contribute to a broader understanding of medical science and that methods of patient care are based on more than just the results of clinical experiments.

The results of this study suggest it is possible to apply the principles of evidence-based journal clubs to qualitative papers. As the new-format club progressed, it became more difficult to find both qualitative and quantitative articles on the same topic. Also, many apparently qualitative papers appear to contain both quantitative and qualitative elements. Furthermore, some of the qualitative papers were very long, and owing to time constraints more guidance as to the areas within the paper to read was needed. Following the conclusion of the study, it was decided to separate the journal clubs into two separate sessions. Box 1 summarises practical tips to help run a qualitative journal club.

The multidisciplinary nature of the Department of Mental Health, Learning Disability meant that initially senior colleagues familiar with qualitative research were able to facilitate the journal clubs. As sessions progressed other people were increasingly able to facilitate the sessions using the guidelines (see Appendix) and the glossary of terms (available from the authors) as a resource. The criteria of the guidelines tended to be strictly adhered to during the early sessions, but as people became more confident, a less rigid adherence developed. This suggests that, with the help of the guidelines, it would be possible to extend this format to other settings, even if experience of reading and appraising qualitative research was minimal.

Box 1. Practical guide to running a qualitative journal club Separate qualitative and quantitative sessions Inform those attending the journal club the title of the paper before the meeting Ensure sufficient time is allowed (at least 45 min but preferably 60 min) Try to pick papers that are not too long With longer articles the facilitator must guide the readers to the main areas to address Provide each group with a glossary of terms and questionnaires Allow plenty of time for group discussion of the main questions The facilitator of each session must be prepared with all the answers and chair the meeting stringently for time Start within guidelines (see Appendix) before asking questions outside them Some experience of qualitative research by a member of the group is useful, especially initially

 

Further benefits are that members of the department are now more aware of qualitative methods and may be more comfortable in using such methods in research projects - an evaluation supported by Owen et al (1995).

The pilot study has shown that the critical appraisal of both qualitative and quantitative papers can easily be introduced to an existing journal club. It suggests that there are advantages achieved by further developing critical appraisal skills to include qualitative research papers.

TOP Introduction Method Results Discussion Appendix References

 
Critical appraisal guidelines for qualitative research

1. What was the aim/research question?
   Was it clear?
   
2. Who took part in the study?
   
   1. Type of participants
      
   2. Number of participants
      
   
   
3. What sampling strategy was used?
   
   1. Theoretical
      
   2. Purposive/purposeful
      
   3. Until saturation reached
      
   4. Convenience
      
   5. Probability (each person has equal chance of being selected)
      
   6. Other Did the sample include the full range of possible cases/settings for conceptual generalisations to be made?
      
   
   
4. What method was used to collect data?
   
   1. Individual interviews
      
   2. Focus groups
      
   3. Observation
      
   4. Analysis of documents
      
   5. Other
      
   
   Were reasons for choice of method explicit? Would a different method have been more appropriate?
   
5. What method of analysis was used?
   
   1. Grounded theory
      
   2. Phenomenology
      
   3. Thematic
      
   4. Content
      
   5. Other
      
   
   Was the researcher explicit in describing analysis process?
   Was the analysis systematic?
   How well did the analysis succeed in incorporating all the observations?
   Was any computer software used to manage the data?
   
6. Results - what were the main themes or other findings discovered in this paper?
   Is it possible to follow links between the data and the explanations or theory given?
   Is the setting/context adequately described so findings could be related to other settings?
   
7. Were any methods used to enhance rigour/trustworthiness?
   
   1. More than one person involved in analysis
      
   2. Respondent validation (feedback to research participants)
      
   3. Searching for negative cases, i.e. those which do not fit the theory
      
   4. Triangulation (the use of more than one method)
      
   5. Reflexivity (considering the effects of the researcher on what is found (e.g. through use of a diary, or inclusion of background, personal characteristics of researcher)
      
   6. Other
      
   
   
8. Your views
   How understandable was the paper?
   How valuable did you find the results?
   Has it contributed usefully to knowledge?
   Did it answer your initial question?
   

Adapted from Pope & Mays (1995) by Owen (2002).

TOP Introduction Method Results Discussion Appendix References

 

1. DHAR, R. & O’BRIEN, A. (2001) Evidence based journal clubs and the Critical Review Paper: Candidates perspective. Psychiatric Bulletin, 25, 67 –68.[Free Full Text]
2. GEDDES, J. (1998) Evidence based practice: a practical approach. Psychiatric Bulletin, 22, 337 –338.[Free Full Text]
3. GEDDES, J. R. & HARRISON, P. J. (1997) Closing the gap between research and practice. British Journal of Psychiatry, 171, 220 –225.[Free Full Text]
4. GILBODY, S. (1996) Evidence based medicine: a new format for journal clubs. Psychiatric Bulletin, 20, 673 –675.[Abstract/Free Full Text]
5. GREENHALGH, T. (1997) How to Read a Paper. London: BMJ Publishing Group.
6. GREENHALGH, T. & HURWITZ, B. (1999) Narrative based medicine; why study narrative? BMJ, 318, 48 –50.[Free Full Text]
7. MALTERUD, K. (2001) The art and science of clinical knowledge: Evidence beyond measures and numbers. Lancet, 358, 397 –400.[CrossRef][Medline]
8. McMASTER UNIVERSITY: DEPARTMENT OF CLINICAL EPIDEMIOLOGY AND BIOSTATISTICS (1981) How to read clinical journals: 1. Why read them and how to read them critically? Canadian Medical Association Journal, 124, 555 –558.[Medline]
9. MURPHY, E., DINGWALL, R., GREATBATCH, D., et al (1998) Qualitative research methods in health technology assessment: a review of the literature. Health Technology Assessment, 2, 167 –198.
10. OWEN, D., HOUSE, A. & WORRALL, A. (1995) Research by trainees; a strategy to improve standards of education and supervision. Psychiatric Bulletin, 19, 337 –340.
11. POPE, C. & MAYS, N. (1999) Qualitative Research in Healthcare. (2nd edn). London: British Medical Journal Books.
12. ROSSER, W. W. (1999) Application of evidence from randomised controlled trials to general practice. Lancet, 353, 661 –664.[CrossRef][Medline]
13. SACKETT, D. L., STRAUSS, S., RICHARDSON, S., et al (1999) Evidence Based Medicine: How to Practice and Teach EBM (2nd edn). London: Churchill Livingstone.
14. WARNER, J. P. & KING, M. (1997) Evidence based medicine and the journal club: a cross sectional survey of participants views. Psychiatric Bulletin, 21, 532 –534.[Free Full Text]

This Article Full Text (PDF) Submit an eLetter Alert me when this article is cited Alert me when eLetters are posted Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via CrossRef Citing Articles via Google Scholar Citing Articles via Scopus Google Scholar Articles by Mukherjee, R. A. S. Articles by Hollins, S. Search for Related Content PubMed Articles by Mukherjee, R. A. S. Articles by Hollins, S.