Hostname: page-component-7c8c6479df-24hb2 Total loading time: 0 Render date: 2024-03-28T02:34:20.278Z Has data issue: false hasContentIssue false

The quality of internet information on depression for lay people

Published online by Cambridge University Press:  02 January 2018

Phil Ferreira-Lay
Affiliation:
SW London and St George's NHS Mental Health Trust, London, UK, email: phillip.ferreira-lay@swlstg-tr.nhs.uk
Steve Miller
Affiliation:
Springfield University Hospital, SW London and St George's NHS Mental Health Trust, London, UK
Rights & Permissions [Opens in a new window]

Abstract

Aims and Method

To examine the quality of information concerning depression available on the internet, identify factors associated with ‘good’ quality, and develop a simple-to-use instrument for assessing websites on depression. The Depression-Website Content Checklist was developed and compared with a previously validated, yet too complex, scale. Websites were assessed using both instruments.

Results

Good quality information is more likely to be found on websites provided by governmental, professional and charitable organisations. The differences we observed in the median scores for these websites using the Depression-Website Content Checklist are significant at the 0.05 level (Ctot; Mann–Whitney, U=24.00; P=0.013). The Depression-Website Content Checklist is a valid and reliable user-friendly tool.

Clinical Implications

Patients can be directed towards better quality information by diverting them to public and non-sponsored websites. Clinicians can use the Depression-Website Content Checklist to determine website quality.

Type
Original papers
Creative Commons
Creative Common License - CCCreative Common License - BY
This is an Open Access article, distributed under the terms of the Creative Commons Attribution (CC-BY) license (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted re-use, distribution, and reproduction in any medium, provided the original work is properly cited.
Copyright
Copyright © Royal College of Psychiatrists, 2008

Eighteen per cent of all internet users have sought mental health-related information (Reference Powell and ClarkePowell & Clarke, 2006) and they are more likely to search for information on depression than on any other condition (Reference TaylorTaylor, 1999). However, information available on the internet has been reported to be of variable quality (Reference BowerBower, 1996). Therefore, clinicians may have a role in helping the public find better quality websites.

Website sponsorship by organisations such as drug companies may be one factor contributing to disseminating ‘poor’ quality information (Reference Christensen, Griffiths and MedwayChristensen et al, 2000) and the public needs guidance on where to find reliable information on health issues. International initiatives such as DISCERN attempt to apply quality control to internet-based health information (Reference Charnock, Shepperd and NeedhamCharnock et al, 1999; Reference Gagliardi and JadadGagliardi & Jadad, 2002). However, DISCERN and other currently available instruments are complex and unwieldy, a user-friendly rating tool is needed.

Method

Websites on depression available on the internet at the time of the research were identified using the single search term ‘depression’ on multiple search engines. The UK and other English language websites were targeted to get an international perspective. Information available on the SearchEngineWatch.com website identifies the most commonly used search engines and eight of them were used for this research.

It is rare for people to look beyond the first ten results of a query (Reference Silverstein, Henzinger and MaraisSilverstein et al, 1998), thus our analysis was limited to the first ten sites listed by each of the eight search engines. The sites were excluded if the website: had no depression content; focused more on other forms of affective illness; was not in English; was linked solely to another search engine; or focused on the non-adult population (i.e. those under 16 years of age).

Two key documents were analysed for common factors suitable for converting into a simple measuring scale: the National Institute for Health and Clinical Excellence (NICE) draft document (NICE, 2001) and the World Health Organization Mental Health Checklist (WHO, 2001). From these, a ten-item scale was developed (Table 1). Each item attracts one mark and therefore all items are given equal weighting. The items address illness domains - aetiology, symptoms, treatment and prognosis. Summation of item marks gives a total score (C tot); the maximum possible score is 10 and the minimum is 0.

Table 1. Depression-Website Content Checklist of key points on websites on depression

Areas of interest Illness domain Items Score
Nature of illness Aetiology/symptomatology 1. Low mood 1
2. Loss of interests 1
3. Decreased energy levels 1
Course of illness Prognosis 4. Duration 1
Treatment options Treatment 5. Antidepressants 1
6. CBT/IPT 1
7. Problem-solving 1
Treatment problems Treatment 8. Side-effects 1
Warning signs Prognosis 9. Suicidal thoughts/plans 1
Treatment 10. Consider referral to specialist mental health services, e.g. psychiatrist for further risk assessment 1

The quality of websites on depression was then assessed using both the Depression-Website Content Checklist and DISCERN - a previously validated instrument used to judge the quality of written healthcare information best applied by ‘experienced users’ (Reference Charnock, Shepperd and NeedhamCharnock et al, 1999). The DISCERN instrument generates a total score (D tot, maximum 75) for 15 key criteria. Scores were calculated for each website.

In addition to their individual quality, websites were compared according to two grouping variables, that is sponsored (yes) and non-sponsored (no). Evidence of financial support for the development and/or content of each of the websites was sought. A statement of such support on any of a website's pages resulted in allocation of that website to the ‘yes’ group, otherwise it was allocated to the ‘no’ group; the Mann-Whitney U-test (the non-parametric equivalent of a t-test) was used to compare the websites grouped in this way. They were also grouped according to organisation type - public organisations (governmental/professional bodies and charities) and private organisations (drug companies and individuals) - and analyses of their respective websites were performed.

Validating the Depression-Website Content Checklist

As DISCERN had been previously validated, correlation between it and the Depression-Website Content Checklist was measured to ascertain concurrent validity. Both are ordinal scales, hence Spearman's rank correlation coefficient was used. Internal scale consistency was measured using Cronbach's α. Each website was rated by a second-rater (S. M.) oblivious to previous scores and interrater reliability for the Depression-Website Content Checklist was measured using the intraclass correlation coefficient (ICC).

Results

We identified 68 different website addresses by analysing the first ten sites listed at each of the eight search engines when using the search term ‘depression’. In total, 39 websites were excluded according to previously stated criteria. The main reason for exclusion was that the site did not contain any depression content. There were 29 websites remaining for further investigation. However, several of those websites linked to the same website; for example, three addresses linked to the website for the National Institute for Mental Health. We were finally left with 23 websites to be further analysed (Table 2). Examination of the websites using the two instruments showed them to be of variable quality (range 2 - 10 on the Depression-Website Content Checklist, 24 - 74 on DISCERN).

Identifying factors that appear to predict quality

Comparison of the sponsored and non-sponsored websites is illustrated in Table 3.

Table 3. Descriptive and statistical analyses of websites on depression classified by absence/presence of sponsorship and type of organisation (public/private), using DWCC total score (C tot) and DISCERN total score (D tot)

Grouping variable DWCC total score (C tot) DISCERN total score (D tot)
No sponsorship (n=15) 9.00 (2–10) 58.00 (24–73)
Sponsorship (n=8) 5.50 (4–8) 36.50 (31–48)
   U 25.500 26.500
   P 0.024* 0.030*
Public organisation (n=9) 10.00 (3–10) 64.00 (30–73)
Private organisation (n=14) 6.50 (2–10) 40.00 (24–61)
   U 24.000 22.500
   P 0.013* 0.011*

Non-sponsored websites have higher median scores than sponsored websites when the two ratings C tot and D tot were used (Table 3). Observed differences are significant at the 0.05 level (C tot;Mann-Whitney U=25.500; P=0.024).

Comparison of the websites when grouped according to whether they belonged to public or private organisations is illustrated in Table 3. Public organisations have higher median scores for C tot and D tot. Observed differences are significant at the 0.05 level (C tot;Mann-Whitney U=24.00; P=0.013).

Assessment of the Depression-Website Content Checklist

There was a high degree of positive correlation between the Depression-Website Content Checklist total score for each website and the corresponding DISCERN total score (Spearmans r=0.935, significant at 0.01 level, two-tailed). Cronbach's α=0.857 for the Depression-Website Content Checklist and α=0.935 for the DISCERN tool. These values (α>0.8) indicate a high level of positive correlation between each of the items on both scales. The intraclass correlation coefficient was 0.85 (CI 95% 0.66-0.94) indicating very good agreement between raters when using the Depression-Website Content Checklist to assess the quality of websites.

Discussion

We aimed to assess the quality of information on depression available to lay internet users. Although other authors (Reference Griffiths and ChristensenGriffiths & Christensen, 2005) have also examined the quality of websites on depression, they were not previously ranked according to their popularity and the multiple search engines had not been employed. Our study demonstrates two things: first, that public, non-sponsored information on depression is of better quality than private, sponsored information available on the internet; and second, a simple user-friendly rating instrument (Depression-Website Content Checklist) can be used to assess the quality of websites.

Overall the quality of websites we analysed was extremely variable. The large proportion of websites excluded in this study (45/68, i.e. >65%) indicates the quality of information on depression on many of the most commonly accessed websites is rather poor. Many websites initially identified by search engines as containing information on depression did not actually have such information, which shows that individuals may be directed to a number of irrelevant sites. Several sites linked to the same provider, for instance the Royal College of Psychiatrists or the National Institute for Mental Health. This indicates that users may access the same information in different ways, but lack of clarity in accessing better quality information may ensue.

For websites with depression content it was possible to identify factors associated with ‘good’ quality. Our data indicate that the absence of sponsorship is associated with better quality information. Statistically significant differences were found when non-sponsored sites were compared with sponsored ones. Analysis of the content of the eight sponsored websites showed that all were provided by drug companies. Possible bias in favour of a drug company's own advocated treatment has been noted elsewhere (Reference Wahlbeck and AdamsWahlbeck & Adams, 1999). One might suggest that the poorer quality information on these websites compared with others is related to businesses wishing to promote their own products. This is not a concern of non-sponsored website providers. Such organisations include the National Health Service in the UK and the National Institute for Mental Health in the USA, which both aim to provide comprehensive information on health. These sites achieved the highest ranking scores on DISCERN and Depression-Website Content Checklist.

Other authors (Reference Griffiths and ChristensenGriffiths & Christensen, 2005) have also found non-sponsored sites to be of higher quality. Further research would help clarify to what extent findings from the UK apply across the rest of the English-speaking world. It is reasonable to assume that users are likely to access both national/regional and international websites.

In order to simplify the assessment of quality of internet information on depression we developed a simple-to-use Depression-Website Content Checklist that we also evaluated. Apart from its high degree of utility the instrument has good internal consistency (Cronbach's α>0.8 is comparable with a rating instrument suitable for making clinical decisions). It remains to be seen, however, how valid and reliable such simple instruments are.

As far as validity is concerned, both the Depression-Website Content Checklist and DISCERN have been shown to provide concurrent results. Moreover, as the Checklist's content was taken from the World Health Organization, its construct and face validity are also significant. It also possesses a high interrater reliability such that different users of the instrument are likely to reach the same conclusions about the quality of any given website. Since a certain degree of experience is needed to use DISCERN, some clinicians may prefer to use the Depression-Website Content Checklist.

In future this methodology might be employed to develop a similar scale for assessing internet information on other major mental illness such as schizophrenia. It would be relatively simple to devise item scores for website information on other illnesses, based around aetiology, symptoms, treatment and prognosis.

The instrument developed in our study concentrates on identifying the quality of information on depression on each of the websites we analysed. We defined the comprehensiveness of a website by the extent to which it displays the ten items of illness information that make up the Checklist. However, there may be other things about the websites such as their layout and design, which make them more, or less, helpful. A poorly designed website may make the information difficult to read and absorb, therefore reducing clarity. These factors were not considered in this study (errors of commission). Rather, we have been able to accurately identify those websites on depression that contained good quality information by checking how they score on the ten-item checklist.

We conclude that healthcare professionals have a key role in directing service users to good quality (public, non-sponsored) websites and the Depression-Website Content Checklist may help them do that.

Limitations

A small number of websites on depression were analysed and investigations focused on sites accessible by UK users. One of the authors (P.F.-L.) undertook the search for relevant websites on depression and rated them in turn, which means that the analyses are prone to rater bias. Also, we only looked at whether the items of the rating instrument were present/absent in each of the sites and did not examine the rest of the site content. Therefore, although the information we gathered is accurate and measured by our instrument, other aspects of the site (e.g. layout and design) were not analysed.

Declaration of interest

None.

References

Bower, H. (1996) Internet sees growth of unverified health claims. BMJ, 313, 318.CrossRefGoogle ScholarPubMed
Charnock, D., Shepperd, S., Needham, G., et al (1999) DISCERN: an instrument for judging the quality of written consumer health information on treatment choices. Journal of Epidemiology and Community Health, 53, 105111.CrossRefGoogle ScholarPubMed
Christensen, H., Griffiths, K. M. & Medway, J. (2000) Sites for depression on the Web: a comparison of consumer, professional and commercial sites. Australian and New Zealand Journal of Public Health, 24, 396400.CrossRefGoogle Scholar
Gagliardi, A. & Jadad, A. R. (2002) Examination of instruments used to rate quality of health information on the Internet: chronicle of a voyage with an unclear destination. BMJ, 324, 569573.CrossRefGoogle ScholarPubMed
Griffiths, K. M. & Christensen, H. (2005) Website quality indicators for consumers. Journal of Medical Internet Research, 7, e55.CrossRefGoogle ScholarPubMed
National Institute for Health and Clinical Excellence (2001) Scope for the Development of a Clinical Guideline on the Management of Depression (http://www.nice.org.uk/nicemedia/pdf/Depression_Scope.pdf).Google Scholar
Powell, J. & Clarke, A. (2006) Internet information-seeking in mental health. British Journal of Psychiatry, 189, 273277.CrossRefGoogle ScholarPubMed
Silverstein, C., Henzinger, M., Marais, H., et al (1998) Analysis of a Very Large Alta Vista Query Log, SRC Technical Note 1998–014. Digital Systems Research Centre (http://gatekeeper.research.compaq.com/pub/DEC/SRC/technical-notes/abstracts/src-tn-1998-014.html).Google Scholar
Taylor, H. (1999) Explosive growth of ‘cyberchondriacs’ continues. The Harris Poll No. 47 (http://www.harrisinteractive.com/harris_poll/index.asp?PID=117).Google Scholar
Wahlbeck, K. & Adams, C. (1999) Beyond conflict of interest. Sponsored drug trials show more favourable outcomes. BMJ, 318, 465.Google ScholarPubMed
World Health Organization (2001) WHO Guide to Mental and Neurological Health in Primary Care. WHO (http://www.mentalneurologicalprimarycare.org/).Google Scholar
Figure 0

Table 1. Depression-Website Content Checklist of key points on websites on depression

Figure 1

Table 2. Details of websites (sponsorship and nature of organisation) with Depression-Website Content Checklist total scores (Ctot) and DISCERN total scores (Dtot)

Figure 2

Table 3. Descriptive and statistical analyses of websites on depression classified by absence/presence of sponsorship and type of organisation (public/private), using DWCC total score (Ctot) and DISCERN total score (Dtot)

Submit a response

eLetters

No eLetters have been published for this article.