© 2008 The Royal College of Psychiatrists
The quality of internet information on depression for lay people
*SW London and St George’s NHS Mental Health Trust, London, UK, email: phillip.ferreira-lay{at}swlstg-tr.nhs.uk
Springfield University Hospital, SW London and St George’s NHS Mental Health Trust, London, UK
 TOP Abstract IntroductionMethodResultsDiscussionLimitationsReferences |
|---|
AIMS AND METHOD
To examine the quality of information concerning depression available on the internet, identify factors associated with ‘good’ quality, and develop a simple-to-use instrument for assessing websites on depression. The Depression-Website Content Checklist was developed and compared with a previously validated, yet too complex, scale. Websites were assessed using both instruments.
RESULTS
Good quality information is more likely to be found on websites provided by governmental, professional and charitable organisations. The differences we observed in the median scores for these websites using the Depression-Website Content Checklist are significant at the 0.05 level (Ctot; Mann-Whitney, U=24.00; P=0.013). The Depression-Website Content Checklist is a valid and reliable user-friendly tool.
CLINICAL IMPLICATIONS
Patients can be directed towards better quality information by diverting them to public and non-sponsored websites. Clinicians can use the Depression-Website Content Checklist to determine website quality.
|
TOPAbstractIntroductionMethodResultsDiscussionLimitationsReferences |
|---|
Eighteen per cent of all internet users have sought mental health-related information (Powell & Clarke, 2006) and they are more likely to search for information on depression than on any other condition (Taylor, 1999). However, information available on the internet has been reported to be of variable quality (Bower, 1996). Therefore, clinicians may have a role in helping the public find better quality websites.
Website sponsorship by organisations such as drug companies may be one factor contributing to disseminating ‘poor’ quality information (Christensen et al, 2000) and the public needs guidance on where to find reliable information on health issues. International initiatives such as DISCERN attempt to apply quality control to internet-based health information (Charnock et al, 1999; Gagliardi & Jadad, 2002). However, DISCERN and other currently available instruments are complex and unwieldy, a user-friendly rating tool is needed.
|
TOPAbstractIntroductionMethodResultsDiscussionLimitationsReferences |
|---|
Websites on depression available on the internet at the time of the research were identified using the single search term ‘depression’ on multiple search engines. The UK and other English language websites were targeted to get an international perspective. Information available on the SearchEngineWatch.com website identifies the most commonly used search engines and eight of them were used for this research.
It is rare for people to look beyond the first ten results of a query (Silverstein et al, 1998), thus our analysis was limited to the first ten sites listed by each of the eight search engines. The sites were excluded if the website: had no depression content; focused more on other forms of affective illness; was not in English; was linked solely to another search engine; or focused on the non-adult population (i.e. those under 16 years of age).
Two key documents were analysed for common factors suitable for converting into a simple measuring scale: the National Institute for Health and Clinical Excellence (NICE) draft document (NICE, 2001) and the World Health Organization Mental Health Checklist (WHO, 2001). From these, a ten-item scale was developed (Table 1). Each item attracts one mark and therefore all items are given equal weighting. The items address illness domains - aetiology, symptoms, treatment and prognosis. Summation of item marks gives a total score (Ctot); the maximum possible score is 10 and the minimum is 0.
|
View this table: [in a new window] | Table 1. Depression-Website Content Checklist of key points on websites on depression |
The quality of websites on depression was then assessed using both the Depression-Website Content Checklist and DISCERN - a previously validated instrument used to judge the quality of written healthcare information best applied by ‘experienced users’ (Charnock et al, 1999). The DISCERN instrument generates a total score (Dtot, maximum 75) for 15 key criteria. Scores were calculated for each website.
In addition to their individual quality, websites were compared according to two grouping variables, that is sponsored (yes) and non-sponsored (no). Evidence of financial support for the development and/or content of each of the websites was sought. A statement of such support on any of a website’s pages resulted in allocation of that website to the ‘yes’ group, otherwise it was allocated to the ‘no’ group; the Mann-Whitney U-test (the non-parametric equivalent of a t-test) was used to compare the websites grouped in this way. They were also grouped according to organisation type - public organisations (governmental/professional bodies and charities) and private organisations (drug companies and individuals) - and analyses of their respective websites were performed.
Validating the Depression-Website Content Checklist
As DISCERN had been previously validated, correlation between it and the
Depression-Website Content Checklist was measured to ascertain concurrent
validity. Both are ordinal scales, hence Spearman’s rank correlation
coefficient was used. Internal scale consistency was measured using
Cronbach’s 
. Each website was rated by a second-rater (S. M.)
oblivious to previous scores and interrater reliability for the
Depression-Website Content Checklist was measured using the intraclass
correlation coefficient (ICC).
|
TOPAbstractIntroductionMethodResultsDiscussionLimitationsReferences |
|---|
We identified 68 different website addresses by analysing the first ten sites listed at each of the eight search engines when using the search term ‘depression’. In total, 39 websites were excluded according to previously stated criteria. The main reason for exclusion was that the site did not contain any depression content. There were 29 websites remaining for further investigation. However, several of those websites linked to the same website; for example, three addresses linked to the website for the National Institute for Mental Health. We were finally left with 23 websites to be further analysed (Table 2). Examination of the websites using the two instruments showed them to be of variable quality (range 2 - 10 on the Depression-Website Content Checklist, 24 - 74 on DISCERN).
|
View this table: [in a new window] | Table 2. Details of websites (sponsorship and nature of organisation) with Depression-Website Content Checklist total scores (Ctot) and DISCERN total scores (Dtot) |
Identifying factors that appear to predict quality
Comparison of the sponsored and non-sponsored websites is illustrated in
Table 3.
|
View this table: [in a new window] | Table 3. Descriptive and statistical analyses of websites on depression classified by absence/presence of sponsorship and type of organisation (public/private), using DWCC total score (Ctot) and DISCERN total score (Dtot) |
Non-sponsored websites have higher median scores than sponsored websites when the two ratings Ctot and Dtot were used (Table 3). Observed differences are significant at the 0.05 level (Ctot;Mann-Whitney U=25.500; P=0.024).
Comparison of the websites when grouped according to whether they belonged to public or private organisations is illustrated in Table 3. Public organisations have higher median scores for Ctot and Dtot. Observed differences are significant at the 0.05 level (Ctot;Mann-Whitney U=24.00; P=0.013).
Assessment of the Depression-Website Content Checklist
There was a high degree of positive correlation between the
Depression-Website Content Checklist total score for each website and the
corresponding DISCERN total score (Spearmans r=0.935, significant at
0.01 level, two-tailed). Cronbach’s =0.857 for the
Depression-Website Content Checklist and =0.935 for the DISCERN tool.
These values (>0.8) indicate a high level of positive correlation
between each of the items on both scales. The intraclass correlation
coefficient was 0.85 (CI 95% 0.66-0.94) indicating very good agreement between
raters when using the Depression-Website Content Checklist to assess the
quality of websites.
|
TOPAbstractIntroductionMethodResultsDiscussionLimitationsReferences |
|---|
We aimed to assess the quality of information on depression available to lay internet users. Although other authors (Griffiths & Christensen, 2005) have also examined the quality of websites on depression, they were not previously ranked according to their popularity and the multiple search engines had not been employed. Our study demonstrates two things: first, that public, non-sponsored information on depression is of better quality than private, sponsored information available on the internet; and second, a simple user-friendly rating instrument (Depression-Website Content Checklist) can be used to assess the quality of websites.
Overall the quality of websites we analysed was extremely variable. The large proportion of websites excluded in this study (45/68, i.e. >65%) indicates the quality of information on depression on many of the most commonly accessed websites is rather poor. Many websites initially identified by search engines as containing information on depression did not actually have such information, which shows that individuals may be directed to a number of irrelevant sites. Several sites linked to the same provider, for instance the Royal College of Psychiatrists or the National Institute for Mental Health. This indicates that users may access the same information in different ways, but lack of clarity in accessing better quality information may ensue.
For websites with depression content it was possible to identify factors associated with ‘good’ quality. Our data indicate that the absence of sponsorship is associated with better quality information. Statistically significant differences were found when non-sponsored sites were compared with sponsored ones. Analysis of the content of the eight sponsored websites showed that all were provided by drug companies. Possible bias in favour of a drug company’s own advocated treatment has been noted elsewhere (Wahlbeck & Adams, 1999). One might suggest that the poorer quality information on these websites compared with others is related to businesses wishing to promote their own products. This is not a concern of non-sponsored website providers. Such organisations include the National Health Service in the UK and the National Institute for Mental Health in the USA, which both aim to provide comprehensive information on health. These sites achieved the highest ranking scores on DISCERN and Depression-Website Content Checklist.
Other authors (Griffiths & Christensen, 2005) have also found non-sponsored sites to be of higher quality. Further research would help clarify to what extent findings from the UK apply across the rest of the English-speaking world. It is reasonable to assume that users are likely to access both national/regional and international websites.
In order to simplify the assessment of quality of internet information on
depression we developed a simple-to-use Depression-Website Content Checklist
that we also evaluated. Apart from its high degree of utility the instrument
has good internal consistency (Cronbach’s >0.8 is comparable
with a rating instrument suitable for making clinical decisions). It remains
to be seen, however, how valid and reliable such simple instruments are.
As far as validity is concerned, both the Depression-Website Content Checklist and DISCERN have been shown to provide concurrent results. Moreover, as the Checklist’s content was taken from the World Health Organization, its construct and face validity are also significant. It also possesses a high interrater reliability such that different users of the instrument are likely to reach the same conclusions about the quality of any given website. Since a certain degree of experience is needed to use DISCERN, some clinicians may prefer to use the Depression-Website Content Checklist.
In future this methodology might be employed to develop a similar scale for assessing internet information on other major mental illness such as schizophrenia. It would be relatively simple to devise item scores for website information on other illnesses, based around aetiology, symptoms, treatment and prognosis.
The instrument developed in our study concentrates on identifying the quality of information on depression on each of the websites we analysed. We defined the comprehensiveness of a website by the extent to which it displays the ten items of illness information that make up the Checklist. However, there may be other things about the websites such as their layout and design, which make them more, or less, helpful. A poorly designed website may make the information difficult to read and absorb, therefore reducing clarity. These factors were not considered in this study (errors of commission). Rather, we have been able to accurately identify those websites on depression that contained good quality information by checking how they score on the ten-item checklist.
We conclude that healthcare professionals have a key role in directing service users to good quality (public, non-sponsored) websites and the Depression-Website Content Checklist may help them do that.
|
TOPAbstractIntroductionMethodResultsDiscussionLimitationsReferences |
|---|
A small number of websites on depression were analysed and investigations focused on sites accessible by UK users. One of the authors (P.F.-L.) undertook the search for relevant websites on depression and rated them in turn, which means that the analyses are prone to rater bias. Also, we only looked at whether the items of the rating instrument were present/absent in each of the sites and did not examine the rest of the site content. Therefore, although the information we gathered is accurate and measured by our instrument, other aspects of the site (e.g. layout and design) were not analysed.
|
TOPAbstractIntroductionMethodResultsDiscussionLimitationsReferences |
|---|
- BOWER, H. (1996) Internet sees growth of unverified
health claims. BMJ, 313, 318
.
[Free Full Text] - CHARNOCK, D., SHEPPERD, S., NEEDHAM, G., et al (1999) DISCERN: an instrument for judging the quality of written consumer health information on treatment choices. Journal of Epidemiology and Community Health, 53, 105 -111.[Abstract]
- CHRISTENSEN, H., GRIFFITHS, K. M. & MEDWAY, J. (2000) Sites for depression on the Web: a comparison of consumer, professional and commercial sites. Australian and New Zealand Journal of Public Health, 24, 396 -400.[Medline]
- GAGLIARDI, A. & JADAD, A. R. (2002) Examination of
instruments used to rate quality of health information on the Internet:
chronicle of a voyage with an unclear destination.
BMJ, 324, 569
-573.
[Abstract/Free Full Text] - GRIFFITHS, K. M. & CHRISTENSEN, H. (2005) Website quality indicators for consumers. Journal of Medical Internet Research, 7, e55 .[Medline]
- NATIONAL INSTITUTEFOR HEALTH AND CLINICAL EXCELLENCE (2001) Scope for the Development of a Clinical Guideline on the Management of Depression (http://www.nice.org.uk/nicemedia/pdf/Depression_Scope.pdf).
- POWELL, J. & CLARKE, A. (2006) Internet
information-seeking in mental health. British Journal of
Psychiatry, 189, 273
-277.
[Abstract/Free Full Text] - SILVERSTEIN, C., HENZINGER, M., MARAIS, H., et al (1998) Analysis of a Very Large AltaVista Query Log, SRC Technical Note 1998-014. Digital Systems Research Centre (http://gatekeeper.research.compaq.com/pub/DEC/SRC/technical-notes/abstracts/src-tn-1998-014.html).
- TAYLOR, H. (1999) Explosive growth of ‘cyberchondriacs’ continues. The Harris Poll No. 47 (http://www.harrisinteractive.com/harris_poll/index.asp?PID=117).
- WAHLBECK, K. & ADAMS, C. (1999) Beyond conflict of interest. Sponsored drug trials show more favourable outcomes. BMJ, 318, 465 .[Medline]
- WORLD HEALTH ORGANIZATION (2001) WHO Guide to Mental and Neurological Health in Primary Care. WHO (http://www.mentalneurologicalprimarycare.org/).
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||