© 2006 The Royal College of Psychiatrists
© 2006 The Royal College of Psychiatrists
Evaluation of a mental health website for teenagers
McGuinness Adolescent Unit, Manchester
McGuinness Adolescent Unit, Manchester
Department of Child Psychiatry, Royal Manchester Children’s Hospital, Hospital Road, Salford M27 4HA, e-mail: leopold.kroll{at}cmmc.nhs.uk
L.K. contributes to the management and development of the ru-ok.com website.
 TOP Abstract IntroductionMethodResultsDiscussionReferences |
|---|
AIM AND METHOD
Ru-ok.com is a recently developed website that includes a self-assessment questionnaire. The aim of this study was to evaluate the website and compare the self-assessment questionnaire with established screening questionnaires. A total of 105 teenagers from schools completed three paper-based questionnaires and the online ru-ok.com questionnaire.
RESULTS
The website receives 730 visits a week. Visits to the advice section and stories about mental health and relationships account for 35% of activity. Of the returned questionnaires, 80% were positive about the website. There were modest and expected correlations between the website questionnaire (RU-OK) and the Mood and Feelings (MFQ) and Strength and Difficulties (SDQ) questionnaires.
CLINICAL IMPLICATIONS
Internet-based self-assessment is feasible and acceptable to teenagers. Self-assessment of perceived need by teenagers may be a useful tool for tier one professionals, including teachers, general practitioners, school nurses, social workers and learning mentors.
|
TOPAbstractIntroductionMethodResultsDiscussionReferences |
|---|
A systematic review of health promotion in schools demonstrated the varying success of self-help programmes designed to change risky behaviour, such as smoking or substance misuse (Lister-Sharp et al, 1999). Internet and computer-based assessment and interventions offer an alternative method to be used alongside more traditional approaches. CD–ROMs have shown promise for changing behaviour and promoting better psychological health in the USA (Bosworth et al, 1994, 2000).
The internet is an accessible resource (McAlindon et al, 2003) and media, such as television, can provide information in a non-stigmatising way (Sanders et al, 2000). The internet also facilitates individual autonomy (Theodosiou & Green, 2003), which is an important part of adolescent development. The internet can also provide self-help assessment and interventions to people who do not seek help for their mental health problems (Marks et al, 2003; Christensen et al, 2004; Kenwright et al, 2004).
There is hardly any knowledge of how self-assessment questionnaires completed online compare with paper questionnaires (Ritter et al, 2004). Frequent concerns are that online data may not be secure or reliable because the rater may not understand the instructions, may not take as much care when completing the forms online, or may be influenced by peers to give higher or lower ratings if the online assessment takes place in too public an area, such as an open library, classroom or drop-in centre.
With these issues in mind, the website ru-ok.com was developed in 2003 with the aid of focus groups of teenagers. The research project was funded by the Health Foundation. Focus groups were positive about the final result and the site went live in 2003. The site allows teenagers to assess their strengths and weaknesses and obtain self-help advice. The site includes, among other psychological self-help material, a widely used educational resource, ‘Retracking’ (Bates, 1997) and a self-assessment questionnaire based on the Salford Needs Assessment Schedule for Adolescents (Kroll et al, 1999). There are a range of other features, including interactive stories, cartoons about depression, eating problems, teenage pregnancy and school problems, and some games.
The aims of this study were: (a) to examine whether teenagers find the website acceptable and useful; (b) to validate the RU–OK self-assessment questionnaire by comparing it with standard questionnaires.
|
TOPAbstractIntroductionMethodResultsDiscussionReferences |
|---|
Participants
The sample was drawn from a Salford comprehensive school and a special hospital school for children with depression and anxiety.
Measures
The RU–OK self-assessment questionnaire examines problems and
strengths of teenagers. The teenager, using a secure method, enters their
chosen user name and password. There are 26 questions about problems, each
question rated on a 5-point scale from 1 to 5, zero equating to ‘not
rated’. The questions are taken from the Salford Needs Assessment
Schedule for Adolescents (Kroll et
al, 1999). A score of 3 or more is judged to indicate a
significant problem and is based on validity data from the original research
study. Data can be saved at different time points and reviewed on revisiting
the site. The online questionnaire takes about 5–10 min to complete and
advice is available for each problem area depending on the score. Low scores
would usually lead to advice about self-help, such as using downloadable
material from the site or links to other recommended sites. Higher scores
would stress the importance of discussing problems with a mentor, parent or
professional.
A qualitative questionnaire asking about acceptability, ease of use and usefulness of parts of the website was administered. Each of the questions is rated on a 5-point scale.
The adolescent self-completion version of the Strength and Difficulties Questionnaire (SDQ; Goodman, 1997) consists of 25 items scored on a 3-point scale. In addition to a total score, there are sub-scales for difficulties in the areas of hyperactivity, conduct, emotions and social interactions. There is also one prosocial scale. For epidemiological research studies, cut-off points on each scale can be set in order to classify teenagers into positive and negative groups depending on the research requirements.
The adolescent Mood and Feelings Questionnaire (MFQ; Costello & Angold, 1988) is designed to screen for depression. It consists of 34 questions scored on a 3-point scale. A cut-off point of 27 is often used, giving a sensitivity and specificity of about 80% compared with a gold standard research interview.
Sub-scales from the RU–OK questionnaire were generated so that Pearson correlation could be assessed with the MFQ and SDQ. To do this, scores from certain areas were added together. The sub-scales were:
- violence: consisting of two areas, violence to property and persons
- emotional: consisting of seven areas, depression, anxiety, trauma,
self-harm, eating problems, self-esteem and paranoid thinking
- relationships: consisting of two areas, peer problems and family
problems
- drugs and alcohol: consisting of two areas, drugs and alcohol
- total score: consisting of all 26 areas.
The sub-scales and individual scores were checked to ensure normal statistical distribution.
Data collection and loss
The study received local ethical approval and was then explained to
teenagres by two research psychiatrists, S.E. and A.K., in a personal, social
and health education (PSHE) lesson. The parent or guardian was then informed
and teenagers completed consent forms. The following week, teenagers supplied
demographic details and completed the SDQ and MFQ. They were given a postcard
with a research identification number and instruction about the website. The
next PHSE lesson was held in the information technology room and teenagers
completed (in confidence) the RU–OK questionnaire and evaluation
questionnaire. The same method of data collection was used with the teenagers
who attended the hospital school.
Of the teenagers approached, only four did not agree to participate in the study. About 40% of evaluation questionnaires were not completed owing to lack of time in the lesson; the final numbers of questionnaires ranged from 64 to 67. A total of 105 teenagers completed the SDQ and 101 the MFQ questionnaires. A total of 103 completed the RU–OK self-assessment questionnaire but about 30% of the data were lost to analysis because of failure to track pupils’ private identification numbers on the questionnaires. The complete set of questionnaires for analysis of correlation between RU–OK and the SDQ and MFQ was thus 69.
|
TOPAbstractIntroductionMethodResultsDiscussionReferences |
|---|
Participants
A total of 105 teenagers were included in the final analysis, 46 boys and 59 girls; 91 teenagers were aged 13–14 years and came from year 9 in the comprehensive school; 14 teenagers came from the hospital school of which 13 were 14- to 15-year-olds (year 10 and 11), the other being from year 9.
Responses
Table 1 shows the responses
to the three questionnaires. The sub-scale scores for the SDQ are not
significantly different from the national SDQ norms
(http://www.sdqinfo.com).
The numbers of teenagers classified as having a significant problem on the MFQ
and the RU–OK questionnaire are also shown.
|
View this table: [in a new window] | Table 1. Responses of participants to the three questionnaires |
Correlation between the RU–OK questionnaire subscales, the MFQ and SDQ total and sub-scale scores are shown in Table 2. This shows modest correlations between scales. The MFQ, SDQ and RU–OK questionnaire emotional sub-scales and conduct sub-scales had better correlations as expected. We thus performed a standard statistical 2 x 2 table screening procedure of two areas (Warner, 2004), despite the small sample size. A cut-off point of 4 on the SDQ for emotional and conduct subscales was used. Using the SDQ emotional sub-scale as the gold standard, the RU–OK ‘depression’ sub-scale gave a sensitivity of 17.4% and specificity of 97%, with a positive predictive value of 80% and a negative predictive value of 70%. Confidence intervals were wide. Comparison with the MFQ showed much lower sensitivity and specificity. When the RU–OK ‘violence to others’ problem area was compared with the conduct sub-scale of the SDQ, the sensitivity was 48%, specificity 94%, positive predictive value 77% and negative predictive value 80%.
|
View this table: [in a new window] | Table 2. Correlations between MFQ, SDQ and RU–OK sub-scales |
The summary results of the questionnaire are shown in Table 3. The 5-point scales were reduced to two categories as shown. Excellent, good and OK were combined into ‘positive’, and poor and bad into ‘negative’. Table 3 shows that 70–80% of respondents were positive about the site. There were no differences between the main-stream school and the specialist hospital school.
|
View this table: [in a new window] | Table 3. Summary results of the questionnaire evaluating the ru-ok website |
We analysed website traffic; in the 20 months since the site went live there were 18 382 visits: 2956 people had logged on to the site and 640 completed the RU-OK questionnaire. Of activity on the site, 23% was dedicated to downloading advice and information files and 12% to looking at the cartoons about depression, eating disorders, school problems and pregnancy. On average, there are about 900 visitors per month, of which 150 logged on to the secure section of the site containing the online questionnaire. There continues to be a steady increase in activity; the number of visitors to the site is now about 730 per week.
|
TOPAbstractIntroductionMethodResultsDiscussionReferences |
|---|
This study shows that the ru-ok.com website was positively evaluated by a sample of representative teenagers aged 13–14 years. Website traffic indicates a steady rise in use, as more teenagers become aware of the existence of the site. The self-assessment questionnaire shows acceptable correlations with established paper-and-pencil screening questionnaires.
Adolescent mental health problems in school are increasingly recognised and questionnaires can be used to screen for these (Stansfeld et al, 2004). The advantage of the RU–OK questionnaire is that it is based on a needs assessment methodology; the paper version (Kroll et al, 1999) has good face validity and acceptability with teenagers. The RU–OK questionnaire does not require any summary analysis to interpret the results or to obtain downloadable advice and information about interventions. Some questionnaires, however, can be misused and misunderstood (Clark & Harrington 1999; Warner, 2004). Many also require scoring and interpretation.
Shortcomings of this study include the limited age range of teenagers studied and the small sample size. Thus, formal screening analysis and receiver operator curve analysis was not possible. Ideally, the RU–OK questionnaire should be compared with a gold standard diagnostic interview and other observer-rated methods using a larger sample of teenagers.
Schools face an increasing challenge of first identifying pupils with problems and then delivering acceptable interventions at an appropriate tiered level. Stepped care models are being proposed for certain conditions (National Institute for Clinical Excellence, 2004), and it would be useful to study further the role of screening questionnaires and self-assessment in schools who use a stepped care approach.
We thank the pupils and staff of the two schools who participated in this study, the Health Foundation who funded the website development, and the website development research team, Jenny Bates, Aaron Davidson, Leo Kroll, John Rowley, Hilary Wait and Heather Sacco.
|
TOPAbstractIntroductionMethodResultsDiscussionReferences |
|---|
- BATES, J. (1997) Retracking. Torquay: Devon Education Authority, Devon Learning Resources.
- BOSWORTH, K., GUSTAFSON, D. H. & HAWKINS, R. P. (BARN research group) (1994) The BARN system: use and impact of adolescent health promotion via computer. Computing and Human Behaviour, 10, 467 –482[CrossRef]
- BOSWORTH, K., ESPELAGE, D., DUBAY, T., et al (2000) Preliminary evaluation of a multimedia violence prevention program for adolescents. American Journal of Health and Behavior, 24, 268 –280
- CHRISTENSEN, H., GRIFFITHS, K. M. & JORM, A. F.
(2004) Delivering interventions for depression by using the
internet: a randomized controlled trial. BMJ,
328, 265
–268.
[Abstract/Free Full Text] - CLARK, A. & HARRINGTON, R. C. (1999) On diagnosing rare disorders rarely: Appropriate use of screening instruments. Journal of Child Psychology and Psychiatry, 40, 287 –290.[CrossRef][Medline]
- COSTELLO, E. J. & ANGOLD, A. (1988) Scales to assess child and adolescent depression: checklists, screens and nets. Journal of the American Academy of Child and Adolescent Psychiatry, 27, 726 –737.[Medline]
- GOODMAN, R. (1997) The Strengths and Difficulties Questionnaire: A research note. Journal of Child Psychology and Psychiatry, 38, 581 –586.[Medline]
- KENWRIGHT, M., MARKS, I. M., GEGA, L., et al
(2004) Computer aided self-help for phobia/panic via internet at
home: a pilot study. British Journal of Psychiatry,
184, 448
–449.
[Abstract/Free Full Text] - KROLL, L., WOODHAM, A., ROTHWELL, J., et al (1999) The reliability of the Salford Needs Assessment Schedule for Adolescents. Psychological Medicine, 29, 891 –902.[CrossRef][Medline]
- LISTER-SHARP, D., CHAPMAN, S., STEWART-BROWN, S., et al (1999) Health promoting schools and health promotion in schools: two systematic reviews. Health Technology Assessment, 3, 22.
- MARKS, I. M., MATAIX-COLS, D., KENWRIGHT, M., et al.
(2003) Pragmatic evaluation of computer-aided self-help for
anxiety and depression. British Journal of Psychiatry,
183, 57
–65.
[Abstract/Free Full Text] - McALINDON, T., FORMIC, M., KABBARA, K., et al
(2003) Conducting clinical trials over the internet: feasibility
study. BMJ, 327, 484
–487.
[Free Full Text] - NATIONAL INSTITUTE FOR CLINICAL EXCELLENCE (2004) Depression: Management of Depression in Primary and Secondary Care. (Clinical Guideline 23). London: NICE.
- RITTER, P., LORIG, K., LAURENT, D., et al (2004) Internet versus mailed questionnaires: A randomized comparison.Journal of Medical Internet Research, 6, 29.
- SANDERS, M. R., MONTGOMERY, D.T. & BRECHMAN-TOUSSAINT, M.L. (2000) The mass media and the prevention of child behaviour problems: the evaluation of a television series to promote positive outcomes for parents and their children. Journal of Child Psychology and Psychiatry, 41, 939 –948.[CrossRef][Medline]
- STANSFELD, S. A., HAINES, M. M., HEAD, J. A., et al
(2004) Ethnicity, social deprivation and psychological distress
in adolescents: School-based epidemiological study in east London.
British Journal of Psychiatry,
185, 233
–238.
[Abstract/Free Full Text] - THEODOSIOU, L. & GREEN, J. (2003) Emerging
challenges in using health information from the internet. Advances
in Psychiatric Treatment, 9, 387
–396.
[Abstract/Free Full Text] - WARNER, J. (2004) Clinicians’guide to evaluating
diagnostic and screening tests in psychiatry. Advances in
Psychiatric Treatment, 10, 446
–454.
[Abstract/Free Full Text]
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||