Hostname: page-component-7c8c6479df-995ml Total loading time: 0 Render date: 2024-03-28T05:46:31.956Z Has data issue: false hasContentIssue false

Evaluation of a mental health website for teenagers

Published online by Cambridge University Press:  02 January 2018

Sukru Ercan
Affiliation:
McGuinness Adolescent Unit, Manchester
Andrew Kevern
Affiliation:
McGuinness Adolescent Unit, Manchester
Leo Kroll
Affiliation:
Department of Child Psychiatry, Royal Manchester Children's Hospital, Hospital Road, Salford M27 4HA, e-mail: leopold.kroll@cmmc.nhs.uk
Rights & Permissions [Opens in a new window]

Abstract

Aim and Method

Ru-ok.com is a recently developed website that includes a self-assessment questionnaire. The aim of this study was to evaluate the website and compare the self-assessment questionnaire with established screening questionnaires. A total of 105 teenagers from schools completed three paper-based questionnaires and the online ru-ok.com questionnaire.

Results

The website receives 730 visits a week. Visits to the advice section and stories about mental health and relationships account for 35% of activity. Of the returned questionnaires, 80% were positive about the website. There were modest and expected correlations between the website questionnaire (RU–OK) and the Mood and Feelings (MFQ) and Strength and Difficulties (SDQ) questionnaires.

Clinical Implications

Internet-based self-assessment is feasible and acceptable to teenagers. Self-assessment of perceived need by teenagers may be a useful tool for tier one professionals, including teachers, general practitioners, school nurses, social workers and learning mentors.

Type
Original papers
Creative Commons
Creative Common License - CCCreative Common License - BY
This is an Open Access article, distributed under the terms of the Creative Commons Attribution (CC-BY) license (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted re-use, distribution, and reproduction in any medium, provided the original work is properly cited.
Copyright
Copyright © Royal College of Psychiatrists, 2006

A systematic review of health promotion in schools demonstrated the varying success of self-help programmes designed to change risky behaviour, such as smoking or substance misuse (Reference Lister-Sharp, Chapman and Stewart-BrownLister-Sharp et al, 1999). Internet and computer-based assessment and interventions offer an alternative method to be used alongside more traditional approaches. CD–ROMs have shown promise for changing behaviour and promoting better psychological health in the USA (Bosworth et al, Reference Bosworth, Gustafson and Hawkins1994, Reference Bosworth, Espelage and Dubay2000).

The internet is an accessible resource (Reference McAlindon, Formic and KabbaraMcAlindon et al, 2003) and media, such as television, can provide information in a non-stigmatising way (Reference Sanders, Montgomery and Brechman-ToussaintSanders et al, 2000). The internet also facilitates individual autonomy (Reference Theodosiou and GreenTheodosiou & Green, 2003), which is an important part of adolescent development. The internet can also provide self-help assessment and interventions to people who do not seek help for their mental health problems (Reference Marks, Mataix-Cols and KenwrightMarks et al, 2003; Reference Christensen, Griffiths and JormChristensen et al, 2004; Reference Kenwright, Marks and GegaKenwright et al, 2004).

There is hardly any knowledge of how self-assessment questionnaires completed online compare with paper questionnaires (Reference Ritter, Lorig and LaurentRitter et al, 2004). Frequent concerns are that online data may not be secure or reliable because the rater may not understand the instructions, may not take as much care when completing the forms online, or may be influenced by peers to give higher or lower ratings if the online assessment takes place in too public an area, such as an open library, classroom or drop-in centre.

With these issues in mind, the website ru-ok.com was developed in 2003 with the aid of focus groups of teenagers. The research project was funded by the Health Foundation. Focus groups were positive about the final result and the site went live in 2003. The site allows teenagers to assess their strengths and weaknesses and obtain self-help advice. The site includes, among other psychological self-help material, a widely used educational resource, ‘Retracking’ (Reference BatesBates, 1997) and a self-assessment questionnaire based on the Salford Needs Assessment Schedule for Adolescents (Reference Kroll, Woodham and RothwellKroll et al, 1999). There are a range of other features, including interactive stories, cartoons about depression, eating problems, teenage pregnancy and school problems, and some games.

The aims of this study were: (a) to examine whether teenagers find the website acceptable and useful; (b) to validate the RU–OK self-assessment questionnaire by comparing it with standard questionnaires.

Method

Participants

The sample was drawn from a Salford comprehensive school and a special hospital school for children with depression and anxiety.

Measures

The RU–OK self-assessment questionnaire examines problems and strengths of teenagers. The teenager, using a secure method, enters their chosen user name and password. There are 26 questions about problems, each question rated on a 5-point scale from 1 to 5, zero equating to ‘not rated’. The questions are taken from the Salford Needs Assessment Schedule for Adolescents (Reference Kroll, Woodham and RothwellKroll et al, 1999). A score of 3 or more is judged to indicate a significant problem and is based on validity data from the original research study. Data can be saved at different time points and reviewed on revisiting the site. The online questionnaire takes about 5–10 min to complete and advice is available for each problem area depending on the score. Low scores would usually lead to advice about self-help, such as using downloadable material from the site or links to other recommended sites. Higher scores would stress the importance of discussing problems with a mentor, parent or professional.

A qualitative questionnaire asking about acceptability, ease of use and usefulness of parts of the website was administered. Each of the questions is rated on a 5-point scale.

The adolescent self-completion version of the Strength and Difficulties Questionnaire (SDQ; Reference GoodmanGoodman, 1997) consists of 25 items scored on a 3-point scale. In addition to a total score, there are sub-scales for difficulties in the areas of hyperactivity, conduct, emotions and social interactions. There is also one prosocial scale. For epidemiological research studies, cut-off points on each scale can be set in order to classify teenagers into positive and negative groups depending on the research requirements.

The adolescent Mood and Feelings Questionnaire (MFQ; Reference Costello and AngoldCostello & Angold, 1988) is designed to screen for depression. It consists of 34 questions scored on a 3-point scale. A cut-off point of 27 is often used, giving a sensitivity and specificity of about 80% compared with a gold standard research interview.

Sub-scales from the RU–OK questionnaire were generated so that Pearson correlation could be assessed with the MFQ and SDQ. To do this, scores from certain areas were added together. The sub-scales were:

  1. violence: consisting of two areas, violence to property and persons

  2. emotional: consisting of seven areas, depression, anxiety, trauma, self-harm, eating problems, self-esteem and paranoid thinking

  3. relationships: consisting of two areas, peer problems and family problems

  4. drugs and alcohol: consisting of two areas, drugs and alcohol

  5. total score: consisting of all 26 areas.

The sub-scales and individual scores were checked to ensure normal statistical distribution.

Data collection and loss

The study received local ethical approval and was then explained to teenagres by two research psychiatrists, S.E. and A.K., in a personal, social and health education (PSHE) lesson. The parent or guardian was then informed and teenagers completed consent forms. The following week, teenagers supplied demographic details and completed the SDQ and MFQ. They were given a postcard with a research identification number and instruction about the website. The next PHSE lesson was held in the information technology room and teenagers completed (in confidence) the RU–OK questionnaire and evaluation questionnaire. The same method of data collection was used with the teenagers who attended the hospital school.

Of the teenagers approached, only four did not agree to participate in the study. About 40% of evaluation questionnaires were not completed owing to lack of time in the lesson; the final numbers of questionnaires ranged from 64 to 67. A total of 105 teenagers completed the SDQ and 101 the MFQ questionnaires. A total of 103 completed the RU–OK self-assessment questionnaire but about 30% of the data were lost to analysis because of failure to track pupils’ private identification numbers on the questionnaires. The complete set of questionnaires for analysis of correlation between RU–OK and the SDQ and MFQ was thus 69.

Results

Participants

A total of 105 teenagers were included in the final analysis, 46 boys and 59 girls; 91 teenagers were aged 13–14 years and came from year 9 in the comprehensive school; 14 teenagers came from the hospital school of which 13 were 14- to 15-year-olds (year 10 and 11), the other being from year 9.

Responses

Table 1 shows the responses to the three questionnaires. The sub-scale scores for the SDQ are not significantly different from the national SDQ norms (http://www.sdqinfo.com). The numbers of teenagers classified as having a significant problem on the MFQ and the RU–OK questionnaire are also shown.

Table 1. Responses of participants to the three questionnaires

Questionnaire Mean (s.d.) Below cut-off, n Above cut-off, n
SDQ, n=105
     Total score 11.8 (5.2)
     Emotional sub-scale 2.6 (2.0)
     Conduct 2.7 (1.7)
     Hyperactivity 4.3 (2.3)
     Peer problems 2.1 (1.6)
     Prosocial score 6.0 (1.7)
MFQ, n=101
     Total score 91 (10)
RU-OK, n=103
     Looking after self 91 10
     Cooking 85 16
     Physical illness 95 6
     School attendance 93 8
     School work 89 12
     Employment 4 7
     Peer 91 10
     Family 88 13
     Cultural/racial 94 3
     Sexual activity 94 3
     Alcohol 100 1
     Drugs 95 4
     Depression 87 14
     Self-harm 96 5
     Anxiety 91 11
     Self-esteem 87 14
     Bullying 96 5
     Anorexia/bulimia 96 5
     Psychological trauma 97 4
     Sexual health/knowledge 97 4
     Paranoid thinking 99 2
     Money 91 10
     Neighbourhood 91 10
     Leisure activities 91 10
     Violence to others 87 14
     Defiance, making trouble 94 7

Correlation between the RU–OK questionnaire subscales, the MFQ and SDQ total and sub-scale scores are shown in Table 2. This shows modest correlations between scales. The MFQ, SDQ and RU–OK questionnaire emotional sub-scales and conduct sub-scales had better correlations as expected. We thus performed a standard statistical 2 × 2 table screening procedure of two areas (Reference WarnerWarner, 2004), despite the small sample size. A cut-off point of 4 on the SDQ for emotional and conduct subscales was used. Using the SDQ emotional sub-scale as the gold standard, the RU–OK ‘depression’ sub-scale gave a sensitivity of 17.4% and specificity of 97%, with a positive predictive value of 80% and a negative predictive value of 70%. Confidence intervals were wide. Comparison with the MFQ showed much lower sensitivity and specificity. When the RU–OK ‘violence to others’ problem area was compared with the conduct sub-scale of the SDQ, the sensitivity was 48%, specificity 94%, positive predictive value 77% and negative predictive value 80%.

Table 2. Correlations between MFQ, SDQ and RU–OK sub-scales

SDQ sub-scales and total score n=105 MFQ n=101 RU—OK sub-scales and total score n=69
Hyperactivity Conduct Emotional Prosocial Peer problems SDQ total score MFQ total score RU-OK total score Emotional sub-scale Violence sub-scale Relationships sub-scale Drugs/alcohol
SDQ
     Hyperactivity 1
     Conduct 0.565** 1
     Emotional 0.283** 0.170 1
     Prosocial -0.293** -0.337** 0.090 1
     Peer problems 0.102 0.074 0.415** 0.034 1
     Total score 0.770** 0.672** 0.694** -0.197* 0.543** 1
MFQ total score 0.337** 0.231* 0.584** 0.110 0.347** 0.557** 1
RU—OK
     Total score 0.196 0.148 0.275* -0.043 0.442** 0.387** 0.373** 1
     Emotional 0.131 0.015 0.420** 0.016 0.534** 0.403** 0.537** 0.814** 1
     Violence 0.329** 0.337** 0.051 -0.310** 0.157 0.326** 0.188 0.690** 0.422** 1
     Relationship 0.107 0.057 0.173 0.058 0.451** 0.281* 0.278* 0.822** 0.575** 0.469** 1
     Drug and alcohol 0.313** -0.280* 0.017 -0.134 -0.067 0.215 0.180 0.260** 0.012 0.222* 0.218* 1

The summary results of the questionnaire are shown in Table 3. The 5-point scales were reduced to two categories as shown. Excellent, good and OK were combined into ‘positive’, and poor and bad into ‘negative’. Table 3 shows that 70–80% of respondents were positive about the site. There were no differences between the main-stream school and the specialist hospital school.

Table 3. Summary results of the questionnaire evaluating the ru-ok website

Positive view Negative view Total n
Evaluation of site n % n %
Navigation of site 59 88.1 8 11.9 67
Self-assessment area 52 78.8 14 21.2 66
Your overall opinion of site 58 86.6 9 13.4 67
Graphics/pictures on site 56 87.5 8 12.5 64
Customising the site 57 85.1 10 14.9 67
Games 50 76.9 15 23.1 65
Interactive stories 54 83.1 11 16.9 65

We analysed website traffic; in the 20 months since the site went live there were 18 382 visits: 2956 people had logged on to the site and 640 completed the RU-OK questionnaire. Of activity on the site, 23% was dedicated to downloading advice and information files and 12% to looking at the cartoons about depression, eating disorders, school problems and pregnancy. On average, there are about 900 visitors per month, of which 150 logged on to the secure section of the site containing the online questionnaire. There continues to be a steady increase in activity; the number of visitors to the site is now about 730 per week.

Discussion

This study shows that the ru-ok.com website was positively evaluated by a sample of representative teenagers aged 13–14 years. Website traffic indicates a steady rise in use, as more teenagers become aware of the existence of the site. The self-assessment questionnaire shows acceptable correlations with established paper-and-pencil screening questionnaires.

Adolescent mental health problems in school are increasingly recognised and questionnaires can be used to screen for these (Reference Stansfeld, Haines and HeadStansfeld et al, 2004). The advantage of the RU–OK questionnaire is that it is based on a needs assessment methodology; the paper version (Reference Kroll, Woodham and RothwellKroll et al, 1999) has good face validity and acceptability with teenagers. The RU–OK questionnaire does not require any summary analysis to interpret the results or to obtain downloadable advice and information about interventions. Some questionnaires, however, can be misused and misunderstood (Reference Clark and HarringtonClark & Harrington 1999; Reference WarnerWarner, 2004). Many also require scoring and interpretation.

Shortcomings of this study include the limited age range of teenagers studied and the small sample size. Thus, formal screening analysis and receiver operator curve analysis was not possible. Ideally, the RU–OK questionnaire should be compared with a gold standard diagnostic interview and other observer-rated methods using a larger sample of teenagers.

Schools face an increasing challenge of first identifying pupils with problems and then delivering acceptable interventions at an appropriate tiered level. Stepped care models are being proposed for certain conditions (National Institute for Clinical Excellence, 2004), and it would be useful to study further the role of screening questionnaires and self-assessment in schools who use a stepped care approach.

Declaration of interest

L.K. contributes to the management and development of the ru-ok.com website.

Acknowledgements

We thank the pupils and staff of the two schools who participated in this study, the Health Foundation who funded the website development, and the website development research team, Jenny Bates, Aaron Davidson, Leo Kroll, John Rowley, Hilary Wait and Heather Sacco.

References

Bates, J. (1997) Retracking. Torquay: Devon Education Authority, Devon Learning Resources.Google Scholar
Bosworth, K., Gustafson, D. H. & Hawkins, R. P. (BARN research group) (1994) The BARN system: use and impact of adolescent health promotion via computer. Computing and Human Behaviour, 10, 467482 Google Scholar
Bosworth, K., Espelage, D., Dubay, T., et al (2000) Preliminary evaluation of a multimedia violence prevention program for adolescents. American Journal of Health and Behavior, 24, 268280 Google Scholar
Christensen, H., Griffiths, K. M. & Jorm, A. F. (2004) Delivering interventions for depression by using the internet: a randomized controlled trial. BMJ, 328, 265268.Google Scholar
Clark, A. & Harrington, R. C. (1999) On diagnosing rare disorders rarely: Appropriate use of screening instruments. Journal of Child Psychology and Psychiatry, 40, 287290.Google Scholar
Costello, E. J. & Angold, A. (1988) Scales to assess child and adolescent depression: checklists, screens and nets. Journal of the American Academy of Child and Adolescent Psychiatry, 27, 726737.Google Scholar
Goodman, R. (1997) The Strengths and Difficulties Questionnaire: A research note. Journal of Child Psychology and Psychiatry, 38, 581586.Google Scholar
Kenwright, M., Marks, I. M., Gega, L., et al (2004) Computer aided self-help for phobia/panic via internet at home: a pilot study. British Journal of Psychiatry, 184, 448449.Google Scholar
Kroll, L., Woodham, A., Rothwell, J., et al (1999) The reliability of the Salford Needs Assessment Schedule for Adolescents. Psychological Medicine, 29, 891902.Google Scholar
Lister-Sharp, D., Chapman, S., Stewart-Brown, S., et al (1999) Health promoting schools and health promotion in schools: two systematic reviews. Health Technology Assessment, 3, 22.CrossRefGoogle ScholarPubMed
Marks, I. M., Mataix-Cols, D., Kenwright, M., et al. (2003) Pragmatic evaluation of computer-aided self-help for anxiety and depression. British Journal of Psychiatry, 183, 5765.Google Scholar
McAlindon, T., Formic, M., Kabbara, K., et al (2003) Conducting clinical trials over the internet: feasibility study. BMJ, 327, 484487.CrossRefGoogle ScholarPubMed
National Institute for Clinical Excellence (2004) Depression: Management of Depression in Primary and Secondary Care. (Clinical Guideline 23). London: NICE.Google Scholar
Ritter, P., Lorig, K., Laurent, D., et al (2004) Internet versus mailed questionnaires: A randomized comparison. Journal of Medical Internet Research, 6, 29.Google Scholar
Sanders, M. R., Montgomery, D.T. & Brechman-Toussaint, M.L. (2000) The mass media and the prevention of child behaviour problems: the evaluation of a television series to promote positive outcomes for parents and their children. Journal of Child Psychology and Psychiatry, 41, 939948.Google Scholar
Stansfeld, S. A., Haines, M. M., Head, J. A., et al (2004) Ethnicity, social deprivation and psychological distress in adolescents: School-based epidemiological study in east London. British Journal of Psychiatry, 185, 233238.Google Scholar
Theodosiou, L. & Green, J. (2003) Emerging challenges in using health information from the internet. Advances in Psychiatric Treatment, 9, 387396.CrossRefGoogle Scholar
Warner, J. (2004) Clinicians' guide to evaluating diagnostic and screening tests in psychiatry. Advances in Psychiatric Treatment, 10, 446454.Google Scholar
Figure 0

Table 1. Responses of participants to the three questionnaires

Figure 1

Table 2. Correlations between MFQ, SDQ and RU–OK sub-scales

Figure 2

Table 3. Summary results of the questionnaire evaluating the ru-ok website

Submit a response

eLetters

No eLetters have been published for this article.