© 2008 The Royal College of Psychiatrists
Progress on NICE guideline implementation in mental health trusts: meta-analyses
National Collaborating Centre for Mental Health (NCCMH), c/o CRTU, 4th Floor Standon House, 21Mansell Street, London E18AA, email: amears{at}cru.rcpsych.ac.uk
NCCMH, London
Oxleas NHS Foundation Trust, Dartford, Kent
Healthcare Commission, London
NCCMH, London
None. The study was funded by the National Collaborating Centre for Mental Health.
 TOP Abstract IntroductionMethodResultsDiscussionReferences |
|---|
AIMS AND METHOD
To investigate implementation of National Institute for Health and Clinical Excellence (NICE) guidelines in mental health, focusing on the schizophrenia guideline. Data analyses centred on implementation of the guideline, as well as looking at a set of markers mapped to the NICE principles of implementation and other identified clinical prerequisites. A self-report questionnaire tool was sent to senior executives at mental health trusts containing questions linked to the markers of implementation and clinical prerequisites; responses were analysed with data from the Healthcare Commission audit of implementation of the guideline to show key relationships.
RESULTS
Information from both data-sets (senior executive data collection and the audit) showed that implementation is patchy, with pockets of good implementation. Findings indicate that higher levels of implementation are linked to corporate commitment and leadership, as well as support from commissioners.
CLINICAL IMPLICATIONS
Implementation might be improved by corporate commitment and leadership and better support from commissioners.
|
TOPAbstractIntroductionMethodResultsDiscussionReferences |
|---|
To date there is little research about how successfully trusts are implementing guidance from the National Institute for Health and Clinical Excellence (NICE), but what there is suggests that implementation is inconsistent (Sheldon, 2004). A large amount of information on evidence-based implementation methods is published by NICE to support trusts with this task (National Institute for Health and Clinical Excellence, 2006). In order to determine the degree of progress and the key implementation processes used, the National Collaborating Centre for Mental Health (NCCMH) carried out a survey of mental health trusts, analysing data alongside findings from the Healthcare Commission audit of the implementation of the schizophrenia guideline.
|
TOPAbstractIntroductionMethodResultsDiscussionReferences |
|---|
We used data from two sources to unpack progress with implementation: data from a bespoke questionnaire tool, and a meta-analysis using this alongside data from the Healthcare Commission audit of the implementation of the NICE schizophrenia guideline. The questionnaire items were derived from two sources, to answer two key questions: first, how far have trusts followed NICE’s principles of implementation; and second, what evidence is there that trusts are implementing guidelines? Question one is answered by assessing the extent to which trusts gave positive responses to specific questionnaire items derived from the NICE principles. To answer the second question, expert advisors to the project have identified key implementation markers, which show that a trust is fully engaged with implementation. Questionnaire items were mapped to the principles summarised in Box 1. The forms were sent to all chief executives and medical, clinical and clinical governance directors at all National Health Service (NHS) mental health trusts (including primary care trusts providing mental health services) in England as well as combined trusts in Wales (a total of 488 questionnaires sent to 79 trusts).
Data from the recent audit of implementation of the schizophrenia guideline (Healthcare Commission, 2006), carried out as part of the Healthcare Commission joint review of community mental health services, were re-analysed alongside data from the present study. The audit identified a number of key implementation markers, and gathered data to assess adherence with these: availability and use of family interventions and cognitive–behavioural therapy (CBT), provision of information and occupational needs, physical health checks, prescribing (atypical antipsychotics, depot formulations, treatment-resistant schizophrenia), monitoring, medication and advance directives. A proxy measure of guideline implementation was created within the data-set by summing the trust-level scores for each of the indicators in the audit.
Data were analysed using SPSS version 14 for Windows. Correlations and one-way analyses of variance (ANOVAs) were used to identify significant univariate relationships; linear regression was used for multivariate effects. In order to examine fully the effect of corporate commitment and leadership, a new variable was constructed comprising identified executive lead, identified clinical director lead, clinical champion, team to support the clinical champion, and reports to trust board. The new variable showed an acceptable Cronbach’s µ value of 0.68.
|
TOPAbstractIntroductionMethodResultsDiscussionReferences |
|---|
A total of 209 usable forms were returned, an adjusted response rate of 47% (209/(488–44)). Data are available for the various professional groups sampled: 59% of forms were returned from chief executives, 42% from clinical directors, 47% from medical directors and 59% from directors of clinical governance. Data were returned from 69 of the 79 trusts sampled (87%), ranging from 0% to 100% of forms returned (mean 48%).
| Box 1. The National Institute for Health and Clinical Excellence (NICE)
six principles of implementation
|
Descriptive results
Corporate and local leadership (mapped to NICE principle 1)
- 56% indicated that the trust had identified a clinical director-level lead
for guideline implementation
- 46% indicated that the trust had identified a clinical guideline
implementation champion; of the clinical champions, 57% had a team to support
them
- 85% indicated that the trust had a committee to oversee implementation of
guidelines
- 57% indicated that trust boards routinely received reports on guideline
implementation.
Staffing and resources (mapped to NICE principles 2 and 3)
- 32% indicated that the trust had appointed staff specifically to implement
NICE guidelines
- 42% indicated the trust had calculated likely resource implications for
guidelines
- 38% indicated the trust had committed funds to implementation
- funds committed ranged from £10 000 to £1.2 million (mean
£34 000).
Commissioner resourcing (mapped to NICE principle 4)
- 84% indicated that recent NHS funding problems had prevented (or would be
likely to prevent) implementation of guidelines
- 71% rated the trust’s support from their commissioning primary care
trust in implementing health technology appraisals as poor, very poor or
non-existent
- 82% rated the trust’s support from their commissioning primary care
trust in implementing guidelines as poor, very poor or non-existent.
Accessibility of guidelines (mapped to NICE principle 5)
- 70% indicated that NICE guidelines were easily available on the trust
intranet
- 67% indicated that the trust also made guidelines accessible in the course
of clinical practice in another way
- 86% had looked for implementation information on the NICE website.
Operational implementation (mapped to NICE principle 5)
The use of outcome measures is a proxy for implementation, because it is a
precursor for appropriate recording of patient information and an indicator
that diagnosis is being recorded. Respondents were asked if clinical teams
routinely used outcome measures. Responses are shown in
Fig. 1. Seventy per cent of
respondents indicated that their trust used a computerised clinical
information system for patient details.
 View larger version (9K): [in a new window] [as a PowerPoint slide] |
Fig. 1. Percentage of respondents indicating that clinical teams routinely use
outcome measures.
|
- 73% indicated that the trust had audited health technology appraisal
implementation
- 84% indicated that the trust had audited guideline implementation.
Inferential results
Commissioner support of health technology appraisal implementation
One-way analysis of variance showed that a lower commissioning score is
associated with:
- lower likelihood that a trust would have calculated the resource
implications of implementation (F=2.464, d.f.=4,
P=0.047)
- lower likelihood that resources had been committed (F=4.217,
d.f.=4, P=0.003).
Correlation showed that a lower score for commissioning support is linked to a lower level of commitment and leadership (see below; R=0.180, P=0.048).
Commissioner support of guideline implementation
Our hypothesis was that, as with support for health technology appraisals,
commissioning support of guideline implementation might exert an effect on
other variables. One-way ANOVA showed that poorer commissioning support of
guidelines was associated with:
- higher level of health technology appraisal audit (F=2.471,
d.f.=4, P=0.046)
- poorer access to guidelines other than on the intranet (F=3.184,
d.f.=4, P=0.005).
NHS funding problems
Our hypothesis was that funding problems might curtail implementation. This
proved not to be the case for audit of health technology appraisals, where
reported funding problems were associated with:
- higher likelihood of audit taking place (F=14.667, d.f.=1,
P < 0.001);
- poorer level of access to guidelines other than on the trust intranet
(F=7.375, d.f.=1, P=0.007).
Commitment and leadership
One-way ANOVA showed that better levels of commitment and leadership were
associated with:
- staff appointed to support implementation (F=2.987, d.f.=12,
P=0.001)
- resource implications calculated (F=2.153, d.f.=12,
P=0.016)
- resources committed (F=3.025, d.f.=12, P=0.001)
- audit of health technology appraisals undertaken (F=2.702,
d.f.=11, P=0.003)
- existence of a clinical information system (F=1.940, d.f.=12,
P=0.032).
Correlation showed that better commitment and leadership is associated with more prevalent use of outcome measures (R=0.194, P=0.008).
Commissioner support, and corporate commitment and leadership
Correlation shows a positive relationship between commissioner support for
health technology appraisals and commitment and leadership (R=0.180,
n=181, P=0.016).
Analyses of the Healthcare Commission audit
A series of analyses were carried out to look at the relationship between
previously identified ‘causal’ factors and implementation as
measured by the Healthcare Commission audit of implementation of the
schizophrenia guideline.
Descriptive results
Items within the audit were collapsed into a number of key markers showing
adherence levels (Table 1).
|
View this table: [in a new window] | Table 1. Adherence with key markers from the Healthcare Commission audit |
Inferential results
In order to look at statistical relationships, the individual scores for
the audit were combined to give an overall proxy measure for implementation.
Correlations showed that better levels of implementation were associated with
the following factors (data at trust level):
- better corporate commitment and leadership (R=0.342,
n=57, P=0.009)
- existence of a committee to oversee implementation (R=0.429,
n=57, P=0.001)
- commissioner support for health technology appraisal (R=0.459,
n=57, P=0.000)
- less likelihood of guideline audit having been carried out
(R=–0.270, n=57, P=0.042).
The data were submitted to a step-wise linear regression. All variables from the NCCMH survey were input as independent variables, the schizophrenia proxy as the dependent. The model yielded a reasonable R2adj of 0.284 using three variables, model ANOVA significant (F=8.410, d.f.=3, P < 0.001), and collinearity statistics were within acceptable range (condition index=7.72). Variable (β) weights showing direction and strength of effect in the model were:
- commissioner support for TA, β=0.39
- corporate support, β=0.26
- identified clinical director lead, β=0.23.
This interaction between commissioner support and corporate support has a powerful predictive effect on implementation.
|
TOPAbstractIntroductionMethodResultsDiscussionReferences |
|---|
Limitations
The response rate was below 50%, and thus the findings must be seen as illustrative rather than definitive. To present the results from the survey we used all individual responses, as opposed to overall trust responses. There is, in some cases, variation between responses from different individuals from the same trust (although analysis of variance showed no significant results). Since the study focused only on mental health trusts, findings are not generalisable to other healthcare settings; the survey also provides a snapshot only of the current situation. The use of a self-report method (for both the questionnaire and the schizophrenia audit) will inevitably bring in elements of bias.
NICE guidance implementation and key implementation indicators
The NICE principles of implementation (Box 1) are not being closely
followed by many trusts. It is encouraging, however, that a majority of trusts
seem to have some identified prerequisites for implementation in place,
showing engagement with guideline implementation, although it is arguable that
these might be seen as prerequisites for an effective clinical service, and
that their existence owes more to this than any link to guideline
implementation.
Markers from the schizophrenia audit
Some of the markers show high levels of adherence (use of depot medication,
documentation of response and side-effects in care plan, avoidance of
polypharmacy), whereas others are poorly adhered to (notably provision of CBT
and family interventions). The high levels of adherence on the top five
markers may be due to coincidence with established best practice rather than
to change driven by guidance. The low levels of CBT and family interventions
are possibly due to lack of availability and funding rather than commitment
from clinical teams to implementation.
Key findings
Hypothesis testing shows a clear link between corporate commitment and
leadership and implementation of guidelines. Corporate commitment is linked to
many key implementation markers - where it is lacking, implementation will not
be very far advanced. This hypothesis is clearly supported by the secondary
analyses of the Healthcare Commission data on NICE schizophrenia guideline
implementation. There is a clear link between corporate commitment and
leadership and implementation of this specific guideline, from both the
correlation and the regression analyses, the latter showing the interactive
effect of commissioner support and commitment and leadership (lack of
commissioner support is linked to poorer corporate commitment and
leadership).
It is interesting to note that a higher level of audit of health technology appraisals is linked to poor commissioning support and funding problems. This may be a function of two possibly unrelated effects: it may be that commissioners are insisting that an audit of an appraisal is undertaken as a prerequisite of funding being allocated; it is also possible that this audit is being conducted as a lobbying activity to persuade commissioners to allocate funding (via a demonstration of need).
Implications
This study shows empirically that in order for implementation to succeed,
the trust’s board and senior managers must be engaged. Further, it shows
that commissioners must demonstrate support. These findings give ammunition to
trusts and to NICE to enable pressure to be put on commissioners in order to
improve support of NICE guidance and of its implementation. Further, these
data show that some trusts have advanced well in implementing guidelines,
providing identifiable best-practice models for others to follow.
|
TOPAbstractIntroductionMethodResultsDiscussionReferences |
|---|
- DEPARTMENT OF HEALTH (1997) The New NHS (Cm 3807). TSO (The Stationery Office).
- HEALTHCARE COMMISSION (2006) Joint Community Mental Health Services Review: Summary of the Scored Assessment Framework. Commission for Social Care Inspection & Healthcare Commission.
- NATIONAL INSTITUTE FOR HEALTH AND CLINICAL EXCELLENCE (2006) Putting NICE Guidance into Practice. NICE.
- SHELDON, T., CULLUM, N., DAWSON, D., et al
(2004) What’s the evidence that NICE guidance has been
implemented? Results from a national evaluation using time series analysis,
audit of patients’ notes, and interviews. BMJ,
329, 999.
[Abstract/Free Full Text]
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||