|
The initial consultation was an online survey (July 200), with invitations sent to everyone who had registered on the site. It must be stressed that the response rate was very small, with only 12 people completing the survey this is clearly not representative of the field. The information was however very useful (thank you..) Most respondents were providers, with only one care manager. The providers were largely supportive of the project (with reservations discussed below). The low response rate from care managers suggests they may be harder to engage or that we need to find a different communications channel. Core principles The survey asked about the core principles of the project. There was unanimous support for the overall project aim of collating outcome data and feeding it back to care managers. Most also thought that care managers collecting this data was a good idea, though a large minority disagreed. Tops concerns The main area of concern, through the whole questionnaire was around the use of TOPS. A large minority thought that the use of TOPS as the primary outcome measure was a bad idea. In the appendix below we respond to most of the actual points of concern. Some of the concerns appear to be a consequence of insufficient information about TOPS or in misunderstanding the outcome projects intentions. In particular, several respondents focused on the progress of individual clients; the RROP in contrast depends on being able to measure the overall (or ‘on average’) success of rehabs compared to each other. To achieve this we need to measure change in individuals in a way that can be readily recorded and aggregated. The tool to measure change must be easy to use, widely accepted, be based on solid evidence that it measures what we need to measure and cover alcohol as well as drugs. So far, I have not seen anything that meets these criteria as well as TOPS On balance, I continue to believe that TOPS should underpin our outcome measurement, though am open to alternative suggestions – whether other tools or additions to the dataset to strengthen the project. NDTMS Concerns The RROP also depends on being able to describe the ‘problem severity’ of the clients pre-admission, so we can take account of some houses seeing more complex clients than others. The NDTMS was proposed as a dataset to supplement TOPS for this purpose. Feedback on NDTMS was mixed, with some for and some against. Looking into this, we agree that the NDTMS dataset does not cover off all the confounding variables that impact outcome. For example; - Detox arrangements, as clients with poor detox arrangements may do poorly through no fault of the house.
- Aftercare affects outcome but may not be the responsibility of the house.
We will look again at NDTMS and TOPS and consider what other confounding factors are not covered in this dataset. Subject to further consultation these will be added to the existing dataset. Possible future elements of the project Future additions to the project were explored. All the following tentative proposals received were welcomed, with the caveat of needing more detail: - Recording care manager satisfaction,
- Recording client satisfaction,
- Making public the results,
- Feeding benchmarking data back to providers,
- Creating IT platform for providers to send updates back to care managers.
Overall conclusions The consultation suggested that providers welcomed the project in principle, with concerns about the validity of the dataset. Actions we take from this are: - Look again at how to strengthen the dataset, particular in the identification of confounding factors that impact treatment outcomes.
- Improve the explanation about TOPS and request alternative suggestions.
- Discuss how private funded clients might be covered by the project.
- Consider how providers might be involved in data collection.
- Work smarter to engage care managers and commissioners
Appendix A – Response to specific concerns about TOPS It is limited to persons needing funding, limiting applicability where there is a mix of private and state funding. Good point, this will need thinking through. We will contact private providers directly to discuss this, and explore solutions that could be generalised to other providers who take self funding clients. I don't know the dataset for NDTMS and TOPS, so I can't comment. From the sample screens shown on your web site, the datasets seem very very poor. The sample screens only show about 10% of the data, see the following sites for more comprehensive information. http://www.nta.nhs.uk/TOP In particular, take a look at the FAQ at the bottom of the screen. http://www.nta.nhs.uk/areas/NDTMS/core_data_set_page.aspx Does not seem to address 'confounds' addressed such as dual diagnosis. TOPS does to some extent, though we agree there is a need to look again at confounders. TOPS not sophisticated enough as rehabs are a very ‘false environment’. TOPS does not provide a true reflection of a clients progress through the treatment process, as they only capture data for the start and end of a clients treatment We are interested in the performance of each rehab on average over a meaningful number of clients. The outcomes measure therefore does not need to track client progress in the house itself – so it does not matter that TOPS is of little use in tracking change during the rehab stay. The validation study for TOPS included rehab. TOPS provides very little qualitative data. It is not clear how we would convert qualitative data on individual clients into aggregate performance data of whole rehabs. TOPS geared towards drug users It is also validated for alcohol. Care managers will fill in the information differently The validation study included a range of substance misuse workers and found that TOPS had excellent reliability across interviewers. There is good reason to believe that on average they will score accurately. Should use data on individual client sessions. This would be a measure of inputs not outcomes; it may be interesting and useful within a service but the commissioning system as a whole must try and focus on the actual outcome of treatment. Involve providers in data collection Very useful feedback, we will explore this more. It should be possible. |
Consultation 
