Evaluation of CQC's local authority pilot assessments

December 2023

Summary

In Summer 2023, CQC started pilot assessments of 5 local authorities to test the new approach.

CQC’s Research and Evaluation team joined the wider project team to evaluate the pilots. The aim of the evaluation was to:

• understand how well CQC is carrying out the assessments and provide assurance that the methods are effective to provide the evidence needed to make a judgement on how well local authorities are discharging their adult social care duties against the Care Act.
• identify early indications of outcomes from the pilots.

Given the pace of this project, the evaluation used a developmental approach. This means evaluation was embedded into the pilots as they happened, focusing on gathering feedback from both the assessment team applying the approach and the local authorities being assessed. To understand people’s experiences of the approach, gathering evidence for the evaluation used largely qualitative methods of:

• in-depth interviewing
• focus groups
• observations

There was a good level of participation and interest in the evaluation from both the assessment team and the local authorities themselves.

The pilots involved using CQC’s new single assessment framework, applying the relevant quality statements to local authorities and scoring the applicable evidence categories. This approach has not yet been fully rolled out to providers, and the piloting and previous ‘test and learn’ exercises have provided rich opportunities for testing the new approach. Overall, the evaluation found that:

• The quality statements at the centre of assessments were broadly right and what local authorities expected. The team found some pockets of duplication when allocating evidence to the quality statements. We expect this will reduce with the benefit of more time, experience, and guidance.
• The team was not always confident in scoring evidence. This was in part about gaining experience in the appropriate level and range of evidence needed to make a judgement among the breadth of evidence gathering undertaken. They also expressed not yet being confident in using the scoring model, which was new to most of the team and required some adjusting to. This was exacerbated by the limited guidance available on the scoring model to support decision making and no guidance available on ratings. The team felt more guidance would create better consistency and reduce challenge from local authorities.
• Pilot timescales were compressed, which had a significant impact on the process. It was challenging to organise the on-site fieldwork timetables in a timely way. This was exacerbated by the local authorities being unclear about some of our requests. It also meant the team did not have time to fully assimilate information such as the information return, self-assessment, provider survey and case tracking to inform the fieldwork in the intended way. This is being factored into timeline planning for future assessments.
• Local authorities found the information return time-consuming to complete and some of the requests ambiguous. For CQC, this resulted in very large volumes of information to manage in the returns. Despite the issues, local authorities were clear that it was a helpful exercise, which sparked improvements. The information return is being reviewed giving opportunity to provide more refined accompanying guidance using the constructive feedback received. This will also ensure that time spent managing the returns is proportionate to other parts of the process for the team.
• Self-assessments were helpful for both local authorities and CQC. These were optional in the pilots, but all local authorities chose to complete one ahead of the fieldwork. Local authorities spoke of this being a helpful exercise and the assessment team found value in the content.
• Improvements were made in gathering evidence from people’s experience, but there is still opportunity to develop the approaches. Case tracking provided rich and wide-ranging evidence about services for a small number of cases, but proved to be a very time-consuming process for both CQC and local authorities.
• The composition of the assessment team with internal roles complemented by the expertise of external specialist advisors and executive reviewers was viewed positively by the local authorities. There are some thoughts on how responsibilities of roles could be better distributed among the team and consideration for how some additional roles may contribute. For example, where Experts by Experience could add most value to strengthen the team approach.
• The structure of assessment reports has been streamlined, but reports are time-consuming to produce. This was the first time the team produced full assessment reports and managed the level of evidence that needed to be collated in advance. The team felt it important to ensure the template, and any accompanying guidance, supports a consistent approach to writing reports and appropriate time spent on them.
• Local authorities found assessment reports valuable in re-affirming the areas of improvement required. They would have benefitted from more understanding of the process of how we make judgements and the scoring model itself, having raised some concerns about the strength of corroborating evidence and scores that did not always align to the narrative and tone of the report. Local authorities welcomed improvement suggestions in the report and some would have valued more definitive recommendations to further improve their scores and ratings. Additionally, local authorities would like reports to better acknowledge their own self-awareness of areas for improvements and actions they were already taking, as opposed to being presented as issues identified by CQC.
• The pilot process prompted improvements within local authorities. Local authorities described strengthening certain policies, improving co-production approaches, reviewing their staff engagement plans and improving how they store and manage files. We will endeavour to understand more about the outcomes and impact of our approach in systems and longer-term improvements as part of research we are currently commissioning.