Making evidence on social outcomes contracting accessible and useful: A report from SOC21

The Government Outcomes Lab (GO Lab), in partnership with Ecorys, are currently conducting a global systematic review on SOC - a central objective of which is to gather and curate the existing evidence for maximum utility to policymakers. The GO Lab’s recent annual conference (SOC21) provided an opportunity to present and receive stakeholder feedback on the review’s progress to date. In a dedicated session, academics and policymakers discussed issues associated with making best use of a mixed body of evidence, as well as how the systematic review can contribute towards that aim. A prototype ‘heatmap’ was displayed, exemplifying how the large number of available studies might be organised for their relevance to specific policy sectors. A tool like the heatmap is critical to finding out what we really do know about outcomes contracting. It will represent the frontier of our cumulative knowledge on SOC, organised thematically and accessible in one place.

Insights from the conference panellists and audience members were incredibly helpful, and will shape the review team’s continued thinking on the kinds of tools we can produce to assist evidence-led policy making. In this blog, I highlight two key themes for consideration that emerged from the discussion.

Navigating terminology in the evidence on social outcomes contracting: Clarity vs. flexibility?

A practical difficulty when seeking evidence on social outcomes contracting is the number and variety of terms used to designate different forms of funding mechanism. An initial consultation with a Policy Advisory Group on the systematic review identified 35 different types of SOC - and those were only the English language terms (the review searched for evidence in multiple languages). Some of these are different names for similar financial instruments, while others differ more substantially in their design. Policymakers considering the most appropriate form of SOC for their specific context might reasonably want some clarity and orientation within this terminology and the variety of forms it represents.

To this end, the systematic review has been intentionally broad in its search terms, aiming to collate information on all the different types of outcomes contract, as well as identify where and when they have been used. This will allow for a greater understanding of similarities, differences, and inter-relations between the various funding mechanisms. While the review therefore seeks to introduce some order into the field, more than one of the conference speakers suggested that flexibility in terminology is in fact necessary for making SOC understandable to multiple stakeholders and in different contexts. Panellist Chih Hoong Sin, for example, was less concerned with the variety of terms than with making SOC mechanisms appeal to, and function within, the specific ecosystem in which a project is being developed. A desire for clear and consistent terminology to guide policy is perhaps, therefore, at odds with the need for adaptability.

Two points follow from this. First, an endeavour to produce a well-defined typology of forms should not lose sight of other (sometimes competing) efforts to make outcomes contracts adaptable to different contexts. Second, it will be important to keep the systematic review updated with any new forms of SOC and their associated terms, so that they can be incorporated into syntheses of the evidence. This will ensure that the review’s database remains comprehensive and up to date even as terminology changes. The review, too, must be flexible in this regard.

Explaining success and failure: The role of good relationships

Ideally, the synthesis of evidence in a systematic review will allow for an understanding not only of whether and where SOC has been implemented successfully, but why it has been successful. Towards the end of the conference session, the discussion turned to ‘quality of relationships’ as a factor explaining the effectiveness of outcomes contracts. That is, the extent to which the success or failure of a SOC-funded service can be explained not by factors inherent in the instrument itself, but by the characteristics of the people involved, the culture of the organisations they work for, and the broader norms of the society in which the project is implemented. As an audience member suggested, ‘to some extent, a bad tool in the hands of good people will still perform well’.

While there appeared to be agreement among the panellists on the importance of this aspect, it is perhaps one of the most difficult themes for the systematic review to produce useful evidence on. First, as a synthesis of research, rather than a primary investigation, the review can only exhibit what is present in the existing studies. If there has not been sufficient study of relationships and culture in and around SOC, then the review will struggle to provide much insight on this theme. Second, even if the evidence does shine a light on relational aspects of SOC, they are by definition more contingent and less easily defined than the technical aspects. How, for example, would one begin to incorporate something like emotional investment into a policy or contract?

Despite this difficulty, the potential importance of quality relationships as a determining factor of SOC success means that the systematic review should attempt, as far as possible, to gather what evidence exists and summarise it for policymakers. The crucial question then becomes how such qualities could be facilitated in the design of new SOC projects.

An invitation for feedback

It was great to hear the wider community’s reflections on the systematic review at SOC21. We were heartened by the enthusiasm for the review itself, as well as the associated heatmap tool, and look forward to further engagement as we progress over the coming weeks, months, and years. The systematic review team therefore welcome further feedback on the review and heatmap, particularly regarding how they can be of most use to policy. If you would like to comment or make suggestions on these, or any other, aspects of the review, please send them to Harry Bregazzi.