Organisation: The Government Outcomes Lab
Most programme evaluations focus on evaluating the impact of a specific intervention. Trying to evaluate a partnership model or contracting mechanism brings with it a whole new set of challenges, not least in trying to disentangle the impact of the contracting mechanism from the impact of the intervention itself. Evaluations of SOCs tend therefore to be qualitative, subjective and rely strongly on evaluator interpretation. This makes their conclusions open to critique, and it can be challenging to defend evaluation findings when stakeholders within the programmes don't agree with the findings. In this session, we reflected on the approaches and challenges adopted in evaluating SOCs over the last decade, and ways to overcome these challenges.
Session 2.1 delved into the challenges surrounding measurement and evaluation within outcome-based partnerships. We acknowledged that evaluations of Social Outcome Contracts (SOCs) tend to be qualitative, subjective, and heavily reliant on evaluator interpretation. This makes it challenging to separate the impact of the intervention from the impact of the outcome contracting approach. James Ronicle from Ecorys likened interventions and outcomes to a "doughnut problem" where it's hard to distinguish what's the jam (intervention) and what's the dough (outcomes contract). Sometimes there's an over-attribution of outcomes to what's actually in the contract. Rajeshwari Balasubramaniam and Meena Vaidyanathan from Niiti Consulting stressed the importance of fostering an evaluation culture by bringing together academics and practitioners. Drawing from their experience in the Together for Childhood evaluation of place-based change, Thea Shahrokh from NSPCC highlighted the significance of a continuous learning approach with an iterative stance. Professor Maren Duvendack from the UEA pointed out the challenge of disentangling the contracting mechanism from the intervention. Kenechukwu Nwagbo from the University of Cambridge emphasised the impact of politics on measurement and evaluation. Tomasso Tropeano from Politecnico di Milano underscored the value of Social Impact Measurement (SIM) linked to interventions, leading to trust, legitimacy, reduced information asymmetry, goal alignment, and new financial opportunities.