Rethinking systems change: The implications for evaluation

The Challenge

Major Canadian funders are increasingly aware that entrenched social problems are best addressed using comprehensive, multi-pronged approaches that coordinate efforts on many different levels. Coordinated interventions like this are sometimes called comprehensive community initiatives, systems change initiatives or complex interventions.

Powerful as they are, these interventions create challenges for program evaluators. Traditional evaluation methods designed to study narrowly focused programs aren't always easily adapted to larger open-ended strategies. In these complex interventions, evaluation deals with theories of change, shared responsibility for research design and data gathering, and multiple audiences with differing - and sometimes conflicting - needs and timelines.

Partners

CCBR has partnered with a number of large Canadian funders and associations to develop evaluation plans. Recent partners have included the Ontarion Ministry of Children and Youth Services, the United Way of Greater Toronto, the Canadian Association for Community Living, and Human Resources & Social Development Canada. 

Innovative Solutions

CCBR's approach to evaluating complex interventions follows several key principles:

• Develop a shared understanding of the intervention's theory (or theories) of change

• Adapt evaluation strategies as the intervention evolves over time: strategies should shift between inductive and deductive approaches

• Build cultures that value experimentation, critical reflection and ongoing innovation

• Embed outcome measurement within a comprehensive approach that also explores process, context and implementation

• Clarify how changes in relationships among system players relate to observed outcomes

• Use evaluation to provoke and inform inclusive, thoughtful and critical dialogue about why the intervention is needed, how it is supposed to help, how the context is changing and what is being learned (otherwise monitoring and evaluation are nothing more than bureaucratic exercises)

 Our strategies for this type of work include:

• Formation of a cross-stakeholder steering committee to guide all aspects of the evaluation

• Collaborative development and discussion of theories of change that involves all key stakeholders using creative alternatives to the program logic model development.

• Analysis of how knowledge is strategically communicated and used throughout a broad network of partners

• Development of a series of small, interconnected but flexible evaluation plans for different elements of the intervention (rather than a single very complex evaluation plan)

One Example

The United Way of Greater Toronto recently developed an ambitious plan called the Building Strong Neighbourhoods Strategy. This program involves working together with agencies to support teams of outreach workers in low-income neighbourhoods. The intervention is designed to build social capital in areas with high concentrations of families at risk and very little social infrastructure.  

CCBR was asked to create an evaluation framework for the Building Strong Neighbourhoods Strategy. After meeting with the teams responsible for each component, a process was facilitated linking the objectives of their work with overall goals. Strategies for communication were also developed together with a plan to reach wider audiences.  

In the second stage of the project, we developed a set of evaluation plans each focusing on a different aspect of the strategy. Each evaluation plan included key indicators, measurement strategies, and suggested data collection tools. A second round of consultations with key stakeholders helped to set priorities for measurement and identify ways in which existing data could be best used. 

United Way is now in the process of implementing this plan. A comprehensive tracking and evaluation plan for Action for Neighbourhood Change will be implemented shortly. An online database capable of tracking anecdotes and examples as well as quantitative data is now being designed. 

Discoveries

In evaluating complex interventions, the distinction between process evaluation and outcome evaluation can break down. Processes can be dependent on the outcomes achieved by the resource development team or the training and capacity building team. As a result, systematic tracking of process at all levels and ongoing sharing of process information becomes critically important.

Complexity is what makes comprehensive community initiatives so powerful and innovative. Yet however complex the process, senior decision-makers and fundraisers need clear and simple explanations in order to do their work.

This tension reflects a larger dilemma. While the importance of addressing social issues though complex interventions is being addressed, there is also a trend at the same time towards management for impact. The challenge becomes managing both sets of ideas at the same time, especially when they appear to conflict. 

-written by Andrew Taylor

• Privacy Statement
• Legal
• Login