- by Andrew Taylor and Rachel Fayter
Canadian non-profit
funders are increasingly interested in demonstrating the impact
of their investments. As they pursue
the goal of
"managing for community impact,"
some funders are being prompted to
rethink a few of the basic assumptions
that guide them in their work. United Ways have been at the
forefront of this movement, and CCBR has worked with several
Canadian United Ways over the last 5 years. We have run
workshops, reviewed reporting formats, and aided in the
development of funding priorities or strategic outcomes. Over the
last year, we have begun to assess the impact of these efforts in
a variety of ways, and that work is the focus of this
article.
During the past year, United Way of Peel
Region and United Way of Kitchener Waterloo and Area came to CCBR
with similar requests. They had already trained funded agencies
in program evaluation and outcome measurement. In an effort to
make their investments more strategic and impactful, they had
identified broad "community outcomes." Both United Ways had also
altered their reporting formats so that funded programs would be
prompted to share what they were learning about the impact of
their work. These two United Ways came to CCBR and asked us to
review and analyze the information they were getting back from
funded agencies. They wanted to know whether their efforts had
led to change. Were funded programs measuring their impact more
effectively? Were the impacts of these individual programs, taken
together, adding up to true community change?
In order to address these questions, we
undertook detailed analysis of the information provided by funded
programs to these two United Ways through their annual reports.
We looked at the outcome objectives these programs were
identifying, and tried to determine how well these outcomes
linked up with United Way community outcomes. We looked at the
evaluation results generated by programs, and tried to determine
whether these results could be brought together in a way that
would show how United Way investments had helped local agencies
to make a difference. We drafted reports in which we highlighted
what we felt had been learned in each of five priority areas
identified by the United Ways. We held community meetings at
which we shared what we had learned with funded agencies, United
Way volunteers, and other key stakeholders.
In many ways, all we were doing here was
helping these United Ways to practice what they had been
preaching to the agencies in a new way. The annual report
completed by each funded program is, really, one of the tools
United Ways use to evaluate their own impact as funders. We were
attempting to analyze this data in a thorough way, come up with
useful new insights, and use these insights as a catalyst for
dialogue with the community.
So what did we learn?
We learned that funded programs in both
regions are at various stages in terms of their capacity to
measure their outcomes. Many programs are doing a great job, and
most are making significant progress over time in reporting their
outcomes. Workshops have been helpful. By identifying a set of
broad community impact areas, by unpacking these impact areas
into more concrete "long term outcomes," and by altering
reporting formats, these two United Ways have made their
expectations around evaluation clearer, and this has helped
agencies as well.
We also learned that evaluation findings
from individual programs are often more meaningful and exciting
when they are placed alongside findings from other, similar
programs and agencies. When agencies share their evaluation
methods and findings with one another, it gives each agency
valuable context for its own evaluation findings, and sometimes
leads to ideas about how different programs might collaborate. By
sharing evaluation tools or providing feedback on logic models,
for example, similar programs can save time and help one another
to improve. For this reason, one of our recommendations to both
United Ways was that they bring small clusters of programs with
similar measurement needs together on a regular basis so that
they can discuss their strategies and their findings. Of course,
when funding limited and similar programs compete for resources,
this type of collaboration is not without its challenges.
However, our community consultations suggested that there was
some enthusiasm for this idea among funded agencies in both
communities.
At the beginning of this process, United
Ways tried to avoid overburdening funded programs by telling them
they didn't have to measure everything. In
Waterloo, the United Way told funded agencies to pick just
one of the community outcomes and report results around that.
However, we learned that the relationship between the "program
outcomes" of a single program and the "community outcomes"
identified by United Ways is more complex than we had imagined.
In our review, we often found that the programs that were doing
the best job of measuring outcomes chose to report several
different types of change that linked up to more than one United
Way community outcome. They tended to report information about
process alongside their outcome findings. Although it was more
work, some funded programs wanted to explain the context
as well as reporting specific outcomes.
This was an important finding. We had gone
into this process thinking that our analysis might lead to
simple, impressive, punchy conclusions that could be used by
United Ways in fundraising. Maybe something like "United Way
investments this year have helped local agencies get 72 youth
back into school." We were able to identify some high-level
summary statements like this. However, we also realized something
else. As funders, donors, volunteers and funded agencies began to
communicate with one another in a more thoughtful, informed, and
reciprocal way about community change, they began to trust each
other more. They began to share not only their outcomes, but also
their mistakes, their unanswered questions, their half-formed
ideas for new programs, and their deep thoughts about why change
is needed and how it happens.
Our recommendations around this issue are
only now beginning to take shape. How can United Way convey to
donors the apparent paradox that achieving clear, measureable
change in the community requires that you begin to think about
change in a more organic, complex, interconnected way? We'll keep
you posted as the conversation unfolds.
