C29 B Outcome Mapping: Measuring Progress in Partnerships and Networks

Outcome Mapping: Measuring Progress in Partnerships and Networks Heidi Schaeffer, Knowledge Management and Learning Lead Association of Ontario Health Centres heidi@aohc.org outcomemapping.ca

Objectives

• Gain an understanding of Outcome Mapping, it’s key concepts and methodology.

• Dabble in applying the concepts of Outcome Mapping to a Health Link network.

The fish soup story: Inputs, Outputs, Outcomes and Impacts

Inspired by Monika Jetzin, Global Water Partnership, Hungary

Inputs or resources  Parents get together fish, fresh vegetables, water, barley, spices, pot, source of heat Activities  Mother or father carefully prepare and cook all the ingredients

Parents control

Output  Children taste the most nourishing fish soup in the world

Outcome  Children consider the soup delicious and eat fish soup once a week for the rest of their lives Impact  Children are healthy adults

Parents influence

Parents contribute a bit

The difference between simple and complex fish soups

? ď ś

Example adapted from Getting to Maybe, How the World Is Changed?, Frances Westley, Brenda Zimmerman, Michael Q. Patton, Random House Canada, 2006.

A simple situation The relations of cause and effect are known.

The parents follow the great grandmother’s recipe for fish soup.

The quantity and nature of the ingredients are spelled out, as well as the order in how they should be combined.

The parents do not need expertise although of course experience in cooking helps.

If they follow the recipe they will produce basically the same soup week after week.

If only life were so simple!

In the real world, the results can be much less certain.  The great grandmother’s recipe is lost.  Her recommended fish is not available in the market every week of the year.  The family’s buying power varies from year to year.  Children are different and change as they grow:  One becomes a vegetarian.  Another goes on a diet.  A third is simply rebellious.  Outside factors and actors influence the children – school, TV, friends and so forth

In such a complex situation  The relationships of cause and effect are unknown until the outcomes emerges.  To produce a nutritious soup that their children will eat once a week for the rest of their lives, the recipe is less important than the parents’ relationships with each son and daughter, and theirs with their social environment.  More than cooking experience, parent’s must rely on their sensitivity to their children’s needs and their own creativity in the kitchen.  And they must accept uncertainty about the results.

When the organisation and its reality are substantially complex , such as with a network of independent organizations‌

Conventional thinking‌

Time

‌ clashes with relationships of cause and effect that are unknown

Time Ricardo.Wilson-Grau@inter.nl.net

Social change can be…

• Complex:

• Unstable:

involve a confluence of actors and factors

independent of project duration

• Non-linear:

• Two-way:

unexpected, emergent, discontinuous

intervention may change

• Beyond control:

but subject to influence

• Incremental, cumulative:

watersheds & tipping points

“Being attentive along the journey is as important as, and critical to, ariving at the destination �

Michael Quinn Patton

Brief definition of OM • A participatory method for planning, monitoring and evaluation • Focused on changes in behaviour of those with whom the project or program works • Oriented towards social & organizational learning

Source: A guide for project M&E: IFAD

OM’s answer

Start from observable behaviour change

Embrace different perspectives

Recognise that all interventions have limited influence

Support people to build their own well-being

Enable interventions to adapt as they engage

Apply a systems understanding

Source: Terry Smutylo / OM Lab 2012

Key concepts in Outcome Mapping

Three key concepts in OM: 1. Sphere of influence 2. Boundary Partners 3. Outcomes understood as changes in behaviour

There is a limit to our influence

Project

Sphere of control

People the project works with/through

Sphere of influence

People who benefit from the project

Sphere of concern

There is a limit to our influence

Inputs, activities, outputs

Sphere of control

Outcomes: Changes in behavior

Sphere of influence

Impact: Changes in state

Sphere of concern

People and families know where to go to have their needs met and are happy with their care

Providers learn About each other And begin to identify complex patients

Complex clients who are heavy users of the health system need coordinated care and special services to be kept well and Out of hospitals

HL partners build vision with clear roles and responsibilities are negotiated

Data sharing systems developed

Coordinated Care plans are designed

People are healthier

Complex clients are utilzing support services and programs

Complex clients and their families are engaged in care planning

Cost of delivering health care is lower without reducing the quality

Source: Terry Smutylo

Focus of Outcome Mapping

Inputs

Activities

Outputs

Outcomes

Outcome Mapping

Impacts

Where is the map? ✓OM is a guide to the journey we take with our partners. We co-create the map.

✓It focuses on the intention and what happens along the way ✓The map is not the territory, it shows the route taken ✓“The only real voyage of discovery exists, not in seeing new landscapes, but in having new eyes” (Marcel Proust)

improved human, social, & environmental wellbeing

I have a dream!

Martin Luther King, Jr. August 28, 1963

A vision statement..

•

Guides • Motivates and inspires • Is an ‘accountability-free zone’

Vision facilitation question Imagine that, 5-10 years from now, the program has been extremely successful. Things have improved beyond your most ambitious dreams.

• What changes have occurred?

• What (& how) are your intended beneficiaries doing?

• What are your partners doing?

• Describe the better world you are seeking.

Step 2: Mission

The mission is that “bite� of the vision statement on which the program is going to focus.

Summary

Vision

Mission

✓ About the future

✓ Feasible

✓ Observable ✓ Idealistic

✓ Identifies activities and relationships

✓ Not about the program

✓ About the program

Step 3: Boundary Partners

Boundary Partners are...

Those individuals, groups, & organizations with whom a program: • interacts directly to effect change • can anticipate some opportunities for influence • engages in mutual learning

Who are your boundary partners?

Programme

Beneficiaries

Stakeholders

Boundary Partners

Families

Specialists

Other service providers

Primary Care HL Leads

Community Service Partners

PCP

PCP

PCP

LHINS MOHLTC

example‌

Strategic partners • Selected for their potential to contribute to the mission • A person or group with whom the program works directly to achieve the mission, without necessarily wanting to change the partner’s behaviour as part of the mission • E.g. Funding agency, contracted service providers, Other Health Links, media agencies

Step 5: Progress Markers

Progress Markers Love to see (Deep transformation)

Like to see (Active engagement)

Expect to see (Early positive responses)

Love to see Like to see Like to see

Like to see

Like to see

Like to see

Expect to see

Expect to see

Like to see

Expect to see

Why Graduated Progress Markers?

• Articulate the complexity of the change process

• Allow negotiation of expectations between the program and its partners • Permit early assessment of progress

• Encourage the program to seek the most profound transformation possible • Help identify mid-course improvements

How can we measure: Greater awareness…

Empowered women…

Community ownership…

Reduced conflict…

Increased collaboration…

Governmental commitment…

Gender sensitivity…

Equal access…

Budgetary transparency…

Active participation…

Poverty alleviation…

Strengthened capacity…

?

Facilitation questions • How can the programme know the boundary partner is moving toward the outcome?

• What would they be doing?

• What milestones would be reached as the boundary partner moves towards their intended role in contributing to the vision?

Progress Marker Checklist Each Progress Marker:  Describes a changed behaviour by the boundary partner  Can be monitored & observed As a set, Progress Markers:  Are graduated from preliminary to more profound changes in behaviour  Describe the change process of a single boundary partner

Outcome Engineering • • • • •

Level 1: Knowing that there is a journey to take (P1) Level 2: Taking the first Steps (P1 level) Level 3: Investing your own resources (all) Level 4: Overcoming resistance to the change (all) Level 5: Identifying with the journey by joining with others with a similar approach (P2 level) • Level 6: Leaving a legacy (now an expert for others) (P3 level).

A PM Framework • P1 Preparation for the Journey: building Knowledge and Capacity • P2 The owned journey begins: building support, collaboration and networks • P3 The owned journey proceeds: sustained continuous actions. Institutionalization, Policies and/or Culture Change

32 sets of progress markers

P1: Knowledge acquisition processes and practices • ...attending forums where (the project) elaborates about the technology • ...raising questions and issues that (the Project) will address to encourage (the BP’s) uptake of the technology • ...seek out additional information on water and watershed issues from external sources • ...requesting position papers from the relevant departments to solicit input into decisions

P2: Getting involved, build support & enroll others • ...brokering or developing partnerships with other agencies to take local action • ...establishing mechanisms to share and review work programmes across departments, especially on research projects • ...establishing and expanding the membership base of the national organization in Indonesia • ...organize ‘popular education’ to increase critical thinking of their members

P3: Owned journey continuous and sustained

• ...contribute to the improvement of the methodology internationally to continually make it even more effective • ...generate their own funds and re-invest in (related) community projects • ...developing and putting in place a communication policy guiding how information is shared within the organization

Change is continuous

Outcome Challenge • Describes behaviour of a single boundary partner

• Sets out the ideal actions, relationships activities

• Describes the boundary partner’s contribution to the vision

Facilitation questions • Ideally, in order to contribute to the vision, what would the boundary partner be doing?

• With whom would they be interacting?

• How could this boundary partner contribute maximally to the vision?

Step 6: Strategy Maps

Strategy Maps For each Boundary partner: • What will the Project do to SUPPORT desired changes in the Partner towards the Outcome Challenge?

• Diverse range (an array) of ACTIVITIES:

- Aimed at Partner - Aimed at Partner’s environment

- Cause, Persuade, Support

Strategy Map causal

Partner

Environment

persuasive supportive

Facilitation Questions causal

Partner

what will be done to produce immediate outputs?

Environment

what will be done to alter the physical or policy environment?

persuasive supportive what will be done to build capacity?

how will sustained support, guidance or mentoring be provided?

how will the what networks or media or relationships will publications be be established or used? utilized?

Step 7: Organisational Practices

Organizational Practices

How does your team or organization stay relevant, viable and effective?

8 practices 1. Prospecting for new ideas, opportunities, and resources 2. Seeking feedback from key informants

3. Obtaining the support of your next highest power 4. Assessing and (re)designing products, services, systems, and procedures

8 practices 5. Checking up on those already served to add value 6. Sharing your best wisdom with the world

7. Experimenting to remain innovative 8. Engaging in organizational reflection

High expectations of planning monitoring and evaluation Address a growing call to show results

Satisfying multiple accountability needs

To show results that are difficult to measure

Stimulating internal learning processes

Learning about unexpected results 67

Extra challenging when dealing with processes of complex change?

1) Multiple actors, different perspectives and different forms of interaction 2) Multiple accountability needs 3) Unpredictable and non-linear link between cause and effect, therefore important to learn quickly what works and what doesn’t.

4) When dealing with complex change you need to be able to change as well.

Relationships

Learning

Dealing with Complex Change

Adaptive Capacity

Accountability 69

69

69

Recent Research on OM 1. The OM approach helps to clarify relationships, roles and expectations

2. The OM approach helps to learn about the progress towards the development objectives

“How can OM

contribute to dealing with complex processes of social change?�

4. The OM approach helps to strengthen upward, downward and horizontal accountability needs

20-6-2013 70

3. The OM approach helps to strengthen the internal adaptive capacity

Using OM has helped to strenghten dialogue among programme stakeholders (N=43) Strongly agree/agree

36 5 neither agree nor disagree

0 2 disagree/ strongly disagree

Nr. respondents

Not applicable / no answer

Adaptive capacity How has OM helped to strenghten elements of adaptive capacity (N=43) More time set aside for reflection about the monitoring data learning about the external context

OM contributing to changes in the internal practices of the programme increased understanding aobut how programme contributes to its effects

34

31

3 42

6 24

Strongly agree/disagree neither agree nor diagree disagree/strongly disagree

29

9 05 Not applicable / no answer

32

6 14

Nr. of respondents

20-6-2013 72

Challenges: • Time & resources • Stimulating informal learning spaces

• Capacity to facilitate reflection

OM and multiple actors and relationships (N=43)

OM has helped stakeholders to clarify their expectations

0 1

39

3

strongly agree / agree

OM has helped programme stakeholders to clarify their responsibilities

35

6

1 1

neither agree nor disagree

disagree / strongly disagree

OM has helped to strengthen trustful relationships

0

11

28

4 Nr. of respondents

73

Not applicable / No answer

Monitoring unexpected effects Has OM helped stakeholders to learn about unexpected programme effects? (N=43) At output level

At outcome level

At impact level

23

Strongly agree / agree

11 1 8 29 9

Neither agree nor disagree Disagree / strongly disagree

1 4 14 14

Not applicable / no answer

1 14 Nr. of respondents

74

E.g.: Sustainable Agriculture Programme Pacific Islands and Territories One of the love to see progress markers for the targeted villages read as follows: “sharing lessons and experiences with the other villages�.

It was an unexpected surprise for the programme team to learn during the monitoring process that it was not the targeted village communities that took the initiative to share experiences but instead the surrounding villages invited them to come and share new technologies for improving their agriculture production.

This happened without any support from the programme.

Case Example: McConnell Foundation Food Security Project