CCQHR Impact Evaluation Report

DEVELOPMENTAL EVALUATION

Centre for Critical Qualitative Health Research University of Toronto, Canada 2017

Report by: Komal Faiz September, 2017

CONTENTS Project Brief

4

Why developmental evaluation?

5

Project phases and process

6

Findings 9 Developmental Evaluation Model

16

Tools to use

21

Future Influences and expansion

22

Bibliography 23 Appendix 1

25

Appendix 2

26

Appendix 3

27

LIST OF DIAGRAM AND IMAGES Image 1: source pexels.com

5

Image 2: Systemic inquiry with CCQHR team

7

Image 3: Participatory workshop with CCQHR fellows

8

Image 4: Participatory workshop with CCQHR fellows

9

Diagram 1: Fields of impact

10

Diagram 2: systemic inquiry, highlighted insights

11

Diagram 3: Three dimensions of impact

11

Diagram 4: Systemic Inquiry

12

Diagram 5: Stakeholder analysis.

13

Diagram 7: Stakeholder roles

14

Diagram 6: Stakeholder analysis, highlighted insights

14

Diagram 8: Developmental Evaluation Model

16

Diagram 9: Demonstrating impact

19

Image 5: Displaying participatory activities (source: pexels.com)

21

Image 6: Set of tools (source pexels.com)

22

PROJECT BRIEF Centre for Critical Qualitative Health Research (CCQHR) requires an integrated impact assessment model designed to help monitor and assess their impact. The purposes of the designed strategy for impact assessment are: »» To communicate impact to the Dean and partners »» To communicate impact to the community »» To expand reach »» To attract potential donors »» To explore other possibilities and scope of expansion »» To observe and quantify their impact »» To inform future strategies

5 Image 1: source pexels.com

WHY DEVELOPMENTAL EVALUATION? According to a primer by J.W McConnel Family Foundation (Dozois et al., 2010), “Developmental evaluation supports the process of innovation within an organization and in its activities. Innovative initiatives are often in a state of continuous development and adaptation, and they frequently unfold in a changing and unpredictable environment.” The important elements of developmental evaluation are its flexibility and adaptability, and support for innovation and change.

CCQHR’s work is dynamic and deals with a non-linear approach to advancing capacity and increasing the knowledge base in the fields of education, research, and healthcare.

A static method of evaluation will limit the progress of the Centre and give a skewed image of its impact.

Salient benefits for CCQHR and its stakeholders are: »» Flexibility that can accommodate the needs of a growing Centre »» Incorporate emergent changes »» Facilitate stakeholders in visualizing the broad spectrum of impact CCQHR encapsulates »» Inform feedback reports for continuous internal improvements »» Use feedback for future strategies and ensure focused growth

6

PROJECT PHASES AND PROCESS The project followed five phases that ultimately led to the developmental evaluation model design and this report:

Phase 1: Design Brief Discussions with the team and defining the project brief

Phase 2: Research and analysis This phase was to understand: »» How can developmental evaluation assist CCQHR? »» What are the current practices of the Centre? »» How CCQHR envisions impact and change? »» The perspectives, needs, and expectations of the CCQHR team and community »» Aspects of impact assessment that CCQHR should consider »» CCQHR’s field, type, and scope of impact »» Any relevant existing models »» Best practices in impact assessment The research phase comprised of five in-person interviews with the leadership of the Centre, secondary and participatory research. The secondary research included literature review, desk research, reading past reports by CCQHR.

Image 2: Systemic inquiry with CCQHR team

7

Image 3: Participatory workshop with CCQHR fellows

Participatory research workshops included: »» Three co-design workshops with the core team on systemic inquiry, stakeholder analysis, and logic model »» One participatory research workshop with instructors on an environment scan for CCQHR »» One participatory research workshop with fellows and strategy lead comprehending CCQHR’s logic model At the end of this phase insights were generated about the system at CCQHR, its impact, and best practices.

Phase 3: Developmental Impact Model design The research findings and insights led to designing an integrated and systemic model for CCQHR to measure its impact.

Phase 4: Presentation and iterations 8

Image 4: Participatory workshop with CCQHR fellows

The aim was to ensure CCQHR team’s involvement and co-creation so that the project could address the needs and steer accordingly.

A detailed presentation was given to the team after the final model design to incorporate their feedback.

Phase 5: Final report After the iterations, final deliverables were given to the team.

9

»»Excellence in health education

FINDINGS Fields of Impact EducationH

ealth Research

Innovation

Qualitative Research

Diagram 1: Fields of impact

CCQHR’s impact lies at the intersection of four key areas: education, health research, qualitative research, and innovation. It does not limit to the core intersection point but expands to all overlaps within the four as the colored areas represent in diagram 1. This means that CCQHR addresses the needs of multiple fields and stakeholder through each initiative.

Scope of Impact Four aspect of the system at CCQHR were analyzed though a ‘systemic inquiry’: process, structure, functions, and context. “Iterations of structure, function, and process in a given context would examine assumptions and properties of each element in its own right, then in relationship with other members of the set. Subsequent iterations would establish validity of the assumptions and successively produce an understanding of the whole (Gharajedaghi, 2011).” The diagram 4 (page 9) shows how the four elements connect with one another and make impact happen.

Among others 10

Diagram 2: systemic inquiry, highlighted insights

the inquiry presented two main insights useful for CCQHR (see diagram 2).

1. The geographical region of impact by CCQHR spreads out from local to national to the larger international sphere.

2. CCQHR has stakeholders ranging from students to fellows to international partners that are instrumental to its growth and impact.

Synthesizing it, the scope of impact is three dimensional for CCQHR. It moves in three directions in terms of the geographical region, stakeholders, and the time-taken as shown in the diagram 3. For every project, the placement on these axes varies.

Diagram 3: Three dimensions of impact

11

Diagram 4: Systemic Inquiry

12

Stakeholders A mapping of stakeholder analysis for CCQHR used two axes: interest and power as shown in diagram 5. The high-interest, high-power stakeholders are: fellows, CCQHR employees, DLSPH (Dalla Lana School of Public Health) and the influential group. The low-interest, low-power group is the continuing education event attendees. The students lie in the low-power, high-interest group and the Deans of contributing partners are in the high-power, low-interest group.

There are two implications from the stakeholder mapping for CCQHR, which are also highlighted in diagram 6.

1. Most of the stakeholders lie in the high-interest region which is a strength for CCQHR. This means that invested stakeholders are a resource to expand impact.

2. There is a need to design strategies to move both the attendee’s group and the Deans of contributing partners to high-interest quadrants.

Diagram 5: Stakeholder analysis.

13

Diagram 6: Stakeholder analysis, highlighted insights

Stakeholders perform multiple roles at CCQHR. Most of the stakeholders are volunteering their time and efforts in different capacities whether instructors, employees or students. The contribution of fellows is completely voluntary. Diagram 7 illustrates their interconnected contributions.

Diagram 7: Stakeholder roles

14

Leverage points in the system Among others, CCQHR has some current practices and systemic patterns, which provide great opportunities to expand and enhance future impact, and can also be useful for data collection. Five of these are: 1. Fellows’ and instructors’ meetings For a discussion on future strategies and collection of impact data.

2. Motivated people that share a collective vision 3. Active participation and sense of belonging of fellows and team members 4. Space for expansion and growth in terms of activities and impact in the three dimensions shown in diagram 3 5. Director’s leadership skills, experience, and high-interest, high-power role driving the Centre and inspiring people to contribute Points 2 and 3 are most effective resources useful for CCQHR’s growth and impact

Other insights There are two more important insights from the research and study of CCQHR’s system and impact: 1. CCQHR has a unique positioning in terms of its impact. No other organization or institute, national or international is working in the four key fields of impact the same way 2. The process of impact is complex and intertwined for CCQHR. This means that impact assessment will be organic and qualitative.

15

Diagram 8: Developmental Evaluation Model

16

Attendees

Practitioners Academics Researchers

Int’l community

Fellows

Fellows

Instructors

Students

SOURCE

Stories

Indicators

DATA

Evidence- based impact measurement

Impact Stories

Benchmarking

Logic Model

SYNTHESIS

Internal feedback

International

Local + National

SCOPE

DEVELOPMENTAL EVALUATION MODEL The model Diagram 8 shows the developmental evaluation (DE) model for CCQHR.

Features of DE Model: 1. The DE model uses three methods to synthesize impact data: logic model, benchmarking, and impact stories.

2. Three levels for impact assessment are local/national, international, and internal feedback. Impact report can have separate segments for each.

Internal feedback can be kept for internal use only.

3. Two methods for data collection are indicators and stories from sources.

The sources of data are students, instructors, fellows, academics, practitioners, researchers, the international community, and attendees of events.

Data sources There are 6 data sources in the DE model. The breakdown and details of the sources are below. The last option in every category mentions the sources for benchmarking.

Students: »» Masters »» PhD »» Current »» Alumni »» Benchmarking: students neither enrolled in CCQHR courses nor are a part of the CCQHR community 17

Instructors »» Teaching CCQHR courses (masters and Ph.D.) »» Invited for conferences and seminar »» Benchmarking: instructors neither teaching CCQHR courses nor any students taking CCQHR courses »» Benchmarking: teaching other courses to students taking CCQHR courses

Fellows »» Current »» Alumni

Academics + Practitioners + Researchers »» Working with CCQHR students or hiring them »» CCQHR Fellows, academics, practitioners and researchers acquiring human resources from CCQHR community »» Local and National Practitioners part of CCQHR community »» Benchmarking: academics, practitioners, and researchers in the field »» Benchmarking: people and organizations looking for human resources in the four impact areas of CCQHR

International community members from outside Canada »» »» »» »»

Professors Practitioners Contributors Students or attendees

Attendees of events »» Continuing education conferences »» Seminars »» Events outside UofT organized by CCQHR »» Benchmarking: attendees of events by institutes working in the same field.

18

Input data 1. Indicators (a list of indicators is in appendix 1) 2. Stories The stories are evidence based examples to humanize the impact of CCQHR’s activities and inputs.

Demonstrating Impact

Diagram 9: Demonstrating impact

Demonstrating CCQHR’s impact comes from three ways of data synthesis: a logic model, benchmarking and impact stories. Diagram 9 illustrated linkages of the methods and how indicators help inform them.

Logic Model The logic model demonstrates the holistic view of impact at CCQHR, and how inputs and activities convert to outputs, outcomes and long-term impact. The logic model is based on the indicators and narratives. Over time 19

the logic model will evolve based on feedback from stakeholders. Appendix 2 shows the current logic model of CCQHR. To communicate impact year after year using logic model, the points to elaborate, observe and analyze are below. »» The assumptions for making the logic model »» What success means to CCQHR and its stakeholders? »» The logic model should demonstrate how CCQHR and stakeholders achieve success »» The evolution of Logic model:

How and why has the model evolved?

Any changes in the assumptions?

Any new discoveries or narratives to demonstrate change.

Any discrepancy observed between the planned output, out

come and impact and the evidence-based impact »» The logic model should be evidence-centric, based on data or narratives and should discuss the assumptions.

Benchmarking The purposes of benchmarking are to enable comparability and to understand what would not have happened without CCQHR’s work. It will help answer some fundamental questions of how is the work by CCQHR different compared to other institutions and worth donors and stakeholders’ time and money. Below are some points to analyze and observe. »» Comparison between institutions in Canada and abroad working towards the same goal or in a similar manner. »» What are the differences between CCQHR and other organizations working in a similar way or in the same fields of impact? »» Comparing the students and attendees of CCQHR courses and events with those that have not been a part of any »» What is CCQHR’s impact, which would not have happened otherwise? »» What standards has CCQHR set to ensure contextual impact?

Impact Stories Impact stories are the experiences of stakeholders. The main purpose of impact stories is to humanize the impact. They provide an evidence for the 20

logic model and benchmarking and illustrate new insights. The possible narratives to collect are below. »» Follow-up with students to see their trajectory and understand how CCQHR’s courses made a difference in their career and knowledge-base »» Fellows experiences »» Logic model evidence »» Analyzing the narratives of students and other stakeholders not part of the CCQHR community

Data collection A mixed method approach should be used for data collection.

Methods already in practice are: »» Student evaluation »» Student interviews »» Google metrics »» Occasional attendees feedback »» Fellows report of activity

Image 5: Displaying participatory activities (source: pexels.com)

Additional methods and processes for data collection for the future: »» A quick snapshot report after each project/ event »» CCQHR evaluations from students »» Group interviews with students »» Participatory activity with fellows and instructors at the annual meetings »» Co-design activity: logic model »» Interviews with the academics, researchers, and practitioners who have already or looking to acquire human resources from CCQHR community

21

TOOLS TO USE Online tools for maps and graphs: »» Network mapping websites like kumu.io which help make causal loops and connection maps »» Pictochart.com and easel. ly are free online resources to make infographics and other visuals

Software for research »» Alt Metrics, used by many researchers and research organizations to evaluate the impact of research

Data collections »» Airtable.com helps make an online database that is shareable and also allows adding pictures and graphs »» Microsoft Excel

Image 6: Set of tools (source pexels.com)

22

FUTURE INFLUENCES AND EXPANSION Based on participatory activity with instructors, external forces and future influences on CCQHR were analyzed. The list below are possible leverage points, and potential areas for growth and innovation for CCQHR. For the detailed visual see Appendix 3

Leverage point to advance and expand impact: »» Worldwide political conditions mean more people coming to Canada with different skillsets- CCQHR can welcome them into their community »» New collaboration opportunities with international organizations and individuals from US, Brexit and others »» Inclusion of LGBTQ to gain support from another community »» Community researchers from marginalized groups and visible minorities »» Health care innovation focus »» Customization and personalization trends »» Commercialization of research »» Technology »» Market gap: not many organizations and centres focusing on qualitative health research »» Online education

Must be considered for growth: »» Less funding for qualitative health research »» Inclusion of young talent and new minds. Millennials have unique ways and innovation »» Expansion on social media outreach »» Keeping it personalized alongside expansion »» Following up on graduated students and alumni fellows after they graduate

23

BIBLIOGRAPHY Aragón, A. M. (2013). A measure for the impact of research. Scientific reports, 3.

Bornmann, L. (2012). Measuring the societal impact of research. EMBO reports, 13(8), 673-676.

Bornmann, L. (2013). What is societal impact of research and how can it be assessed? A literature survey. Journal of the Association for Information Science and Technology, 64(2), 217-233.

Cabaj, M. (2014). Evaluating collective impact: Five simple rules. The Philanthropist, 26(1).

Chowdhury, G., Koya, K., & Philipson, P. (2016). Measuring the impact of research: lessons from the UK’s Research Excellence Framework 2014. PloS one, 11(6), e0156978.

Dozois, E., Blanchet-Cohen, N., & Langlois, M. (2010). DE 201: A practitioner’s guide to developmental evaluation. JW McConnell Family Foundation.

Gamble, J. A. (2008). A developmental evaluation primer. Montreal: JW McConnell Family Foundation.

Gharajedaghi, J. (2011). Systems thinking: Managing chaos and complexity: A platform for designing business architecture. Elsevier.

Herranz, Jr, J. (2010). The logic model as a tool for developing a network performance measurement system. Public Performance & Management Review, 34(1), 56-80.

Hill, S. (2016). Assessing (for) Impact: Future Assessment of the Societal Impact of Research.

McLaughlin, J. A., & Jordan, G. B. (1999). Logic models: a tool for telling your programs performance story. Evaluation and program planning, 22(1), 65-72.

McLaughlin, J. A., & Jordan, G. B. (1999). Logic models: a tool for telling your programs performance story. Evaluation and program planning, 22(1), 65-72.

24

Patton, M. Q. (2016). What is essential in developmental evaluation? On integrity, fidelity, adultery, abstinence, impotence, long-term commitment, integrity, and sensitivity in implementing evaluation models. American Journal of Evaluation, 37(2), 250-265.

Poston, B. (2009). Maslow’s hierarchy of needs. surgical technologist, 41(8), 347353.

Puttick, R., & Ludlow, J. (2013). Standards of Evidence: An approach that balances the need for evidence with innovation. London: Nesta.

Ramírez, R., Brodhead, D., Solomon, C., Kumar-Range, S., Zaveri, S., Earl, S., & Smith, M. (2013). Utilization focused evaluation: A primer for evaluators. Southbound, Penang, MY. Ramírez, R., Kora, G., & Shephard, D. (2015). Utilization focused developmental evaluation: Learning through practice. Journal of MultiDisciplinary Evaluation, 11(24), 37-53.

Rao, A., Spasojevic, N., Li, Z., & DSouza, T. (2015, October). Klout score: Measuring influence across multiple social networks. In Big Data (Big Data), 2015 IEEE International Conference on (pp. 2282-2289). IEEE. Savaya, R., & Waysman, M. (2005). The logic model: A tool for incorporating theory in development and evaluation of programs. Administration in Social Work, 29(2), 85-103.

APPENDIX 1

Click here for detailed view

APPENDIX 2 [Final indicators list to be attached]

APPENDIX 3