Looking at Change in Community-Based Research: An Organic Approach

Looking at Change in Community-Based Research An organic approach Briefing Paper Introduction Evaluation is a necessary component of most research and community-based initiatives or projects. Evaluation is commonly done to meet the needs and requirements of funders as it is a tool for determining if a project has met its expected goals. Evaluation can also be conducted to identify any necessary changes that should be made to an ongoing project. Evaluation happens throughout an ongoing project as well as at the end.

The process of studying change and evaluating can be a significant undertaking for project participants. It can be burdensome, as participants often are required to answer questions repeatedly. This can take time away from important work on the ground and in communities. The particularly challenging times of 2020 and 2021 led the universitybased implementation team of the More Than Words project to seek out ways for the implementation team members to take on more of the evaluative work, leading to the development of the organic evaluation method.

Organic evaluation is a process that maximizes existing data and information that has been shared with researchers and project managers to study change and fulfill evaluative needs. Organic evaluation can be used with different research methods, including creative and arts-based methods.

Researchers and project managers create reports using the stories, artwork and conversations shared with them by the project participants.

Organic evaluation aims to reduce some of the burdens that are often inherent in traditional evaluative methods and participating in research. This community centered approach challenges the top-down academic norms of research and evaluation by producing evaluations that are participantcentred and align with community values.

This briefing paper introduces the concept of organic evaluation, firstly situating the evaluation method in the More Than Words research project, then highlighting ethical considerations for evaluating communitybased research before providing the steps necessary to implement organic evaluation in your own work.

Page 01

Background Information This method of evaluation comes out of doing research as part of the More Than Words project in 2020. During this time, priorities in the communities shifted and the research team had to think critically and creatively about how to best go about continuing research and evaluation. Ultimately, the research team found that the method of organic evaluation facilitated meaningful community engagement by removing barriers to participation, making it an on-going evaluation practice for the project.

More Than Words More Than Words, a 4-year project which is part of a national $50 million Gender-Based Violence program Promising Practices to Support Survivors and their Families. MTW investigates and learns from the use of Indigenous-focused youth-led survivor engagement through the arts, looking at impacts on the producers themselves (young people) and on their families and communities in relation to their experiences of sexual and gender-based violence (SGBV). More Than Words works with Indigenous youth from 3 different and distinct communities and cultures; Eskasoni, located on Cape Breton Island within the Unama’gi district of Mi’kmaq territory (Nova Scotia, Atlantic Canada), Treaty 6, Saskatoon and the Homeland of the Métis is the site of Young Indigenous Women’s Utopia and Rankin Inlet is an Inuit community located on the Kudlulik Peninsula, on the west coast of the Hudson Bay in Nunavut. Additionally, there is a McGill University-based implementation team, which supports the on-the-ground work in all three sites in varying capacities and manages the majority of the project report writing.

Context In 2020 and 2021 the MTW project sites faced tremendous external stressors and challenges, from the COVID-19 pandemic to the uncovering of many Indigenous children in unmarked graves at residential schools across Canada. As such, the proposed project timeline was greatly impacted and evaluation was not a top priority for the sites, as the site leaders and youth participants were occupied with urgent community matters and familial needs.

The McGill University-based implementation team sought to find new ways for sites to conduct evaluation, without overburdening site leaders, youth participants, their families and communities, while maintaining authenticity in the evaluation. The team first turned to readily available information and maximized what had been shared by the sites in the past by bringing new lenses and evaluative questions to this artwork and stories.

Page 02

Next, when possible meetings were arranged with the site leaders to share and have conversations about the evaluative work that was happening. In these conversations, site leaders shared stories of other projects and incommunity work and consented to having these stories shared in a final report. From this process, an evaluation report entitled “Studying Change: Keeping Up” was produced.

This experience led the implementation team at McGill to recognize that there might never be a time when evaluation and evaluationrelated tasks are a priority or appropriate to ask from project participants. As the sites share pieces of their stories on an ongoing basis, the implementation team recognized the potential roles and responsibilities they could take on in a more long-term capacity regarding evaluation. The updates and stories from sites have now become integral to the process of studying change in the More Than Words project.

These stories and ongoing communications are part of an organic evaluation process that aims to lessen the administrative work expected from sites.

Adopting these practices has included sensitizing the team to organic evaluation, highlighting the importance of capturing communications with the sites and within project communications. Ensuring the whole team was noting communications meant the site leaders and participants did not have to spend time participating in formal or traditional evaluation to provide the same answers already shared.

The Principles of Organic Evaluation Founded on on-going and informed consent Removes barriers to participation Makes the most of the information readily available Encourages ongoing engagement and dialogues Challenges academic top-down evaluation norms

This method of evaluation has been successfully implemented since 2020. Having strong and established relationships with the site leaders, and supporting academics and youth participants has opened important channels of communication and storytelling that are central to organic evaluation. Youth participants, site leaders and academic supports of the project came together at McGill University in April 2023 to validate the evaluative work that has taken place.

Page 03

Ethical Considerations Trauma-informed evaluation practices Trauma-informed evaluation practices actively avoid re-traumatizing participants by acknowledging the presence and prevalence of trauma for all people. Additionally, they recognize and respect each individual's response to trauma. It is essential to recognize the trauma an individual may have experienced directly, as well as intergenerational trauma and the trauma of systemic oppression. Trauma-informed approaches will respect the experiences of the individual or/and accommodate healing space and ceremony.

Colonial histories and academic norms Trauma-informed evaluation, particularly when conducted by academic and universityaffiliated projects, needs to recognize how these institutions have historically discredited or not acknowledged Indigenous knowledge. Universities have been active participants in the settlercolonial project and established harmful norms that create and perpetuate systemic oppression. Finding evaluative methods that situate the power with the participants is an important step in challenging the harmful academic norms of evaluation.

Ongoing and informed consent Ongoing and informed consent challenges the academic standard of seeking consent once at the start of a project and never returning to the subject. Ongoing and informed consent positions the power and authority with the participants. Throughout the project, researchers and project managers will continually go to the participants for consent to display their work and the data from the work. The process of getting consent includes clearly explaining where and how the work will be used, who will see their work, and the possible risks associated with consenting. Additionally, consent requires a real option for participants to say “no” without any negative consequences.

Authenticity of participant experiences and beyond It is essential that evaluative work centres the voices and perspectives of the participants.

Standard evaluation practices risks overlooking the experience of participants by not creating the space for or allowing participants and their networks to discuss the impacts of the project outside of the intended goals. Honouring the authenticity of participant experiences involves expanding evaluative practices to include the space for participants to reflect and discuss their experiences in a holistic manner. Additionally, it can require expanding the evaluative practice to include the perspectives of the participants' families and communities. This practice recognizes that projects can and will likely have impacts beyond the participants and that seeking out the authentic experiences of participants may require additional perspectives.

Page 04

Method

1

Collect ‘data’

Review everything you currently have access to, including artwork, conversations, emails, meeting notes, event reports, self-reporting, interviews, and social media.

No piece of information is too small Create strategies to continuously collect and also organize information. Identify how you collect, file and store information in a way that participants have consented to and in ways that will help with future evaluation Consider providing templates to participants. However, don’t rely on templates. Some folks respond well to reporting templates that they can fill out and submit, while other participants might find templates limiting and timeconsuming.

2

What is data?

Often the data used for evaluation is facts and statistics collected through interventions and structured methods of data collection, like surveys, focus groups or interviews.

Approaching evaluation more holistically and organically challenges traditional academic notions of what data is by centring participants' voices in what they chose to share.

Creating Questions to Study Change

The evaluative questions should reflect the desired or anticipated outcomes of the work. Consider reformulating anticipated, short-, medium- and long-term outcomes as questions.

Consider shifting evaluation timeframes from “years” to “phases” or "seasons".

Centering the evaluation around phases or seasons, versus years, maximizes the work that has already been done and provides greater flexibility for studying change.

Shifting away from the time frame of years allows the evaluation process to look at how activities or work from the beginning of the project might have fulfilled outcomes anticipated for later in the project. It also challenges linear notions of time and progress embedded in traditional forms of evaluation.

Traditional linear evaluation risks missing out on positive impacts and differences that might not have been anticipated at certain points in the project or that are happening in the lives of individuals beyond the participants, in this case including parents, siblings, survivors and community members, as well as in the larger networks of each community.

Beyond years or phases, projects might consider a seasonal approach, based in the local landscape and geography of the project

Page 05

3

Studying Change

Begin by reviewing previously collected information for evaluative work and categorizing it as answers to the new studying change questions Re-categorizing previous evaluative work highlights new perspectives and insights into previous work accomplished, answering new questions related to evaluating the overall project.

The process of reviewing and re-categorizing previous studying change work prevents site leaders and participants from having to provide the same answers to new questions.

Engage with the newly collected information, finding ways to connect the stories shared through art, reports, conversations, etc., to the studying change questions.

During the process of engaging and reengaging with the information collected and shared by the participants ensure there is open communication with site leaders and/or project participants.

Open communication creates the space for site leaders and/or project participants to provide evaluative insights if they have the capacity and desire to do so.

Seeing all the work compiled together can inspire participants and site leaders to remember different aspects of their work that might respond to or answer specific evaluative questions.

Open communication ensures consent is centered in the studying change process, actively seeking consent to include different components in the evaluation and in the final product of studying change is essential.

For MTW seeing all the studying change work compiled together to answer the evaluative questions inspired site leaders, as it showcased just how much the youth in the project had accomplished, especially during the incredibly difficult times of 2020-2022.

Identify potential gaps in the studying change work to determine if additional work needs to be done to meet the anticipated outcomes or if the projected goals and activities need to be modified.

Charts can be used during this step to make the studying change process and sharing process easier.

Example Studying Change Chart. Phase 1 for Eskasoni

Given that MTW works with three unique sites, charts were created for each site representing the two phases of the projects. Each chart had a column for the evaluative questions and the responses and a final column for additional supporting documents and comments.

Example Studying Change Chart. Phase 3 for Rankin Inlet

Page 06

4

Another Set of Eyes

After the studying change questions have been answered, and the site leaders and participants have consented to having it shared, consider having another set of eyes review the work.

The person reviewing the work might be community members, individuals with relevant lived experience, or individuals with careers as evaluators working in similar areas or with similar communities.

Additional evaluators with relevant experience can be helpful to review evaluative work and provide critical feedback.

MTW worked with Suzanne Methot is an Asiniwachi Nehiyaw (Rocky Mountain Cree) writer, editor, educator and community worker. Suzanne connected with the MTW project in 2019 when she spoke at the More Than Words Dialogue event. Suzanne reviewed the “Studying Change: Checking- In” report. She provided critical insight into language choices and affirmed the importance of the work happening in each community.

Additionally, Suzanne provided feedback on the method of organic evaluation used to create the report. Her feedback was instrumental in creating this document.

Benefits of Organic Evaluation Maximizes the use of readily available data Removes the burden of evaluation from participants, while creating space for them to participate to varying degrees depending on their capacity and interest Creates evaluation reports and documents that are authentic to the participants’ experiences for funders Creates an inventory and archive of project outputs that are accessible to participants Creates opportunities for new and unexpected impacts/benefits of research to be discovered

Page 07

Check Out More Evaluation Resources from MTW A ToolKit on Creative Approaches to Studying Change: Looking Back to Determine the Path Ahead is a toolkit designed to introduce readers to evaluation generally, and creative evaluation specifically. The toolkit comes from the More than Words and Networks 4 Change projects and is informed by Indigenous youth-led research. The toolkit includes ethical considers, a step-by-step breakdown and example of each evaluation method. Click the report cover to see the resource.

Acknowledgements We are grateful to Suzanne Methot's support and feedback in the process of creating this document. We are also grateful to the girls and young women in Eskasoni, Treaty 6 and Rankin, as well as their community leaders and academic supporters for sharing with us their stories and art-work. More Than Words is supported by the Social Sciences and Humanities Research Council of Canada (SSHRC), The Pierre Elliot Trudeau Foundation and Women and Gender-Equity Canada.

Recommended Citation Booker, E. (2023). Looking at Change in Community-Based Research: An organic approach.

Participatory Cultures Lab, McGill University

Principal Investigator

Project Coordinator

Dr. Claudia Mitchell claudia.mitchell@mcgill.ca McGill University

Leann Brown leann.brown@mcgill.ca McGill University

Connect Project Website: mcgill.ca/morethanwords/

Page 08