Developmental Evaluation Report

Developmental Evaluation Report January 2022

Project Background The Build Together Project provided the Nova Scotian community housing sector (CHS) - defined as co-op and non-profit housing for the purpose of this project - with opportunities to share experiences, successes, challenges, and ideas for working together. The work will contribute to a collective understanding of the supports and resources required to be successful in developing and sustaining community housing programs, projects, and services. As illustrated in the Theory of Change below (Figure 1), the program aim is to better equip, support, and empower the community housing sector to contribute to the availability of safe and affordable community housing in Nova Scotia. Contributing to this overall goal were three outcomes: • • •

Identifying effective supports and resources for the CHS. Drafting a road map for support, mobilization, empowerment, and sustainability of the CHS. Mobilizing the CHS in Nova Scotia. This outcome is less tangible but essential in bringing people together and building the momentum toward the overall goal.

Figure 1. Theory of Change

STFX EXTENSION

1

Methods This report summarizes the learnings that emerged throughout the project and how they influenced the path taken. It is an account of how the work took shape and evolved and responded to new information, data, emerging trends, and input through reflective practice.

A Developmental Evaluation (DE) approach was selected for the project to align with the work’s innovative nature, emphasis on learning and the underlying philosophy of the organization. DE focusses on real-time adaptive learning and is defined as a process that “…works with the team and help foster reflection, gather data and use it to help with planning” (J. W. McConnell Foundation, 2010). It is a flexible process that emphasizes individual and institutional learning.

To ensure learning and reflection were embedded throughout the process a learning framework was designed to complement the project in scale and purpose. The process began with understanding the project model, prospective journey, and identifying key learning themes and provocative questions that emerged from the team (see Table 1).

These framed a series of tools and events designed to gather feedback, monitor, and support implementation, as well as individual and team reflection and analysis.

Table 1. Learning Themes and Provocative Questions Theme 1: Equity and Inclusion in the Process • • • •

How do we reach communities and engage them in a consultation process in an online environment? How do we make sure all the voices in a virtual room are heard? What about those with poor or no connectivity?

How do we build equity into the project? How do we ensure marginalized voices are being heard on a digital platform?

How do we include / meet the needs of community organizations who are in different states of readiness?

What are we giving stakeholders for their participation? What are their ongoing motivations to participate?

Theme 2: Building Momentum for Long-Term Change • • • • • •

How is the project coalescing and continuing to support and strengthen the sector beyond the project? What is Extension’s role in moving this forward?

How do we move from consultation to engagement? How do we build momentum? How are we building coalitions and networks?

How do we articulate our implicit values in the process and networks?

What practical knowledge/skills/tools does the sector need to provide more/better community housing in Nova Scotia?

How will the program serve as a matchmaking or coordination platform for organizations in the sector?

How will we make the database dynamic and engaging for people?

Theme 3: Balance and Scope

STFX EXTENSION

2

• • • • •

How do we acknowledge important networks connected to – but just outside the scope of the project?

How does the community sector engage with various levels of government, for-profits, the Health Sector, Social Services, etc.?

What are the strengths and weaknesses of these partnerships?

What are the boundaries or scope of the project? How do we avoid scope creep?

How do we avoid burn-out? How can we reduce the workload without losing the most valuable aspects of the project?

Theme 4: Learning and Reflection • How do we integrate reflection into a fast-paced program?

• What are enabling factors or ingredients of our achievements and how do we maintain/replicate them? What can we do better?

• How do we integrate real-time data into our decision-making?

• Can we document the online process in a way that facilitates training others?

Reflective practice and learning unfold in surprising and dynamic ways. While a learning framework cannot possibly catch all the rich nuances embedded in every project, it can provide a net to ensure the critical pieces are captured and shared with a wider community of learning and practice.

A snapshot of the project learning framework is captured in the diagram below. Starting with the blue boxes, the key stakeholders (participants, Engagement Team members, project staff, and representatives of the Centre and Nova Scotia Government) were identified as people able to provide feedback on the project or the sources. A set of tools were designed to collect feedback in a light but thoughtful manner (see table 2).

The information generated from these tools and the project documentation and analysis was reviewed monthly and formed the basis of a participatory analysis. The resulting lessons learned were incorporated into the project activities.

Learning Framework: Strengthening the Community Housing Sector in Nova Scotia

Project Activity

Documentation & Analysis

Session Feedback Forms Participants Representatives from NS Government & the Centre

Session Debriefs Project staff Pre – post interviews Engagement Team

Monthly Analysis Sessions: designed to foster participatory analysis of the data and liveexperiences over the past month

Regular team check-ins

Insight Journal

Meeting Feedback Form

Incorporate Lessons Learned

STFX EXTENSION

3

Table 2 Learning Framework Tools Session Feedback Form Session Debriefs

Pre/Post Interviews

Regular Team CheckIns

Insight Journal

Meeting Feedback Form

Monthly Analysis Sessions

A short online survey was created to gather participant feedback at the end of each session. The purpose was to ask for participant feedback on the sessions and make improvements.

The facilitators coordinated their own informal debriefs after each consultation session. Highlights and significant lessons learned were recorded in the Insight Journal.

The project was launched in the middle of a global pandemic in which many people are working from home and communicating online. This limited the informal discussions and side conversations that often provide richer insights into the process of implementation and potential ways to accentuate results. In lieu of these opportunities, the Monitoring, Evaluation and Learning (MEL) lead met with each of the core project staff before and after the project. The MEL lead also held a reflective project wrap up debrief with the project team.

The staff members held regular bi-weekly meetings throughout the project.

The focus was on understanding how the team was doing and to either (a) reflect on a provocative question, and/or (b) address something arising from the work. The facilitated discussions also aimed to build teamwork and understanding.

The purpose of the insight journal was to encourage reflection and learning by asking project staff to share observations, insights, stories, lessons learned or questions about the project as it is being implemented.

The journal entries were based on three simple open text questions.

1. Please share an observation, insight, expectation, story or question about the project.

2. Why is it significant?

3. Please provide three words describing how you feel about how the project is progressing.

As a critical component of the project, the Engagement Team provided general direction for the project and support for the equity lens. As such, a small survey was designed to gather feedback on the meetings and provide a high-level check-in. A project wrap-up reflective debrief session was also held with the Engagement Team.

Once a month all the data from the project and monitoring and evaluation process was reviewed for updates and highlights. The information was used to prepare a monthly project newsletter and support participatory analysis and reflection among the team.

Summary of Learnings Learnings from the four thematic areas helped shape the project journey and will also influence the proposed path of future work. Lessons learned from reflective conversations will also influence the internal team building at Extension. This section will provide a summary of the overall learning captured under each theme, followed by a more detailed discussion of the developmental moments these learnings and reflections sparked. Before moving into thematic learnings however, there are two variables external to the identified learning themes that had an impact that rippled through all aspects of the project – the ongoing Covid 19 pandemic and the work happening in the broader affordable housing ecosystem.

STFX EXTENSION

4

A Note on Covid 19 Impacts The impacts of Covid 19 on the work were present throughout meeting notes, reflective sessions, and debrief interviews. The original project plan was built on a model of inperson engagements and regional place-based outreach. Pivoting to 100% online engagement meant not only a steep learning curve for the project team, but also for many of the community groups and stakeholders the project hoped to engage. A consultant was brought in to support the online engagement strategy and the team adapted facilitation and meeting styles to fit emerging needs and styles of community stakeholders. Detailed, heavily structured facilitation guides and online techniques that required a higher level of technical skills, for example, were often traded for prompts and conversation themes, to allow participants more open time to speak and engage with one another in a more natural way in small break out rooms on Zoom.

Along with the technological learning curve, the move to 100% online engagement flagged equity and access issues. While online sessions enable access for some who may not be able to attend in-person events as often, the digital divide in terms of access to high-speed internet and equipment such as webcams and microphones poses very real challenges for equitable engagement. Zoom and screen fatigue soon settled in as groups and stakeholders found themselves spending most of their time online. The demands of the pandemic on service providers and groups within the community housing sector, both professionally and personally, was also taking its toll. This context left participants burnt out and without a lot of energy to engage in consultations or planning outside their daily workloads.

The Affordable Housing Ecosystem The conversations and actions around affordable housing have been gaining momentum over the past 24 months. On the Federal side there is a new housing minister, and provinces and municipalities have also been increasing their focus on the issue. While the current spotlight on affordable housing provides a window of opportunity to push the Build Together Project’s findings forward, the busy affordable housing ecosystem also posed some challenges for the work. Due to various delays and timing issues beyond the project team’s control, Build Together launched and began consultations at the same time that the Government of Nova Scotia was holding engagement sessions across the province that ultimately contributed to the NS Affordable Housing Commission’s final report released in May 2021, “Charting a New Course for Affordable Housing in Nova Scotia”, and the Canadian Centre for Policy Alternatives – Nova Scotia, with the Housing for All Working Group, also released its report “Keys to a housing secure future for all Nova Scotians”. While the energy and momentum around housing was positive for the project, it led to some confusion and consultation fatigue among stakeholders. Reflecting on the need to be clear where the Build Together work fits with the emerging plans and existing work, the team developed a comparative analysis that was shared with stakeholders and on the project website (https://coady.stfx.ca/wp-content/uploads/2021/10/CHS-Comparative-Analysis-FINALSept-29-2021.pdf).

STFX EXTENSION

5

Theme 1: Equity and Inclusion in the Process Equity and inclusion remained front of mind throughout the project. It was intentionally built into the values of the Engagement Team. Project team members and stakeholders felt those values had been actioned throughout. Gaps in first voice and lived experiences from African Nova Scotians, Indigenous community groups, LGBTQ2S+ organizations and community members, among others, was recognized early on. Upon reflection the project team increased efforts to reach out to African Nova Scotian and Mi’kmaw community members, holding two sharing circles to deepen their understanding of the issues and barriers to engagement. Key learnings that emerged from this process included the need to step back and take some time to understand why some groups may not feel comfortable or willing to engage in the process, and to understand the history and context surrounding this reluctance. The team was challenged to think about why all communities would engage with the work, and what type of reciprocity existed. They also explored questions around what it means to decolonize and decentralize whiteness in the work, and the challenges this poses when working within colonial funding and project-based structures.

The need for an intersectional approach also emerged through the group’s focus on equity and inclusion. There was an increased recognition of the complexity and intersectionality of identities of those working in, and needing support from, the community housing sector. Elements of identity such as age, sexuality, gender identification, racialization, etc., all come together in many ways to impact access to housing and the supports people and communities require. While the internal project team recognized this complexity from the start, there was consensus that there needed to be more learning and strategic thought put into how to address equity and inclusion, and ensure an intersectional lens, going forward.

Theme 2: Building Momentum for Long-Term Change A learning identified by several internal project team members was their increased understanding of just how big the community housing sector is in Nova Scotia, but also how disconnected and dispersed it is. By continuing to reach out and broaden the consultations’ reach as the project progressed team members and stakeholders indicated they felt there was an increased sense of connection and joint momentum being built. To continue that momentum, a collective understanding was beginning to emerge by the end of the project, that the group will need to find a balance between ongoing engagement and consensus building, and action and demonstrable results.

Some Engagement Team members felt that moving forward with concrete action may also improve the project’s reach and participation, as others who have not yet come to the table may start to be drawn into the work when they see results starting to take shape.

The survey data provided a useful starting point for gaining a deeper understanding of the capacity building needs of the community housing sector. However, there was an identified need to delve further into specifics around what supports and targeted funding are required to move the sector forward. There was also a growing understanding that given the diversity within the community housing sector, the capacity building plan that STFX EXTENSION

6

emerges will need to be adaptive and responsive to multiple needs and cannot be a one size fits all approach.

Theme 3: Balance and Scope The scope of the project shifted and stretched as the work progressed. Originally the community housing sector and housing co-ops were the target groups, but as the project moved along, a lot of the really engaged groups emerged who were organizations that intersect with housing (shelters, women’s groups, municipalities, addictions services, etc.). They represented organizations that are impacted by the housing crisis but are not typically at the table when we talk about the community housing sector.

What the project heard through surveys and consultations was that an increasing number of groups wanted to be at the table and discuss community housing because they see it as a huge barrier to the work they are trying to accomplish in their respective sectors. The broadening of the scope of who was involved led to concerns about mission creep. Project team members however, emphasized the importance of having these groups at the table, as one team member shared: “their voices added a lot of humanity and a dimension of urgency to the need to address these problems to the people most effected”. The growing understanding of who wanted to be involved in the work influenced the final model in the form of the regional approach being built to allow space for all these voices and a focus on place-based regional needs.

Questions around balance and scope remain as the project ends. While there is broad consensus that the wide engagement process was a success, there was also learning around the time and effort taken to broaden the scope of work, and the need for a balance between trying to be everything to everyone and remain focused on the core of the work to move from consultation to action.

Theme 4: Learning and Reflection Learning and reflection were there from the outset of the project as the team had to quickly adjust to virtual engagement and pick up new technologies and methods.

Members of the project team who were most involved with the public engagements indicated that more time to adjust and get up to speed on online facilitation would have led to a smoother process.

The need for more time came up frequently in notes and reflective conversations, contributing to a consensus that the timelines for the overall project were too tight and that the project team could not keep up with the project workplan. In addition to time to adapt to new engagement tools, time for more upfront relationship building, learning data analysis tools, and to engage the whole team in course change and strategy sessions were identified as areas that felt rushed.

In general, however, there was a positive feeling toward the rapid learning that did happen, with team members indicating it would serve the group well in future projects.

STFX EXTENSION

7

As one project member shared “we learned to ask better questions, facilitate better conversations, and share our information out in a better way”.

In addition to the rapid incorporation of learning around technology and online engagement methods, learnings from the first three thematic areas were consistently incorporated through the project, leading to developmental moments and pivots in strategy that are described below.

Key Developmental Moments The section that follows identifies ‘key developmental moments’ in the life of this project that occurred because of the reflective practice built into the learning agenda (see figure 2 for project journey). Developmental moments can be a helpful way of articulating the productive challenges and key learning experiences that emerge throughout the life of a project. Dozois, Langlois and Blanchet-Cohen (2010, p. 41) define developmental moments as “instances when the initiative shifts or moves forward in some significant way; moments of clarity, strategic insight, serendipity, connections, and/or movement”.

It is important to emphasize that these moments can only be identified in the moment, or sometimes only hindsight – it is not a timeline developed in advance of the group but rather emerges in collaboration with the project team, stakeholders and partners, and in relation to external events.

Figure 2 Build Together Project Journey

Developmental Moment 1 – Increased focus on relationship building (Spring-Fall) The learning that was happening on equity and diversity, and balance and scope led to conversations in April 2021 about who was coming to the engagement sessions and STFX EXTENSION

8

completing surveys, what were the sessions achieving, and what voices were being heard. This reflection led to the recognition of large gaps in representation and the need for more intentional relationship building with African Nova Scotian and Mi’kmaw communities. There was also an identified need to deepen relations with the sector as a whole to increase participation and to set this work apart from the earlier two consultations which were releasing their reports in May. The decision was made to shift away from larger, broad engagement sessions and instead focus on holding one-one-one interviews and a focus group meeting while continuing to build relationships and growing the project team’s understanding of the community housing sector. As a result, the originally planned 22 engagement sessions were reduced so the team could focus on a more in-depth analysis of the survey data, spend more time building relationships, and hold a session in May to help prepare for deeper, more meaningful conversations with Indigenous community members.

Developmental Moment 2 – Cancel second survey in favour of deeper analysis of existing data (Spring and Summer) How to analyze and communicate the data emerged as an internal challenge, with diverse opinions on how to look at the data, as well as the purpose of the analysis and subsequent knowledge transfer. It took time to develop a shared vision of how the data needed to be presented. The different approaches to data were complicated with the steep learning curve team members faced with the qualitative data analysis software (NViVo).

As challenging as they were, these conversations led to the decision to take a deeper look at the trends and themes that were emerging beyond the engagement numbers and contributed to the production of thematic briefs which were highlighted as key pieces of knowledge transfer by Engagement Team members.

This process led to a deeper understanding of how to share findings in a way that guides decisions, reflecting a move from pure knowledge transfer toward knowledge mobilization. The deeper dive into the data and thematic analysis also allowed time to focus on the comparative document, to tease out the areas of overlap, intersection, and difference between Build Together and the two reports released in May, contributing to a deeper understanding of the ecosystem in which the project was operating.

Developmental Moment 3 – Shift to multimedia engagement and knowledge transfer (Fall) The initial plan was to complete another round of broad engagements in the fall, to seek feedback on the data that had been analyzed over the summer. Reflecting on experiences with earlier engagements, and hearing concerns about Zoom fatigue and engagement burnout, the group pivoted to a digital, asynchronous engagement phase.

A video was released, and a panel presentation was recorded and shared accompanied by a survey to gather feedback on the findings of the project to date and inform the model that was taking shape. This culminated in an in-person workshop in late October, STFX EXTENSION

9

bringing everyone together in one place for the first time. This pivot was highlighted by several Engagement Team members as a clear indication the project team was learning how to use a mix of synchronous and asynchronous engagement methods to make the best use of people’s time while still collecting feedback and input to guide the work.

Developmental Moment 4 – Model redesign (December) The model that began to emerge from the work had a significant shift in November/December (see figure 3 below). The shift was to move to a design that was less hierarchical/colonial in how it visually represented the relationships between the stakeholders and project partners.

Figure 3 Draft Model

Moving Forward Learnings gleaned throughout the developmental evaluation and implementation of the Learning Framework will also be used to inform the path forward (figure 4). The table below provides a summary of key insights and questions the project team feels have been resolved, requires further learning and exploration, and are priorities to be taken forward into the next phase of work of implementing the roadmap.

What are we sure about?

Our communication strategy was strong. The STFX EXTENSION

What are we still thinking What will we take about? forward into the next phase?

Balance – how do we The learning we’ve done balance the needs of around how to be nimble 10

What are we sure about?

format and amount of documentation shared was appreciated.

There is a solid foundation and broad consensus for moving the work forward.

What are we still thinking What will we take about? forward into the next phase? organizations to see action and the importance of versus the participatory adapting to evolving engagement process? circumstances.

How do we continue our decolonizing and decentering whiteness?

What do we need to unlearn and learn to continue this work?

There is value in the approaches taken so far and the learning that led us to where we are.

How do we build sustainable online We need to continue to be engagement and avoid intentional and thoughtful burning out stakeholders around equity and and community members? diversity.

How to we ensure this learning gets carried forward into future projects?

Continued growth of online engagement and tech skills.

A focus on diverse groups and first voice.

An understanding to focus on passion as well as skills alignment when putting together the project team.

Figure 4 Draft Roadmap

STFX EXTENSION

11