February 5, 2010
Hawai’i Community Foundation’s Youth Matters Initiative Final Evaluation Report
Weaving Networks: Hawaii Community Foundation’s Youth Matters Initiative Final Evaluation Report, February 2010
Presented to:
Hawai’i Community Foundation Chris Van Bergeijk Carrie Shoda-Sutherland
Presented by:
Melanie Moore Kubo, Ph.D. Cati Brown, Ph.D.
1
February 5, 2010
Hawai’i Community Foundation’s Youth Matters Initiative Final Evaluation Report
INTRODUCTION What is Youth Matters? The Youth Matters Initiative was launched in late 2007 with support from the W.K. Kellogg and Hawai’i Community Foundations to improve the long-term outcomes of youth in Hawai’i through a collaborative leadership approach. With help from the Monitor Group, and other advisors, the Foundation designed the Youth Matters Initiative as a network of forward-thinking youth program providers from throughout the state of Hawai’i. The Youth Matters Initiative has concluded its second year of implementation, and the final year of direct management of the youth providers’ network by the Hawaii Community Foundation. The work of the Initiative will continue to be managed by the Foundation throughout the grant term, but this work will focus on the activities other than building the network. The first year, 2008, culminated in the production of a Youth Outcomes Scorecard, which was designed to guide the work of initiative partners. Additionally, the first year of work included the creation of a 19-member Youth Matters Network made up of youth program organizations statewide, conclusion of a set of “Positive Deviance” interviews meant to outline unusual paths to success in Hawai’i, initial discussions with stakeholders who might be able to provide data to underpin scorecard measurements, and the incubation of nine projects designed by Network members to enrich their capacity to assist youth, or move their spheres of influence toward positive scorecard outcomes. This second year of the initiative, 2009, focused primarily on catalyzing the collective work of the nineteen youth-serving organizations in the network. The Foundation’s expectation was that Action Learning Groups (ALGs) within the Network – focused on Policy Change, Quality Programming and Youth Engagement – would develop and advance Network members’ organizational capacity as well as make direct changes on youth outcomes statewide, primarily through a systems change approach. Additionally, the Youth Matters Initiative’s second year was intended to yield a sustainability plan for the Network, as 2010 would mark a transition of the Foundation’s direct operational role. In July 2009 at the Network’s annual retreat, unexpected events changed the nature of the Network’s activities for the second half of the year, but overall, the year ended with a transition plan in place. This report will detail the evolution of the Youth Matters Initiative and Network in 2009, and discuss the outcomes of the initiative.
What Was HCF’s Overall Goal for the Initiative? The Youth Matters Initiative (YMI) was launched in January 2008 with the goal of creating positive change through collective action in the systems and conditions that affect youth ages 13-24 in Hawai’i. Nineteen youth-serving organizations from around
2
February 5, 2010
Hawai’i Community Foundation’s Youth Matters Initiative Final Evaluation Report
the state were brought together to form a network. The “collective action” envisioned by the Foundation was to be driven by the Network members. HCF combined several strategies into the YMI. The strategy that received the most investment of time from HCF staff was the selection and convening of a network of youth-serving organizations. Network members receive two types of grants: participation grants to cover travel and cost of time away, and program-related grants to support individual or collaborative youth development projects. Network members came together approximately 6 times a year for education and collaboration. HCF also emphasized the role of research and evaluation in the initiative. By establishing the Hawai’i Youth Outcomes Scorecard, YMI placed research on youth wellbeing and evaluation of the initiative’s progress toward its stated goals at the center of their efforts. Building the evaluation capacity of the Network members was an important objective as well.
Plan for This Report Five sets of research questions guided our evaluation of YMI. A strategic assessment report provided in June 2009 addressed the first two sets of these questions: questions about the Foundation’s role in YMI, and about the characteristics and functioning of the YM Network. These questions are below: A: Questions about the Foundation's Role
A1) Is HCF implementing the Youth Matters Initiative smoothly and according to plan? A2) Is HCF providing facilitation and technical assistance to the Network that is necessary and sufficient for it to be productive?
B: Questions about the Characteristics and Functioning of the Network
B1) What are the characteristics of the Youth Matters Network? For example, is it a diverse and representative network? Does information flow throughout the network, or are there communication gaps? Are ties between network members strong or weak?
A3) Are HCF's decisions about the Youth Matters Initiative strategic with regard to the targeted B2) Does the network have access to external indicators? Are the decisions about resource knowledge and resources? Does new knowledge allocation and the activities of the Youth Matters affect network decisions? Is the network a Initiative likely to have an impact on the scorecard learning community? indicators? B3) Does the network include youth members? What is their role in decision making within the network? B4) Are the network’s decisions and actions strategic with regard to the targeted indicators? Are network-level projects likely to have an impact on the scorecard indicators?
3
February 5, 2010
Hawai’i Community Foundation’s Youth Matters Initiative Final Evaluation Report
The remaining three sets of research questions focused on changes in youth development practices at community-based organizations, the community and policy context affecting the initiative’s work, and youth outcomes. Given the changes in the nature of the YMI in 2009, including the Foundation’s decision to reallocate evaluation resources, answers to these research questions must be postponed indefinitely, though there is a possibility that the YM Network could take up the investigation of these research questions at some point in the future. Instead, this final evaluation report is organized in the following sections: ♦
Summary of Evaluation
♦
Evolution of the Network in 2009 and Early 2010
♦
Network Connections Resulting From Youth Matters Initiative
♦
Lessons Learned About Network Initiatives
Summary of Evaluation See Change conducted the following evaluation activities for the Youth Matters Initiative in 2009: •
Building evaluation capacity for Hawai’i youth-serving organizations and Youth Matters Network: o Evaluation 101 Workshop for about 50 youth-serving organization staff o Evaluation 101 webpage including the presentation used in the workshop (https://sites.google.com/site/evaluation101training/) o Design and implementation of Youth Empowerment Stories (YES) program, involving youth from YM Network organizations in evaluation and youth engagement training. A staff member from the Boys & Girls’ Club of Kauai also observed this program. o Presentations and workshops within Network meetings: Evaluation introduction; Theory of Change training; Presentation of the Wheel Implementation Tool; YES video and youth-led evaluation; Youth Leadership & Engagement strategies (led by Ed Center) o Digital Storybanking Workshop for Maui Network members and other Maui-based youth-serving organizations
•
Theory of Change development for HCF o Calls with ALGs for input on Theory of Change o Work with HCF staff to shift theory of change from “fireball” to “hourglass”
4
February 5, 2010
Hawai’i Community Foundation’s Youth Matters Initiative Final Evaluation Report
•
Site visits and interviews o Site visits and interviews conducted at ¾ of all Youth Matters Network Members (the remaining ¼ of visits were scheduled to take place after the Foundation chose to reallocate evaluation funding)
•
Program Quality Assessment o Training and discussion delivered to Quality Programming Action Learning Group in May o Updated and delivered to HCF in Fall 2009
•
YES Project – 11-day program and Evaluation Video o Design and implementation of the Youth Empowerment Stories Project, a youth-led video evaluation of what makes a great youth development program o With in-kind assistance from HCF, See Change provided youth from YM Network organizations with expert training in Youth Engagement and Empowerment, Videography, Editing, Interviewing Techniques, Hands-on Research and Evaluation o Delivered professionally polished youth-led and youth-edited video to HCF in August o Delivered video data files (from interviews and on-site video shooting) to HCF in Fall
•
Facilitation consulting and debrief provided for 4 Network meetings
•
Strategic Analysis: Report regarding HCF’s implementation of YMI
•
Ongoing strategic consulting with HCF staff regarding Network development and support
•
Network Survey Redesign (to have been implemented on day 2 of July’s Network meeting)
Evolution of the Network in 2009 and Early 2010 As See Change’s Strategic Assessment Report in June concluded, the YM Network was in a challenging position at mid-year. Although pockets of collaboration existed, and the Network members were positively disposed towards each other, the group as a whole had failed to gel around common goals or action agendas. There was very little Networklevel work taking place between Foundation-led meetings, and our interviews with Network members revealed a high degree of frustration about the YM Network. Network members did not understand the purpose of the group, did not have a sense of its goals, and did not have a clear or uniform vision of how (and whether) the Network would manifest beyond the period of the Foundation’s direct involvement.
5
February 5, 2010
Hawai’i Community Foundation’s Youth Matters Initiative Final Evaluation Report
The Network was only one component of the YMI. Other components included the Foundation’s independent work building support for network initiatives among other key stakeholders in the state, and completing the Scorecard, including working with data experts to develop needed information. However, feeling the pressure of a 2010 deadline for the YM Network with respect to leadership hand-off, the non-Network activities of YMI took a backseat in 2009. By mid-year, HCF staff were almost entirely focused on organizing and supporting the Network, which included managing the grants involved and managing consultants, including evaluators, meeting facilitator and the technical assistance provider (also a Network member) hired to coordinate and drive the work of the ALGs between Network meetings. Additionally, HCF staff was also planning and coordinating all logistics and content for the Network meetings, which took place every two months. July Retreat At the July 2009 YMI Network two-day annual retreat, a number of grantees became concerned that they were learning for the first time of HCF’s intention not to fund a second year of their project grants. Although there was never a guarantee from the Foundation of second-year funding, there was a perception among many Network members that this funding would be provided, and many groups had based programming and hiring decisions on the expectation of these funds. Creating an additional challenge, the way in which the information about second-year funding was conveyed to the grantees was indirect. Rather than receiving a direct communication from Foundation staff, grantees heard about the Foundation’s decision through conversations the Network had during the first night of the retreat. By the time many Network members arrived on the second day of the July retreat, they were quite angry about the situation. The planned agenda for the second day was set aside, and group members addressed the Foundation about what many felt was a serious breach of honor and trust. For their part, Foundation staff felt that Network members had failed to grasp that the intention of the program grants was to accomplish the broad goals of the Initiative, including systems change, and that grantees had incorrectly assumed that the funding would be continued without evidence of progress toward these goals. The meeting was tense and emotional, but concluded positively with three key outcomes: 1. The stated intention of the Network to stay together and continue to work with Foundation staff on the YMI, 2. An affirmation by the Network members of their understanding and valuing of the Foundation’s role and position, 3. A statement by the Foundation that they would consider the emerging requests of the network. The Network requested a high degree of autonomy going forward, including decisionmaking power with regard to budget, and a high degree of transparency from the Foundation, so they could better understand the Foundation’s goals for YMI, and tailor their activities accordingly.
6
February 5, 2010
Hawai’i Community Foundation’s Youth Matters Initiative Final Evaluation Report
Transition of Network Operations to Network Members Through a proposal process, the Foundation allocated $50,000 to the Network for ongoing operations in 2010. A steering committee of Network members formed, which submitted a proposal for the use of those funds. Their proposed focus is on building a “community of practice” among Network members, exploring the issue of youth development program quality. Network members would manage the planning and facilitation of their meetings, and there would be no further participation grants, nor program grants attached to the participation in the Network. HCF will use the remaining funds in the Youth Matters Initiative budget for 2010 to work on a range of projects in-house, including, but not limited to, the following: ♦ Completing the Scorecard ♦ Youth Landscape Mapping for the state ♦ Distribution of the film produced through the YES Program At the time of this report, the Network’s proposal was pending approval by the Foundation, and there was optimism among Network members and Foundation staff about a productive relationship going forward, with a shared goal of improving the lives of the state’s youth.
Network Connections Resulting From Youth Matters Initiative One of the key outcomes of the Youth Matters Initiative, and in particular the Network, has been the creation of substantial connections among Network members, resulting in professional friendships, knowledge transfer, collaborative capacity-building, and even formal partnerships. The instances outlined below are the most evident and productive Network connections. These should not be considered the full extent of connections resulting from YMI, as we also heard in our interviews that Network members were developing less formal connections that are less visible but also valuable. ♦
New Safe House for Teens on Oahu The Salvation Army (in Hilo, HI) & Partners in Development (in Oahu), partnered to create a new safe house for teens on Oahu, based on the successful model implemented by the Salvation Army on Hawai’i Island.
♦
Kauai Youth Network The Boys and Girls Club of Kauai & the Kauai YMCA partnered through a collaboration grant to create the Kauai Youth Network. This network of youth devoted to social change has already been making positive changes in Kauai for youth, including working with schools to provide healthier lunch options.
♦
Youth Asset Building Learning Hui HACBED led a learning hui of four other grantees (KEY Project, Kula No Na Po’e Hawai’i, Paepae O He’eia, MA’O Farms) to explore asset building theories and opportunities and support each other in collaborative learning. This multi-island hui is the first of its
7
February 5, 2010
Hawai’i Community Foundation’s Youth Matters Initiative Final Evaluation Report
kind to investigate asset building as a way to address the systemic economic and resource challenges faced by many Hawai’i youth
Lessons Learned About Network Initiatives A key part of the initial intent of HCF in designing the YMI was to experiment with the strategy of network building. The Foundation learned a great deal through the initiative about the nature of “network weaving.” These primary lessons emerged: ♦
Network members must be selected, in part, because they are “networkable.” In other words, an organization that has not participated in collaborative work previously, or that is highly isolated in the way it conducts its work may not be a good candidate for inclusion in a networked initiative. While it might seem that isolated organizations are the ones that should be invited to a network, there is a level of readiness for network participation that must be assessed in advance, both for the benefit of the network and the organization itself. For example, an executive director must be able to think beyond the needs of his own organization to understand his potential role in a network. As network-building was a new endeavor for HCF, organizations that were invited to participate were not screened for “network readiness.” The Foundation believes this is an important criteria going forward, as they utilize network strategies in other issue areas.
The selection process was most critical. I don’t know what the selection process was, but I question how they chose which organizations. I don’t feel that just because they were a youth serving organization, they should be there. But there were plenty of youth-serving organizations that were not there, so that wasn’t it. There were some organizations that I wasn’t sure why they were at the table. HCF is green at network building. They probably didn’t know what they were looking for. Cheryl Kauhane Lupenui, Member, YM Network Steering Committee
♦
Networks are built on strong relationships, and strong relationships take time and intentionality to form. The Foundation poured more energy and resources into the Network than they had anticipated. Additionally, some reorganizing within HCF resulted in staff changes with the Network. Initial strong relationships that had taken time and effort to create shifted, requiring new staff to work twice as hard to “catch up” on relationship-building. Knowing now the time commitment and importance of staff continuity for building a network based on strong relationships, the Foundation is in a position to make more informed decisions about network weaving going forward.
When I walked in, I thought it was a community of practice. But you have to intentionally take time and create space for them share their work. I thought network weaving was an organic thing that automatically happened, but now I’m not as sure. The focus of meetings was more actionoriented, figuring out a plan; more “Let’s go, let’s go.” There was not a whole lot of space for building a community of practice. The Foundation thought that Network members were
8
February 5, 2010
Hawai’i Community Foundation’s Youth Matters Initiative Final Evaluation Report
connecting offline. But when we brought them together, we discovered that that was the only time they spoke. There was not a lot of time to develop relationships. Carrie Shoda Sutherland ♦
Funders may not be the appropriate front-line network weavers. Since networks are built on strong relationships, a first task for a front-line network weaver is building comfortable, trusting, level-playing-field relationships. This is practically impossible to do in a funder/grantee relationship, where decision-making power will always naturally rest with the funder. The Foundation in the future will look for intermediaries to support network weaving.
This is a challenge for network initiatives in philanthropic space. The culture of nonprofits is to be only responsive to funders, and not lead or come up with their own project. HCF set that same type of culture, too. I think HCF did set the agenda. But in the literature, it says that it’s harder for funder to build a network. They can play multiple roles, but will always be a funder first. Cheryl Kauhane Lupenui, Member, YM Network Steering Committee Forming relationships between grantee and funder really complicates the boundaries. We didn’t use an intermediary. For us to do all these roles complicates things. Now, going forward, we will be more distant, and use intermediaries. Carrie Shoda Sutherland ♦
Working in networks involves skills and knowledge about networks. Scholarship has emerged over the last two years that was not around when YMI was formed. Training for all network members in key network concepts could have helped set the stage for a more informed network creation process. The Foundation will look in the future for one of two things: training opportunities for staff and network members about network theory and implementation, or a quorum of network members already inculcated in network theory.
Building networks is not easy. You have to have some knowledge of building networks in the group. That knowledge is not present in the group outside of Marlene and me (YWCA of Oahu). But we’re green, we’re learning. Community building is the same as network building, but on a different scale. You’d think people who do community building for a living would know how to operate well in a network, or with group dynamics, but it’s not always true! I’m a proponent of training in that area. We’re proposing to evolve the Network in 2010 to next stage of alignment. We were successful in creating a “network of connectivity.” But the idea that HCF had at the beginning was that we would be a “production network.” In order to even get close to being a production network, we have to get to the next stage: affinity network. An affinity network aligns around a common, shared value proposition, a shared identity. We are aligning around the ALG work, the quality programming work. Out most common denominator is the desire to provide quality youth programming. We’re building off the foundation already established. Our outcome will be a document of guiding principles of youth programming in Hawaii. Cheryl Kauhane Lupenui, Member, YM Network Steering Committee
9
February 5, 2010
Hawai’i Community Foundation’s Youth Matters Initiative Final Evaluation Report
CONCLUSION The Network, a core aspect of this Initiative and the main focus of the 2009 Initiative implementation, is in the process of redefining itself and its goals outside of the specific context of YMI. The Foundation, while taking some time to look back at the work done to date, and especially lessons-learned with respect to network implementation, must look ahead to intense work towards overall YMI goals for this last year. The goal of this report is to document the overall outcomes of YMI to date, to concretize knowledge about opportunities for growth, and additionally fuel and inform the critical work to be done in 2010. With that in mind, what are the overall outcomes of the YMI? Spontaneous collaborations are a significant outcome of the YMI, representing work and knowledge-sharing that would simply not have happened on this scale in this short amount of time without the direct support of the YMI and the YM Network. These collaborations have resulted in new services for youth, and better connectivity for participating youth-serving organizations. As highlighted in the Strategic Assessment Report, there are ways in which the Networkmembers’ program grants did support progress towards the Youth Matters Scorecard’s broad goals for youth: Scorecard Area Plentiful Opportunity
Projects
Reason for anticipated impact
Kohala Center – Developing Career Paths to Support Hawai’i’s Sustainable Future
Adequate Preparation
Youth Asset Building Learning Hui
Connection to Community
Kaua’i Youth Network
Resilience
Ke Ala Kupono Youth Leadership Program
Having identified sustainable agriculture as a potential positive living wage career path for youth, the Kohala Center is introducing the idea of this opportunity to high school youth to pique their interest and directly prepare them for possible future sustainable agriculture careers Directly addressing wealth and asset building in a culturally sensitive setting will increase awareness of assets and give youth tools to build assets (monetary and non-monetary) of their own This youth-driven action network is building a policy agenda, and has also already set its sights on impacting aspects of their community, changing the status quo to create community systems that include and respond to youth voice. Neighborhood Place of Wailuku is successfully engaging The GAP Youth Training Services to create spirituality- and Hawaiian valuescentered retreats and workshops giving “at promise” youth the opportunity to orient their identities and actions to Lokahi and Ho’omana.
10
February 5, 2010
Hawai’i Community Foundation’s Youth Matters Initiative Final Evaluation Report
While the anticipated impacts of Network collaboration projects have not yet come to fruition at this time, if these broad goals remain a focus for HCF and for the emerging Network, it is perhaps useful for the Foundation to check in periodically with the progress of these initiatives, which were largely borne out of the YMI. As a learning organization itself, HCF has benefitted from the Youth Matters Initiative by becoming incubator of network lessons learned and promising practices. HCF now has the experience and insight to create, support and set-loose other successful networks that might also eventually be poised to create systemic change for Hawai’i. This following hourglass theory of change emphasizes the sustainability of the network beyond the initial formation and launch by HCF. Assumptions are also made explicit in this theory about the importance of network members taking on leadership roles outside their organizations, and at a statewide level. We believe it is potentially useful for HCF to review the Network’s work over the coming years against this theory of change, to determine whether some of the expected changes do indeed take place.
11
February 5, 2010
Hawai’i Community Foundation’s Youth Matters Initiative Final Evaluation Report
Lastly, the Youth Matters Initiative has truly given birth to a stand-alone Network of youth-serving organizations from across Hawai’i that are devoted to quality programming. Program quality has been shown to influence the youth outcomes a program can expect. A program with the knowledge and tools to enact high quality programming is much better poised to achieve desired outcomes. Thus, as the Network develops and distributes information about what it means to be a quality youth-serving program in Hawai’i, they will be systemically enabling better outcomes for youth involved in programs reached by the Network, or by Network influence. The Youth Matters Network is a network of connectivity. That has brought value to the individual organizations that make up the network. Every network member could cite direct benefits for their organization. The level of commitment to the network is very strong, and members are expressing willingness to be there next year. For organizations that have not been part of a network before, and a funder that has not done networks before, that’s impressive. Cheryl Kauhane Lupenui, Member, YM Network Steering Committee
12