4.0 FUND III ANNUAL EVALUATION: 2022

PUBLICATION

JUNE 2022

4.0 FUND III ANNUAL EVALUATION: 2022

Authors: Allison Karpyn, PhD Nicole Kennedy, MPA Candace Young, MS

Funding Agency: 4.0 Schools

1

CRESP is committed to addressing education and social policy challenges with rigorous, relevant research.

The Center for Research in Education and Social Policy (CRESP) within the College of Education and Human Development at the University of Delaware conducts rigorous research, program evaluation, and policy analysis to help policymakers and practitioners in education, community health and human services determine which policies and programs are most promising to improve outcomes for children, youth, adults and families.

Founded in 2013, CRESP recognizes that poverty, educational achievement, and chronic disease prevention are intertwined in a complex social web that challenges communities and policymakers alike. CRESP’s mission, values, and scientific priorities seek to inform program and policy development across local, state, and federal levels. We work alongside program professionals, academic leaders, and students to foster engagement in high-quality, practice-driven research and evaluation.

CRESP researchers are trained in evaluation methodology, randomized field experiments, natural experiments, qualitative methods, statistical analysis, mixedmethod evaluation and survey research.

Please feel free to contact us should you have any questions about us or our research.

Center for Research in Education and Social Policy University of Delaware Pearson Hall, Suite 107 125 Academy Street Newark, DE 19716 cresp-info@udel.edu (302) 831-2928 cresp.udel.edu Twitter: @udcresp CRESP Leadership Team Henry May, Director (hmay@udel.edu) Allison Karpyn, Co-Director (karpyn@udel.edu) Sue Giancola, Senior Associate Director (giancola@udel.edu) Jeff Klein, Associate Director (kleinjef@udel.edu) Suggested Citation Karpyn A., Kennedy, N. & Young C. (2022). 4.0 Fund III Evaluation: 2022. Newark, DE: Center for Research in Education and Social Policy.

2

Contents EXECUTIVE SUMMARY............................................................................................................ 5 INTRODUCTION ....................................................................................................................... 12 DESIGN ...................................................................................................................................... 13 EVALUATION QUESTIONS EXAMINED IN THIS REPORT ......................................... 14 DATA COLLECTION METHODS & TOOLS ......................................................................... 15 ALUMNI SURVEY ................................................................................................................. 15 ALUMNI INTERVIEWS ......................................................................................................... 17 EARLY CHILDHOOD EDUCATION SPECIFIC PORTFOLIO ANALYSIS AND ALUMNI INTERVIEWS ......................................................................................................... 18 GRADUATION SURVEY...................................................................................................... 19 FINDINGS .................................................................................................................................. 20 FOCUS AREA 1: TALENT DEVELOPMENT .................................................................... 20 How often do 4.0 alumni continue applying 4.0 mindsets/skills in their professional lives? ................................................................................................................................... 20 In what ways do fellows apply what they learned post-program? .............................. 29 To what extent do fellows become leaders in the education field? ............................ 39 FOCUS AREA 2: IDEA DEVELOPMENT .......................................................................... 41 What early-stage ideas have been supported by 4.0?................................................. 41 How have fellow ventures contributed to education innovations?.............................. 45 What can be learned from ideas that fail? ..................................................................... 47 How have early-stage ideas grown and pivoted? ......................................................... 50 FOCUS AREA 3: ECOSYSTEM DEVELOPMENT .......................................................... 53 How does 4.0 foster networks and collaboration within the alumni network? .......... 53 FOCUS AREA 4: RESEARCH DEVELOPMENT ............................................................. 53 What are 4.0’s key focus areas and how do they align with best practices in education? .......................................................................................................................... 54 What are 4.0s strengths and needs around applied data collection and evaluation skill building, for both ventures it supports and as an agency? .................................. 57 MONITORING OUR RPP..................................................................................................... 57 What lessons learned regarding small-scale pilot education innovations can be shared with the broader research and education communities? ................................ 60 FINAL THOUGHTS ............................................................................................................... 61 3

RECOMMENDATIONS ........................................................................................................ 65 REFERENCES ...................................................................................................................... 65

4

EXECUTIVE SUMMARY As an organization, 4POINT0 Schools (4.0) believes in supporting community-centered education innovation, rooted in equity. Education Innovators, called fellows, receive coaching, curriculum, community and monetary support for their ventures. Over the past ten years, more than a million students have been impacted by 1,200 leaders.

In May of 2020, a partnership was established between 4.0 and the University of Delaware’s Center for Research in Education and Social Policy (CRESP) in order to take a closer look at 4.0’s impact and to help build capacity to collect and use data to inform program operations. This report summarizes findings from a series of evaluation efforts to date, where evaluation questions are nested within 4.0’s four focus areas (Talent Development, Idea Development, Ecosystem Development, and Research Development). Below is the 2021-2022 evaluation question framework.1 Evaluation Questions Focus area 1: Talent Development – Increased leadership capacity within communities ● How often do 4.0 alumni continue applying 4.0 mindsets/skills in their professional lives? (T1) ● In what ways do fellows apply what they learned post-program? (T2) ● To what extent do fellows become leaders in the education field? (T3) Focus area 2: Idea Development – Increased identification of promising educational innovations in key focus areas through 4.0 funding and support ● What early-stage ideas have been supported by 4.0? (I1) ● How have fellow ventures contributed to education innovations? (I2) ● What can be learned from ideas that fail? (I3) ● How have early-stage ideas grown and pivoted? (I4) Focus area 3: Ecosystem Development – Increased development of promising ventures in key focus areas through facilitation of fellow-funder partnerships and alumni networks ● How does 4.0 foster networks and collaboration within the alumni network? (E1) ● How does 4.0 facilitate connections of funders to promising ventures? (E2)

1

Evaluation questions noted in italics are largely planned for evaluation in 2022-2023.

5

● How is 4.0 engaged with the broader community of education funders? (E3) Focus area 4: Research Development – Increased sharing of lessons learned and best practices for entrepreneurship in education ● What are 4.0’s key focus areas and how do they align with best practices in education? (R1) ● What are 4.0’s strengths and needs around applied data collection and evaluation skill building, for both ventures it supports and as an agency? (R2) ● What lessons were learned regarding small-scale pilot education innovations that can be shared with the broader research and education communities? (R3) Findings The Alumni Survey, administered in 2021 and 2022, was used to address the first Talent Development evaluation question, how often do 4.0 alumni continue applying 4.0 mindsets/skills in their professional lives? (T1). Based on responses from 574 participants across the two years: •

An average of 94.2% of 4.0 alumni consistently apply the mindsets, skills, and values that they learned at 4.0 in their professional lives, regardless of venture status (see Table 2).

In order to answer T2, in what ways do fellows apply what they learned post-program? the evaluation team examined alumni interview data to learn about the specific ways that fellows applied what they learned. Interviews were conducted with a subset of 31 alumni in the spring of 2021, in which alumni highlighted the value of questioning assumptions and being open to feedback from peers, as well as coaches.

•

Alumni apply what they have learned regarding the ways that 4.0 values community input and testing assumptions.

•

Openness and peer feedback are highly valued skills learned as part of 4.0, and frequently applied.

•

Alumni appreciate and apply the “questioning of assumptions” mindset they learn as part of the 4.0 process later in the development of their ventures and leadership.

6

o Allowing ventures to pivot and accepting that change is good are elements of the 4.0 program that alumni highly value. One alumnus noted, “you can’t be so attached to your idea that you’re not willing to let it morph and change.”

In relation to T3, to what extent do fellows become leaders in the education field?

•

The alumni survey revealed that the overwhelming majority of respondents (83.3%) are considered leaders in this field.

•

Many report that 4.0 was instrumental in their development as leaders (see Table 6).

Based on these findings, it was determined that 4.0 is successful at building increased leadership capacity within communities.

The first evaluation question for focus area 2, Idea Development, aims to understand what type of early-stage ideas have been supported by 4.0. Findings from the Graduation Survey revealed that 4.0 supports a variety of ventures, with the most common type being a program or service.

•

Specifically, the program or service categorization made up nearly one-half of ventures between the years of 2021 and 2022 (average of 48.31%).

•

Furthermore, results show that 4.0 continued to have an impact during the pandemic, with nearly 6,000 participants taking part in pilot and pop-up project tests between 2019 and 2021.

•

Additionally, it is known from the Alumni Survey (2022) that 87.7% of ventures are continuing in some way and more than of half of them have received additional funding from a source other than 4.0, indicating that fellow’s earlystage ideas are experiencing growth after their participation in 4.0 programming.

I2 focuses on education innovations and asks How have fellow ventures contributed to education innovations?

•

A review of the portfolio including 4.0 venture data and data from the fellows show that 4.0 ventures include a broad range of focus areas including those that are addressing innovation in increased academic and socio-emotional learning 7

(SEL), improved education, improved quality of life, and improved communitybased, culturally relevant educational programming.

I3 asks, what can be learned from ideas that fail? This is closely related to the final evaluation question for the idea development focus area, I4, which explores a topic known as pivoting.

•

Only a relatively small proportion of ventures (12.3%) are no longer continuing.

•

Interviews with alumni (2021) teach us that ideas fail for a variety of reasons including: o funding cuts, o lack of technical expertise, and o life circumstances.

•

Additionally, interviews suggested the ways in which 4.0 can support alumni through transition periods, including: o creating opportunities to extend the role of coaches to ongoing advisement or, o establishing thought partners with coaches and other alumni. o During one interview an alumnus said, “we learned to pivot and we pivoted constantly,” which highlights the importance of being flexible and cultivating dynamic ventures.

While much of the evaluation for Focus Area 3: Ecosystem Development is planned for 2022-2023, the 4.0-CRESP team did begin investigation into E1, how does 4.0 foster networks and collaboration within the alumni network?

•

The Alumni Survey asks respondents, “How often do you keep in touch with other fellows you met or collaborated with through 4.0? And Have you participated in any of the following 4.0 Communities?” Results show that over the past year, participation in events was limited. o While 75% reported participating in the alumni network, about one-third of those participated annually or a few times a year (“rarely”).

8

o Participation in communities was a bit more limited, where 40% of alumni in 2022 reported some level of participation while 60% did not participate at all.

Evaluation efforts related to Focus Area 4: Research Development will continue through the next evaluation cycle. Findings to date have revealed that, in response to R1, what are 4.0’s key focus areas and how do they align with best practices in education? two key practices were found to be foundations for the 4.0 approach.

•

The first is design thinking, a creative process for solving problems that keeps humans at the center.

•

The second is the science of pivoting, which is a plan designed to test a new hypothesis in the face of failure, real or potential.

Several significant activities have been undertaken to begin evaluation in relation to R2, what are 4.0’s strengths and needs around applied data collection and evaluation skill building, for both ventures it supports and as an agency? These include monitoring our RPP and the development of a logic model in concert with the program teams which reflects both evaluation objectives as well as current activities undertaken as part of fellow-programming.

•

Based on the Quick Check administered early this year, the overwhelming majority of those involved in the 4.0-CRESP RPP feel positively about their experience in the partnership. Specifically, 100% of respondents reported that yes, their opinions were taken into account “to a great extent” in the partnership.

Recommendations Based on these findings, the researchers from CRESP propose several recommendations for consideration by 4.0: 1. Consider an in-person alumni meeting for those who completed the program during COVID. a. Alumni from this time would like to have some in-person experience. A gathering, follow-up refresher or similar attempt to invite them to network and celebrate their accomplishments would be welcome.

9

2. Assist alumni in connecting to 4.0 in new ways and through existing channels they might not be aware of. a. If not done already, consider informing graduates before they leave the program of the alumni opportunities available to them, and sign them up at that time. Additionally, other opportunities to increase outreach to alumni communities should be explored, as alumni are mixed in terms of their familiarity with what communities exist and how they can get involved with them. b. Create additional supports for alumni around funding, which is an important reason that they would like to stay connected. Alumni would like ongoing opportunities to learn about where to find resources, how to apply for funds, and how to write grants and structure proposals.

3. Explore the ways in which 4.0 training could tailor efforts to best support entrepreneurs with similar interest areas, perhaps offering content expertise and evidence-based ideas, as well as examples of effective strategies to foster fellows’ understanding of current best practices. a. Additional needs described during the application stage may also be a framework for common instruction, such as app development and technology infrastructures. i. A venture self-assessment completed by fellows periodically throughout 4.0 participation, as well as the opportunity for alumni to pick workshops that are themed around where their idea is now, including topics such as pivoting may be of use. This could particularly support those that are re-designing ventures/pivoting. b. Furthermore, teambuilding during and after the fellowship for common venture ideas could be expanded upon. c. Finally, consider how fellows are educated about the variety of activities, theory and content options used in similar programs. For example, creating a mix of activities that include opportunities for physical activity and de-stressing, especially if programming extends the school day for youth.

10

4. Ascertain fellows’ incoming knowledge and utilization of mindsets and skills such as joy, anti-racism, equity and determination. This is critical so that programming is not redundant with their existing knowledge. Rather, programming should expand upon the areas where there are gaps or dive deeper into areas where they already have background knowledge.

5. Continue developing evaluation tools and measures that are as targeted and efficient as possible. a. Consider a process that occurs annually to review survey tools from the prior year. As part of this strategy, reflect on the alignment of tools with program components and priorities. Maintain consistency whenever possible and update the tools as needed for the subsequent year, eliminating questions when they are not needed. b. Furthermore, a strategic review of the graduation survey in concert with the alumni survey and data collected on fellow applications may be of use.

Questions such as those that focus on participant records of their assumptions, for example, may be lengthy for respondents to complete and difficult to use in reporting.

The full report provides a detailed accounting of evaluation findings and recommendations. Researchers from CRESP are available to answer questions regarding analyses presented in this report or to assist in their interpretation. For more information, please contact Allison Karpyn at karpyn@udel.edu.

11

INTRODUCTION The University of Delaware’s Center for Research in Education and Social Policy (CRESP) conducts rigorous research, program evaluation, and policy analysis to inform decisions at the local, state, and federal levels, thereby improving outcomes for children, youth, adults, and families. CRESP is now in its third year of serving as a partner with 4POINT0 Schools (4.0), the largest and earliest investor in the education field today. Since May of 2020, CRESP has worked with 4.0 to provide feedback on a variety of outcomes in two annual reports and five thematic reports (see Figure 1), as well as an alumni survey metric review presentation and revised tool.

Figure 1: The Five Thematic Reports2 ● 4.0 Schools: Early Childhood Education Portfolio, Case Studies, and Interview Results ● Strategic Insights from 4.0 Alumni: The Importance of Pivoting ● An Equitable Future in Education: Learning from 4.0 Alumni Experiences ● 4.0 New Normal Wave: Evaluation Findings ● Ten Current Trends in Early Childhood Education: Literature Review and Resources for Practitioners

Our Research Practice Partnership (RPP) has emphasized the critical nature of internal systems and processes, as well as alignment with measures, goals and objectives.

CRESP and 4.0 have worked in close partnership throughout the conceptualization, design, implementation, and analysis phases of a variety of evaluation efforts undertaken to date. Together, the two partners have come to understand the challenges, benefits, and realities of evaluation. As was true from the start, the evaluation procedures employed by the team are aligned with the standards established

2

Published in 2021 and accessible at https://www.cresp.udel.edu.

12

by the Joint Committee on Standards for Education Evaluation and the American Evaluation Association’s Guiding Principles for Evaluators.3 DESIGN Our work is based on the RPP model established during the first year. The intent of the RPP was to create a true partnership to support 4.0 in reaching its goal of shifting the culture of innovation in education, through collaborative and rigorous research. The RPP included: (1) a professional team of evaluators including and led by Drs. Allison Karpyn and Sue Giancola; (2) representatives from the 4.0 Evaluation and Learning and Programs Teams; and (3) other key stakeholders as determined jointly by CRESP and 4.0.

The outcomes of the RPP are twofold: generate knowledge in 4.0’s four key focus areas and increase capacity within 4.0 to continue utilizing the products and strategies developed by the partnership. The framework for the CRESP-4.0 partnership is based on the five dimensions of effective RPPs. See Figure 2 for a description of each dimension.

Figure 2: Dimensions of Effective RPPs Dimension

Description

D1

building trust and establishing partnership relationships

D2

conducting rigorous and relevant research to use in decisionmaking

D3

supporting the practice partner in reaching its goals

D4

producing knowledge for broader impacts

3

Yarbrough, D.B., Shula, L.M., Hopson, R.K., & Caruthers, F.A. (2010).The program evaluation standards: A guide for evaluators and evaluation users (3rd. Ed). Thousand Oaks, CA: SAGE Publications.

13

D5

increasing capacity within the partnership

True to the RPP with 4.0, Year 1 of the 4.0-CRESP RPP established joint goals and evaluation plans and produced work products to serve as a strong foundation for documenting and sharing 4.0’s innovative contributions and impacts through the support of education and entrepreneurial ventures. Year 2 was a time to broaden stakeholder involvement in the evaluation work, while also working to focus the evaluation at every stage, from understanding the role of networks and training to capturing impacts on 4.0 program participants.

EVALUATION QUESTIONS EXAMINED IN THIS REPORT The present report is organized around the Evaluation Questions Outlined in the initial proposal. These evaluation questions focus on supporting measurement and tracking infrastructure building, as well as examining program effectiveness and impact at the participant level. A summary of these questions is below.

Of note, the evaluation questions served as a basis for the three-year effort and as such, not all were scheduled to be addressed in the current year. Evaluation questions noted in italics below are largely planned for 2022-2023.

Focus area 1: Talent Development – Increased leadership capacity within communities ● How often do 4.0 alumni continue applying 4.0 mindsets/skills in their professional lives? ● In what ways do fellows apply what they learned post-program? ● To what extent do fellows become leaders in the education field?

Focus area 2: Idea Development – Increased identification of promising educational innovations in key focus areas through 4.0 funding and support ● What early-stage ideas have been supported by 4.0? ● How have fellow ventures contributed to education innovations? ● What can be learned from ideas that fail? ● How have early-stage ideas grown and pivoted?

14

Focus area 3: Ecosystem Development – Increased development of promising ventures in key focus areas through facilitation of fellow-funder partnerships and alumni networks ● How does 4.0 foster networks and collaboration within the alumni network? ● How does 4.0 facilitate connections of funders to promising ventures? ● How is 4.0 engaged with the broader community of education funders?

Focus area 4: Research Development – Increased sharing of lessons learned and best practices for entrepreneurship in education ● What are 4.0’s key focus areas and how do they align with best practices in education? ● What are 4.0’s strengths and needs around applied data collection and evaluation skill building, for both ventures it supports and as an agency? ● What lessons were learned regarding small-scale pilot education innovations that can be shared with the broader research and education communities?

The next section of this report will describe the methods used to collect data in order to answer these questions. In it we will provide information about the length of these tools, the number of participants, and when they were administered. Findings are reported in the second half of the report, where we structure results according to evaluation questions.

DATA COLLECTION METHODS & TOOLS Four distinct approaches to collecting data form the basis for the present report. These include the 2021 and 2022 Alumni Survey data, Alumni Interviews conducted by UD, 2019/20 and 2021 Graduation Survey data and Early Childhood Education (ECE) Case Studies also conducted by UD. ALUMNI SURVEY An alumni survey is administered annually. The 2021 4.0 Alumni Survey was distributed to all 4.0 alumni over ten weeks from October 2020 to January 2021 and a total of 285 responses were collected. Response rates were similar in 2022 with 289 participants responding to the survey which was administered in February and early March.

15

Between the years of 2021 and 2022, the survey was revised through an iterative process conducted with 4.0 to reduce the overall survey length while maintaining key questions including those of key interest to funders and program staff. Of note, the survey included a total of 55 questions in 2021 and 34 questions in 2022. The survey is divided into four parts: (1) About You, (2) Social Network Analysis and Mindsets, (3) Updates on Your Venture (if applicable), and (4) Future Engagement with 4.0. A crosswalk for the Alumni Survey, which describes the specific questions that were asked in 2021 and 2022, was developed by the team and is available here as a reference.

Alumni Survey Participation Responding participant characteristics for 2021 and 2022 are provided below (Tables 1), including racial and ethnic identity, identification as a person of color, gender identity, region of focus, and content area of focus. Specifically, the most prevalent racial & ethnic background of participants in the Alumni Survey in both 2021 and 2022 was Black/African American. Approximately 56% of participants in 2021 and 58% of participants in 2022 identified as a person of color. Additionally, approximately 61% and 71% of respondents were female, in 2021 and 2022 respectively (see Figure 6).

Table 1: Participant Characteristics (Alumni Survey) Year

2021

2022

Racial & ethnic background Black/African American

37.87% 42.0%

White

34.02% 35.0%

Mixed-Race/Multi-Racial

5.92%

7.5%

Asian American/AAPI

7.40%

6.2%

Latinx

8.28%

5.3%

Middle Eastern

1.18%

1.8%

16

Identity not listed4

1.78%

0.9%

Prefer Not to Answer

0.89%

1.3%

Total

97.34% 100%

Identify as a person of color Yes

56.10% 58.4%

No

42.07% 38.9%

Prefer Not to Answer

1.83%

2.7%

Total

100%

100%

Gender identity Woman

60.98% 70.8%

Man

35.55% 25.2%

Genderqueer / Genderfluid / Gender non-conforming

2.02%

2.2%

Non-binary

0.87%

0.9%

Prefer Not to Answer

0.58%

0.9%

Total

100%

100%

ALUMNI INTERVIEWS In 2021, a subset of the 285 respondents was invited to participate in an in-depth interview about their venture and their experience with 4.0. To do so, a random sample of 50 alumni survey participants received an email invitation to participate. Participants were eligible initially if they responded “Yes” to, “Did you participate in 4.0 with an idea/venture?” Thirty-eight alumni responded to our request, however of these, two were deemed ineligible because of conflicts of interest (ex. current employees of 4.0).

Interviews were conducted with 31 of 38 eligible alumni (82%) in the spring of 2021. Of the seven who did not participate: one declined to be interviewed, five did not respond

4

Note - “Identity not listed” was phrased as “other” in the 2021 Alumni Survey.

17

to three separate contact attempts; and the final person responded after the interviews ended.

Interviews were approximately 30 to 60 minutes. All interviews were led by trained CRESP evaluation team members and recorded. Alumni were assured of their confidentiality and encouraged to provide both positive and negative feedback.

Participants received a $25 Amazon gift card in acknowledgment of their participation.

The UD Institutional Review Board approved all human subjects-related aspects of the process, including the content and process for communication as well as the administration of the interview protocol.

The goal of the interview was to receive feedback about the 4.0 fellowship process from alumni whose ventures were continuing as well as those whose ventures did not continue. Interview questions addressed topics such as: ● venture background and goals, ● how 4.0 supported the venture, ● strengths and areas of improvement for 4.0, and, ● advice for 4.0.

While interviews did not include specific questions or prompts about equity, anti-racism, or diversity, these major themes emerged organically. A hybrid approach to analysis using both inductive and deductive methods was used, and all data were examined using Dedoose qualitative analysis software.

EARLY CHILDHOOD EDUCATION SPECIFIC PORTFOLIO ANALYSIS AND ALUMNI INTERVIEWS In order to better understand the nature of the early childhood investments, a “deep dive” was undertaken to both examine existing secondary data and then, to contact alumni to interview them about their early childhood work. The process began with the database of funded ventures kept by 4.0.

18

Initially, 602 ventures were identified in the 4.0 portfolio database5, and of these, 43 were listed under the “Organization: Theme” area as Early Childhood related ventures.

Each was examined in more depth through google searches, a review of application materials and phone calls and ultimately 36 were ultimately confirmed as ECE ventures and formed the basis of our secondary data analysis. To complement this work, indepth interviews were conducted with alumni leaders from a subset of six of these ventures. Venture titles included: ● Mathematician, Jr., ● Early Partners, ● Family Literacy Empowerment Program (FLEP), ● Training Grounds Inc., and ● Cognitive ToyBox. GRADUATION SURVEY What is commonly referred to as the graduation survey, consists of two separate tools administered at the same time. One tool is called the Graduation Application Survey and the other is called the Graduation Programming Feedback Survey.

The Graduation Application survey data included in this report reflects combined years 2019 and 2020 (n=174) as well as 2021 (n=122). The Grad Application Survey is administered following the completion of the pop-up/pilot phase of training and serves in part to develop a one-page description of the program that is provided to fellows for use with future program needs including recruiting and fundraising. The 27-page survey captures information about when, where and how the pilot was conducted, as well as successes. The types of data collected and outcomes from that effort are also captured in this tool.

The programming survey includes 2019/20 and 2021 data as specified. The programming survey included 20 pages of questions capturing data on how funds were

5

The following sources were reviewed: Program plan doc, 4.0 Early Childhood Portfolio, New3102019Master4.0, completed 2020 alumni survey and Spring 2018app, and internet searches (i.e. websites, LinkedIn pages etc.).

19

spent, feedback on how tools and resources provided by 4.0 to carry out the test worked, as well as feedback about the coaching, support and belonging aspects of the program and if these aspects were utilized by the participant. In 2019 and 2020 125 participants completed the survey and in 2021, 127 participant responses were recorded.

FINDINGS As noted earlier in this document, the report is structured in terms of 4.0 focus areas, and specifically, responds to the established evaluation questions identified in each section, for which data is available to date.

FOCUS AREA 1: TALENT DEVELOPMENT HOW OFTEN DO 4.0 ALUMNI CONTINUE APPLYING 4.0 MINDSETS/SKILLS IN THEIR PROFESSIONAL LIVES?

This evaluation question seeks to understand how alumni completing their fellowship, inclusive of those that participated in essentials, tiny, new normal, or pathways, persist in their use of mindsets (i.e. attitudes), skills (i.e. proficiencies), and values (i.e. beliefs).

Specifically, the Alumni Survey asks: ●

“Generally, do you apply mindsets, skills or values learned at 4.0 in your professional life?,” and,

● “Which mindsets / skills / values do you still use?” (listing them for ratings).

Data from the Alumni Survey (2022) indicate that 95.1% of total respondents generally apply the mindsets that they learned at 4.0 in their professional lives, which constitutes a small increase from the previous year. This figure is similar to the responses in (2021) where 93.3% agreed that they generally apply the mindsets that they learned at 4.0 in their professional lives (see Table 2 ).

20

Table 2: Generally, do you apply mindsets, skills or values learned at 4.0 in your professional life? (Alumni Survey, 2021 & 2022) Year

2021

2022

Yes

94.80% 95.1%

No

5.20%

4.9% 100.0

Total

100%

%

Table 3 provides information from the 2021 and 2022 alumni survey which report the frequency of use of specific mindsets and skills. These include: identifying assumptions you might be making, sharing an idea with people who have not heard about it before, falling and learning from the experience, breaking down a problem and testing it in multiple ways, trying again after failing, and evaluating the strength of my connection with a community I want to work with. The 2022 Survey also included questions about: joy, anti-racism, equity and determination. In 2022 the most frequently used mindset or skill was determination, followed by trying again after failing. All of the ten mindsets for the year 2022 were applied by 70% or more of respondents “always” or “very often.” (see Table 3)

Table 3: Which mindsets / skills / values do you still use? (Alumni Survey, 2021 & 2022)6 Year

2021

2022

Always

31.60%

31.6%

Very often

48.80%

52.0%

Identifying assumptions you might be making

6

The question asked in 2021 about mindsets was different. It asked “How often do you currently apply this mindset/skill that you learned at 4.0 in your professional life?”

21

Sometimes

19.60%

15.1%

Rarely

0.00%

1.3% 100.0

Total

100%

%

Always

27.14%

34.7%

Very often

52.38%

48.4%

Sometimes

17.62%

12.4%

Rarely

2.38%

4.0%

Never

0.48%

0.4%

Sharing an idea with people who have not heard about it before

100.0 Total

100%

%

Always

46.11%

41.8%

Very often

37.72%

44.4%

Sometimes

15.57%

12.4%

Rarely

0.60%

1.3%

Failing and learning from the experience

100.0 Total

100%

%

Always

27.51%

31.1%

Very often

50.79%

39.1%

Breaking down a problem and testing it in multiple ways

22

Sometimes

20.11%

27.1%

Rarely

1.59%

2.7%

Total

100%

100.0 %

Trying again after failing Always

49.64%

47.1%

Very often

37.23%

41.3%

Sometimes

13.14%

10.2%

Rarely

0.00%

1.3% 100.0

Total

100%

%

Always

30.56%

32.9%

Very often

45.83%

43.1%

Sometimes

20.83%

20.0%

Rarely

2.08%

2.7%

Never

0.69%

1.3%

Evaluating the strength of my connection with a community I want to work with

100.0 Total

99.99%

%

n/a

44.4%

Joy Always 23

Very often

n/a

35.6%

Sometimes

n/a

16.0%

Rarely

n/a

2.7%

Never

n/a

1.3% 100.0

Total

n/a

%

Always

n/a

52.0%

Very often

n/a

32.9%

Sometimes

n/a

12.9%

Rarely

n/a

0.4%

Never

n/a

1.8%

Anti-racism

100.0 Total

n/a

%

Always

n/a

59.1%

Very often

n/a

33.3%

Sometimes

n/a

6.2%

Never

n/a

1.3%

Equity

100.0 Total

n/a

Determination 24

%

Always

n/a

55.6%

Very often

n/a

33.3%

Sometimes

n/a

9.3%

Rarely

n/a

0.4%

Never

n/a

1.3% 100.0

Total

n/a

%

A Deeper Dive: Understanding Active Ventures vs. Non-Active Venture Mindset Differences In 2021, mindset results were compared for 4.0 alumni respondents with active ventures, inactive ventures, and those who participated in 4.0 before ventures were required, in order to better understand differences between these groups. Results in Tables 4 and 5 show the percentage of alumni that responded “always” or “very often” to each item. Tables 4 and 5 also show p-values from chi-square statistics, which are measures of association used to explore statistical significance across categorical data.

These comparisons explore whether alumni with active ventures are more likely to currently apply 4.0 mindsets and skills in their professional lives compared to alumni with inactive ventures and alumni who participated before ventures were required. For the purpose of these analyses, a p-value less than 0.05 is considered statistically significant.

All 4.0 alumni groups reported very high levels of “generally applying mindsets and skills learned at 4.0 in their professional lives.” Among alumni that had an active venture, 94.9% indicated generally applying mindsets and skills learned at 4.0 in their professional lives. Of those that no longer had an active venture, 94.6% reported generally applying mindsets and skills learned at 4.0 in their professional lives. For those from before ventures were required results were still high: 86.5% (Table 4).

25

Table 4: 4.0 Alumni Characteristics And Mindsets By Group (Alumni Survey, 2021) Characteristic

GROUP

Total

p-value

(n=285) Active

Inactive

Ventures “failed” (n=177) Ventures (n=56)

Before Ventures Were Required (n=52)

Reported generally applying

94.9%

94.6%

86.5%

93.3%

0.170

mindsets and skills learned at 4.0 in their professional lives

Table 5 shows how specific 4.0 mindsets and skills are currently applied by alumni in their professional lives. Results are shown for all alumni that completed the survey (“Total” column), and by group (active venture, inactive venture, or alumni who participated before ventures were required). The percentages of alumni that reported “always” or “very often” applying each mindset are shown. Mindsets in Table 5 are sorted from highest to lowest reported use across all groups in the “Total” column. Pvalues less than 0.05(*) indicate that comparisons between alumni groups are statistically significant. For example, for the item, “Sharing an idea with people who have not heard about it before” data indicate that those with active ventures are more likely to respond affirmatively than those with inactive ventures.

Table 5: How Often Do 4.0 Alumni Continue Applying 4.0 Mindsets/Skills In Their Professional Lives?* (Alumni Survey, 2021) 4.0 Mindsets/Skills

GROUP

26

Total

p-value

Active

Inactive

Before

Ventures

“failed”

Ventures

(n=177)

Ventures

Were

(n=56)

Required

(n=285)

(n=52) Identifying assumptions you

59.9%

57.1%

40.4%

55.8%

0.003*

51.4%

35.7%

34.6%

45.3%

0.025*

44.1%

44.6%

44.2%

44.2%

0.672

44.6%

41.1%

32.7%

41.8%

0.237

50.3%

32.1%

19.2%

41.1%

0.000*

48.0%

35.7%

21.2%

40.7%

0.001*

35.6%

39.3%

44.2%

37.9%

0.028*

might be making Sharing an idea with people who have not heard about it before Asking people experiencing a problem for their feedback Failing and learning from the experience Breaking down a problem and testing it in multiple ways Understanding how tiny tests apply to the bigger picture Understanding of design thinking

27

Applying design thinking to

38.4%

28.6%

40.4%

36.8%

0.168

Trying again after failing

38.4%

30.4%

17.3%

33.0%

0.135

Evaluating the strength of

36.7%

26.8%

11.5%

30.2%

.005*

20.3%

26.8%

15.4%

20.7%

0.782

15.8%

25.0%

15.4%

17.5%

0.593

a problem

my connection with a community I want to work with Determining whether I am the best person to lead an idea Evaluating whether I have enough experience to create or develop an idea *Shows % that responded “Always” or “Very Often” for each mindset

Table 5 shows that, overall, fellows whose ventures are still continuing report the highest levels of applying 4.0 mindsets/skills in their professional lives. Alumni with active ventures are significantly more likely to report always or very often applying five of the 12 mindsets examined: Identifying assumptions you might be making; Sharing an idea with people who have not heard about it before; Breaking down a problem and testing it in multiple ways; Understanding how tiny tests apply to the bigger picture; and Evaluating the strength of my connection with a community I want to work with. One factor that is applied significantly more by alumni who participated before ventures were required (before 2016) is: Understanding of design thinking. This reflects changes in curriculum as design thinking was emphasized in the 4.0 fellowship program prior to the venture development approach. Collectively these data demonstrate that alumni are 28

more likely to report applying mindsets and skills that were integral to the 4.0 curriculum they receive.

Alumni with active ventures are more likely to report always or very often applying 4.0 mindsets and skills in their professional lives. Differences in applying 4.0 mindsets and skills among distinct alumni groups reflect shifts in focus areas of the 4.0 curriculum and programmatic emphasis over time. The mindsets that are applied more actively and continually after 4.0 by the active venture group were emphasized in the 4.0 venture development curriculum.

IN WHAT WAYS DO FELLOWS APPLY WHAT THEY LEARNED POST-PROGRAM? Findings from the Alumni Survey suggest that alumni have continued to apply the 4.0 mindsets and skills years after participating in fellowships. In order to better understand the ways in which fellows applied those skills and how the mindsets continue to support their work, the evaluation team examined alumni interview data to learn about the specific ways that fellows applied what they learned.

Ten key thematic findings emerged from 31 alumni interviews. Outlined below are four of those key thematic findings, along with sample supporting quotes from interviews.7 Each of these themes addresses the second talent development evaluation question in its own way.

Finding 1: Alumni found value in 4.0’s commitments to deep understanding, questioning of assumptions, and putting community perspectives first and foremost when examining problems.

A fundamental component of the 4.0 process is the examination of assumptions and personal biases regarding proposed solutions and the communities that fellows aim to

7

For more information on the ten key thematic findings, please see Strategic Insights from 4.0 Alumni: The Importance of Pivoting created by CRESP for 4.0 in April of 2021.

29

serve. Alumni recognize the value of this commitment as they identify the goals of their initiative and move forward as a people-centered venture: “Initially our assumption was housing is a great area to do more. And so we started working with 4.0 to really gain clarity about what it is that we're doing. I thought we went in having clarity and I thought that they were really going to help us with like the lift. Hey, this is what we're doing, and they are going to help us get there. But really what 4.0 did was they said, ‘Okay. This is what you're doing, or this is what you think your solution is,’ and they would always say, ‘Break up with the solution and fall in love with the problem.’ And so we would always go back, back, back, back. And we got to that point where it was like, ‘Hey, I know that you really think that housing is the crux of this, but your goal is to help students be successful in college, to get students to and through community college. And it takes more than housing to do that.’ And so we just started from a more basic level of like, what kind of infrastructure do we need to really get this moving. And 4.0, I think, was really instrumental in just helping us think about our goals and clarify what it is we're trying to do. And having that clarity gave us the ability to move forward, in really smart ways instead of just jumping into one solution.”

These questions of assumptions and pivoting applied at the venture level as well as the individual fellow level: “Because 4.0's help looking at the foundation, the true purpose of the work, they helped me see that we shouldn't just jump forward to think about how to be a successful, robust, sustainable organization, and we should step back and say like, ‘What are actually the true needs of this community that you're trying to serve and how you do that the right way?’ And so that's taking us longer to figure that out, but I think it's a good thing to do it right instead of do it fast... Throughout that process, I also pictured myself running this organization. And so it was hard for me to envision a future for the organization that didn't include that, but I also realized like, hey, well, because of what the community actually needs, I also realized I don't really want to be the CEO. I don't want to be the person doing that work. I think I could do it but I just don't really want to… I haven't figured out why 30

I don't really want to but I don't think that's really important. I think what's important is just that we took the time to figure out what the community needs and now we're taking the time to figure out who would be the best person to lead this organization and take my ego out of it.”

Similarly, another interviewee noted: “… it's very centered on people. Who are the people you're trying to serve? What is the community served? What do they need? How are you figuring this out? Are you asking them? Are you just kind of coming to the table with your assumptions and not...Once we started talking about those assumptions, we start to see how tied into the identity work that is. I think it was very smartly structured in that sense. If it's going to be people-centered, you've got to start with what you're coming to the table with as a person, and then you've got to focus and anchor the idea, ideating around people.”

This finding is consistent with the alumni survey result that “identifying assumptions you might be making” was the most frequently reported 4.0 mindset/skill that alumni currently apply in their professional lives (55.8% reported “Always” or “Very Often” applying this mindset). Those with still active ventures had the significantly highest reporting of applying this mindset/skill (59.9%), followed by those with inactive ventures (57.1%), and then 4.0 alumni who participated before a venture idea was required (40.4%) (p-value = 0.003).

Finding 2: Alumni highly valued 4.0 training approaches that emphasized openness, quality constructive feedback from peers, accessibility for varied backgrounds, and a sustained social entrepreneur mindset.

Alumni discussed how a major strength of the 4.0 process was gathering a community of peers to challenge one other in a constructive and supportive way. They described specific approaches they found valuable and unique to the 4.0 experience and how these helped develop their venture ideas: 31

“It doesn't even have to be a formal plan. They really just want you to start the idea, and then do the interviews. You submit the interviews, and then you get to [the workshop] with everybody else. Then that's when you're actually writing it out, and working... Everybody can take your idea, you do bites of your idea, little snippets, and people just can put, they put post-its about assumptions. Then you go home, you read through all the assumptions, you come back the next day, fine tune your value proposition, and it goes through another iteration of assumptions… It really is a thinking process that I think is really important, because before you spend a lot of time and money doing something that you think is great, you have a lot of feedback to really help you think about, ‘Do I need to do this now? Should I maybe do something else? Or do I need to just fine tune and wait a little while?’... Somewhere, I still have the post-its they did, because they were really thoughtful questions. I think it's also important to have the input from people who are not necessarily in the same space, because if they can see value in your idea, then you really have a better chance of it working.”

Alumni discussed the importance of openness to feedback cycles and how it fostered lasting personal and professional growth: “What advice would I give to someone who was thinking about working with 4.0?... being as open minded as possible, to not being so attached to your idea when you go into it…. There's this phrase in the art world, ‘kill your darlings’ or something like that, where you can't be so attached to your idea that you're not willing to let it morph and change, and let other people go through it and hash it out. I felt like there were just so many points where someone said something that I was like ‘I don't want to hear that.’ Like ‘I don't want that to be true’, but it was true… I think that going in with the idea that this is a learning journey for you as a person, and not going in thinking that ‘Oh, I'm in this program because I'm here to execute a pilot.’ It's like that's it, but that's barely it. It feels like it's the tip of the iceberg… Yeah, you're going to do a series of tests and whatever, but there's also all of this mass of learning that will shape you as a person and your work that if you're not open to that then you're missing the point of all that goes into it… I think just being open to, not being too attached to your ideas, and being 32

open to changing not only your ideas but your core beliefs as a person, I think that would be my biggest advice.”

The 4.0 approach breaks down big ideas into small, feasible steps, even for educators who have no background as entrepreneurs. This alumnus also described how the 4.0 culture encourages critique in a way that inspires confidence and helps to “build folks” rather than bring them down: “[4.0] strengths are accessibility… having completely no knowledge around entrepreneurship or very skewed views of it, everything seemed so accessible. And what I mean by accessible, I mean they use language that we could understand. It was not a lot of jargon. They were very accepting. The critical minds, they used their critique to build folks, not to bring them down… [I]f anything, that created some sort of confidence around the project in and of itself.

And it actually has inspired me personally to really start being more serious about my ideas and to take the proper steps to flesh out and get data… which I would never have done in any other context.”

Another alumnus described the shift in mindsets fostered by participation in 4.0 and how those mindsets continue to be applied as a problem-solving “roadmap” years later: “Professionally I think... the biggest thing was [4.0] transitioned my mindset from an employee to an entrepreneur. That was really the first base where I really started to think about myself as an entrepreneur. I started several ventures since then. And it's just because I feel like, now, when I see a problem, I feel like I have a roadmap for how I can start to solve it through different business ideas. That really opened up just some mental doorways for me professionally in terms of how do I think about things from a business perspective instead of a program perspective. That's probably been their biggest influence on me professionally.”

Finding 3: Coaching is a critical part of the 4.0 process, and when utilized to its full potential, it becomes a key asset for fellows to launch, adjust, and meaningfully transition their ventures.

33

Some ventures pivoted as part of the 4.0 process and coaching was critical to help fellows launch projects, adjust, and transition. Coaches are a key asset in helping each venture find its path and in supporting ventures to apply the mindset of approaching a problem in multiple ways: “The venture started off really broad… We didn't end up doing what we planned to do, but we just ended up doing what we could do and created some lesson plans. We work with teacher professional development, so we ultimately ended up creating some activities that we [still] use in our workshops. It was very helpful, but it was not what we planned to do, which I think is par for the course…

So we learned the meaning of the word pivot… we learned to pivot and we pivoted constantly… It probably wasn't accidental… I had a very good coach.

That was very helpful, I think we had, it was either weekly or maybe it was every other week coaching. I found that valuable.”

However, there is a critical balance between supporting projects to pivot/pause and to keeping them accountable. The value of the coach-fellow relationship was highlighted by other alumni that described lost opportunities in the coach-fellow relationship. Alumni suggested that coaches should actively facilitate the process of pivoting and “pushing” or challenging fellows. For example: “We had coaching. We had a coach who we really loved personally and could get a drink with, and all that, really, really great. The type of coaching that it was, though, if anything, it played into a lot of our insecurities. What I mean by that is, we saw a lot of failures. And they weren't really trying to keep that motivation, to keep pushing. It was positive, but it was kind of like, ‘Okay, that's just where it is.

All right. We'll just leave it.’ And it seemed kind of complacent… I think a stronger, more rigorous coaching opportunity where, when meeting with our coach, there are a lot of expectations, more expectations or more deliverables, or anything like that, to kind of keep us going, and if anything, to inspire us to pivot or find new ways to really grow within that venture… The coach wasn't somebody who you meet with once. We met with our coach, for a while, monthly, then biweekly, and then weekly. And you really get to know that person, and you really care about them. And I think, not that this person wasn't a great person. I highly 34

admired them. I just think I needed someone to be like, ‘Let's pick ourselves up.

We got this. Let's take this idea and move it here.’ They don't have to give us the plan, the major plan, but maybe just give us a little push or nudge into that next step… [A]t the core of 4.0, I think they really cherish the autonomy of participants… And I don't think they want to set too much structure around everything, because then that might scare someone off, or it may stop a really great idea from blossoming into something new… I understand that 100%, and I know that's probably a really hard line to toe by any means. I just think for us, being so green to everything, I think we really needed to build up the grit or that resilience.”

The need for an expanded coaching role was echoed by another alumnus who ultimately never successfully launched their pilot: “I would credit them [4.0] with getting me to walk away from what I would call a training or workshop thinking about what I may need to do more to launch [the venture]. So I think that was positive, but ultimately I think with those questions leaning out there and coinciding with a time in my life that was a bit tumultuous, it got left on the shelf… I think there was a follow-up call and maybe a coaching call a month or so later. I think perhaps having one or two additional touch points with 4.0 about my venture specifically, and perhaps helping me explore the tension in those questions that I raised and the reality of the space, etc., I wonder if it would have maintained a motivation for launch, because I came into the workshop or the fellowship Essentials geared up for, I think, a fall launch that year or whatever. I left feeling like I needed to do more work to launch. And so I think the gap between that and launch could have been just maybe one or two additional really substantive and robust touch points with them potentially would have kept me moving toward a launch, even if it was just a small pilot.”

A fellow whose venture is still continuing discussed how highly effective and impactful their coach was in helping them transition their venture into a nonprofit organization.

The match process of this fellow with a coach who had relevant firsthand experience was critical to achieving this success: 35

“And I think that was possibly the most beneficial experience, just working so closely with someone who had done that before and started their own nonprofit.

And I think it was mainly through our conversations with [them] that we were like, ‘Okay, we can do this. We should do this, start our own thing.’”

Finding 4: The mindsets, values, and tools that fellows learned in 4.0 remain an active part of their personal and professional skillset even when a venture transitioned or closed.

Several alumni described how major life events (moving, new jobs, going back to school, pregnancy, illness) affected ventures ending. There is little that 4.0 can or should do to intervene on those events. However, alumni continue to carry the 4.0 venture idea with them and to apply mindsets learned in 4.0 in their continued education, new jobs, and other aspects of their lives: “[I]t was a crazy time for me anyways, because I was also starting a new job… I know I was moving and I was starting my PhD program also… There was a lot going on… Actually, it was so crazy that I didn't know if I was going to be able to hold it together… because I also have two kids… I was like, ‘Oh, this will be a great way for me to start organizing my thoughts around this and trying to maybe more concretely figure out what this could look like.’ And I'm not blaming the program or anything because it also could have very much been me with everything that I had going on… I don't know that in that moment I had the capacity in my brain to take on really putting all of the thought and energy into it that it needed… I felt like I failed and I just was never sure, even in terms of 4.0, I'm like, ‘Have I now been blackballed because I didn't finish it, did I burn this bridge?’ And not really ever knowing. I've continued to just have a lot going on. I mean, I'm still in school, I've been teaching full-time, I still have two kids. The one thing that I learned is making sure that I'm in the right place to be able to give it the time and attention that it would need…but it is something I think about a lot because it's still the direction that I want to go. I want to be coaching teachers and have an opportunity to push into schools and potentially have virtual professional development classes or whatever it may be, virtual and in person… I 36

think about it all the time, but I still can't say I've gotten to a point where I feel like, ‘Okay, I can make a go of this.’ It's also scary as a single mom. How do I stop working to really put in the time and attention to do this? When do I make this happen?”

Another alumnus shared that she still uses empathy interviews and clarity of purpose skills that she fine-tuned during her 4.0 involvement, even though her original venture ended in 2019: “I think one is the empathy interview skills. It wasn't a new concept, but I think having it reaffirmed and have the space to practice it was really helpful and something I still use. That's the one that stands out the most. But also I would say actually getting really clear about what you do and don't do has been really helpful as well. I remember that being kind of a through line throughout the series too. And saying no to things that aren't in alignment, like not stretching yourself too far outside of your venture. That was really helpful.”

Other interviewees shared additional ways that they continue to use resources and content from their 4.0 experience in their present context: “I have referred multiple times to their handout worksheet that was how to scale from pop up all the way to larger venture. And I have continued to use that in my work in philanthropy, helping people who apply to us start tiny and move into something bigger and really think about their program. But I pull in other tools from other places too.”

Another person explained: “…having stakeholders have a voice, having the people in the community help build the idea with us is something that I've done. I think prior to going there [to 4.0], it was just me and I really didn't even know how to approach people on the venture outside of like, ‘Hey, we got a school coming. You want to enroll your student?’ But not really knowing like, ‘Hey, we have a school coming. What would you like it to look like?’… It changed my thinking… it is the content and the self-exploration and then the equity piece because… I can use that in my 37

everyday life, my professional life, and entrepreneurial ventures that are outside of the education field. Engaging your community, engaging stakeholders, what questions to ask, all of those things came from 4.0 and it's something that I can use forever as a professional and in my personal life.”

Other alumni mentioned 4.0 improv as something they remembered. For example: “One of the sessions we had was improv. That was powerful for me, just as an experience, it was something I had not done ever before. I actually ended up taking an improv class, maybe a summer or two ago, just because I remembered that experience and how I enjoyed it. But it also, I think for me, crystallized the importance of the… ‘Yes and’, the building off of, the always looking to see how you can build, connect, extend.”

Another alumnus talked about the role of 4.0 alumni communications and resources, and how it is useful knowing those supports exist even when ventures do not develop or continue: “Oh so what I really, really like is that there's an alumni community, they keep the engagement going. They send out, I don't know how often it is, maybe monthly.

There's a portal that alums can log into for resources and there's I think like a monthly, maybe it's monthly, email blast where they list a whole host of not just updates on other alum and where they are and other, like how their venture is growing and expanding, but also resources around funding, seed funding, also job openings, also just general news. So even for someone like me who never got their venture off the ground and whenever I do return to it, I have a host of resources that I can use.”

In sum, interviews with alumni exemplified the pivoting-related 4.0 mindsets/skills of: identifying assumptions; sharing an idea with people; breaking down a problem and testing it multiple ways; and, trying again after failing. Interviews revealed 4.0’s emphasis on engagement and connection to community. Alumni elevated strengths and lessons learned from their 4.0 experiences and offered specific suggestions where 4.0 itself can pivot as an organization. Interviews revealed opportunities to enrich fellow experiences, especially those who did not fully launch or sustain their 4.0 ventures, by 38

more carefully fine-tuning the coach-fellow process and engaging coaches to specifically guide ventures that pivot, pause or stop in order to bring intentionality to those processes.

TO WHAT EXTENT DO FELLOWS BECOME LEADERS IN THE EDUCATION FIELD? Results from the Alumni Survey (2022) suggest that after participating in 4.0 fellows do become leaders in the education field. When asked “would you consider yourself, or would others consider you, a leader working in education?” 83.3% of respondents reported that yes, they would be considered leaders in this field (see Table 6).

Consistent with 4.0’s model, these 195 individuals can be expected to be catalysts in their communities, positively impacting those that they encounter. Furthermore, 86.7% of fellows reported that 4.0 somewhat all the way up to directly and strongly contributed to their leadership capacity (See Figure 3).

Table 6 Alumni Survey 20228 Leadership Data Would you consider yourself, or would others consider you, a leader working in education?

Yes

195

86.3%

No

31

13.7%

Total

226

100.0%

8

Question was not asked in 2021

39

Figure 3: To what extent did your 4.0 experience help develop your leadership capacity for your current role? (Alumni Survey, 2022, mean = 7.8)

40

FOCUS AREA 2: IDEA DEVELOPMENT WHAT EARLY-STAGE IDEAS HAVE BEEN SUPPORTED BY 4.0?

This focus area centers on the increased identification of promising educational innovations supported by 4.0 funding and support. Two sources of data inform this question. First, we have examined participation data from the pop-up pilot tests that fellows conducted during the years 2019-2021. These data are generated from the Graduation Survey. Further, at the completion of program training, graduates were asked to describe their venture 240 characters or less. The following is a table of a selection descriptive examples of ventures funded by 4.0 in the respondents’ words.

4.0’S PILOT/POP-UP REACH (2019-2021) Despite the effects of the COVID-19 epidemic on communities across the country, those participating in the 4.0 program during that time maintained considerable reach between 2019 and 2021, including nearly 6,000 participants taking part in pilot and pop-up project tests (Table 7). A breakdown of participants by type and year is provided in the table below. As noted, in 2019-2020 nearly a quarter of participants were high school students, and a second quarter were teachers. This shifted in the later year (2021) when most participants were middle school students, with high school students also representing a substantial portion of the overall participation sample.

Table 7: Pop-up and Pilot Participants 2019-2021

2019-2020 Grad

2021 Grad Survey

Survey

(Who attended your

(Who attended your

project? How many

pilot? # of each

of each community

community)

most affected by the problem?)

Community

#

% of Total

#

% of Total

41

Early Childhood (Pre-K and earlier)

115

4.8%

64

1.9%

Students (Elementary; about grades K-

360

14.9%

475

14.0%

Students (Middle; about grades 6-8)

187

7.7%

877

25.9%

Students (High; about grades 9-12)

544

22.5%

697

20.6%

Students (Higher Ed)

134

5.5%

137

4.0%

Teachers

538

22.3%

296

8.7%

Parents/Families

362

15.0%

386

11.4%

Administrators / Districts

88

3.6%

105

3.1%

Other #

87

3.6%

349

10.3%

GRAND TOTAL

2,415

100.0%

3,386

100.0%

5)

The Alumni Survey data also helps to provide an understanding of the kinds of projects funded by 4.0. Table 8 below indicates that most projects are programs or services.

Table 8: Which of these best describes your venture? (Alumni Survey) Year

2021

2022

Program/Service

52.61%

44.00%

School

17.16%

17.33%

Educational Technology

16.42%

16.44%

7.09%

4.89%

Early Childhood Education

42

Physical Product

0.37%

0.44%

Education Support

N/A

2.22%

Employment or Training

N/A

9.33%

Other

6.34%

5.33%

Total

100%

100%

Funded Early-Stage Ideas in Under 240 Words Looking beyond numbers, we offer the following descriptions of ventures. Fellows are taught to succinctly describe their venture or idea and report these descriptions in the Graduation Survey (Graduation Application). These short (240 character or less) descriptions capture the variety and intention of the work funded by 4.0. Below is a random selection of 20 of these descriptions.

•

Global Citizens School aims to eliminate the barriers to educational opportunities for Black and brown children, Pre-K3 - 5th grade, living east of the Anacostia River, by providing equitable access to a high-quality dual language school.

•

We help refugee single mothers get beyond "just getting by" in a traumainformed live/learn/workspace where they can build skills & community. A culturally diverse early childcare center is the backbone to our success.

•

He is Me hosts SEL-centered programs for Black male college students to gain authentic practice teaching middle school boys of color. After a year-long fellowship, our alumni matriculate into a teacher residency to hone their skills.

•

The Young Men of Color Symposium is a learning environment centered around equitably and emotionally educating black and brown young men. It is a school grounded in a culture of excellence in academics, an increased self-awareness

•

The Middle Ground is a Professional Development Program for middle managers (i.e. assistant principals, deans and instructional coaches) to develop skills in the areas of Leadership, Management, Instruction and Therapeutic Rapport.

43

•

The Village Community Project provides culturally and community centered programming and resources catering to Black and Brown families, communities, and institutions that serve them using a liberatory pedagogy lens.

•

We will provide aspiring educators of color with professional development in CGI and real-world classroom application of math, while creating a space that is inclusive of their experiences, background and passion.

•

Lifting Our Stories (L.O.S.) seeks to empower the youth of the IE to lead, serve, and enact social change by providing a unique outdoor-based leadership curriculum, mentorship, and civic engagement opportunities.

•

Indigenous families and students know what they need from schools, but they are rarely asked. The First Step Fellowship will push teachers to listen radically, deepen community ties, and support help Indigenous students thrive.

•

A content series to explore skilled trades and careers in an interactive and informal way. We show you how to get started as a professional by breaking it down step by step.

•

Global Voice offers solutions for ELL students, their teachers and schools that are personalized, right-sized, and just-in-time to make every minute in class count.

•

It will take 228 years for blacks to build the wealth of their white counterparts. We are not waiting. Our school model goes beyond a college degree, it's about knowing how to invest for tomorrow.

•

A guided journal that focuses explicitly on racial identity, mental health, and resiliency, for children ages 4-8.

•

Transformative Teach supports educators and all leaders to unpack their implicit bias in spaces with young people, and uplift the joy, center socio-emotional health, and invest resources in BIPoC youth.

•

We are recruiting Filipino-American professionals to share college admissions information and to provide our Filipino-American attendees with role models who may have similar family and cultural backgrounds and experiences.

44

•

Achieve Your Purpose Academy is a K-8th school where students explore different components of their identity through literacy across core subjects and produce projects that highlight their core self-identified values and overall being.

•

Reading Train provides 100s of print and digital story books with audio and visual supports so that the very earliest readers can read and record the books independently, building self-confidence, vocabulary and foundational reading skills.

•

Ivy New Orleans Leadership is New Orleans' only all girls charter school designed to empower scholars to lead their way through our innovative program which integrates academic support, leadership development, and wellness.

•

Immersive gatherings transform and re-ignite educators to find wholeness individually and thrive together. We build solidarity and vulnerability through joyful and contemplative activities. We nourish, nurture, and flourish as one.

•

Imagine students emboldened to create a future Chicago. Imagine school experiences that inspire passions, infuse strong, supportive relationships and catalyze students to impact their communities as social entrepreneurs and innovators.

HOW HAVE FELLOW VENTURES CONTRIBUTED TO EDUCATION INNOVATIONS? Fellowship ventures that are supported by 4.0 exemplify a broad range of innovation and diversity. While 4.0 provides early investments for a broad variety of projects, all are unified by 4.0’s underlying long-term goals of increased academic and socio-emotional learning (SEL), improved education, improved quality of life, and improved communitybased, culturally-relevant educational programming. Some examples of innovation areas addressed by 4.0 ventures are: ·

SEL tools and resources

·

Mindfulness, mental health, self-discovery, empathy and healing/restorative practices in school and community environments

·

College and workplace readiness, mentoring and reducing financial burdens of higher education

45

·

Character and leadership development, civic dialogue, community organizing, social justice, and social impact projects to address community issues

·

Anti-racist teachings and workshops for educators to strengthen culturally responsive teaching

·

STEM education, reading and non-English language programs

·

Most ventures also have a particular focus to reach and uplift members of marginalized communities (BIPOC, LGBTQIA, special needs, teenage girls of color, and more)

In addition to these overall educational innovations, early childhood education (ECE) is one of 4.0’s key investment areas and serves as a foundational example of how ventures have grown to contribute to education innovations. In the ECE Case Studies (2021), Alumni identified four ways that 4.0 contributed to the launch of their six ECE ventures, and to their development as a founder (see Table 9).

Table 9: The Six Innovative ECE Ventures (ECE Case Studies, 2021) Mathematician, Jr

The program recognizes that for many, the idea of math instruction is laden with poor experiences.

Mathematician, Jr. works to provide a novel tool, along with other resources for parents and children, to “accelerate math learning.”

Early Partners

Recognizing the importance of early childhood education for both kids and parents, Early Partners seeks both to provide equitable access to high-quality early childhood education for all children and to support their teachers and families. This program’s core values include: learning, kindness, creativity, teamwork, and responsibility. Early Partners currently operates as a non-profit, employer-based childcare center with a family academy component to support parents.

46

Family Literacy

Focused on supporting children and parents to develop

Empowerment Program

pre-emergent literacy skills, this program conducted

(FLEP)

workshops (prior to Covid-19) addressing pre-emergent literacy skills. FLEP also sought to reduce stigma and notions that “low socioeconomic minority parents…don’t care about literacy skills.”

Training Grounds Inc.

This non-profit organization helps families and professionals to understand the development of children aged 0-5yrs. The program provides a free We PLAY Center and parent workshops as well as fee-based professional workshops. The program often works with foster care parents, as well as parents and professionals, all of whom can benefit from understanding typical developmental milestones, and how children, including those that are on the autism spectrum, might be different.

Cognitive ToyBox

This venture is a unique school-based assessment platform for teachers; it includes technology-, observation- and game-based elements, seeking to make childhood assessment more efficient and effective.

WHAT CAN BE LEARNED FROM IDEAS THAT FAIL? From the Alumni Survey (2022) it is known that 87.7% of ventures are continuing in some way. Specifically, 70.4% of respondents’ ventures are still active and for 17.3% of respondents, while their original venture is not active, they continue working on ideas developed through 4.0. This pivoting is of interest to the research team, and the lessons that can be learned from this (and, therefore, ideas that fail) is captured below in content sourced from the Alumni Interviews (2021) described below.

47

Due to a variety of factors, many alumni have been faced with the decision to pause, transition, continue, or end their venture. These crossroads moments are a crucial, and many times difficult, period for 4.0 fellows. Several interviewees shared that they would benefit from assistance navigating these decisions. Here an alumnus describes how the venture idea has slowed yet stayed with them and what questions they are exploring about how to continue being impactful with their efforts.

“Now I'm a full-time student, so I'm not doing any of the work explicitly… I have a website, I put posts up every week. But in terms of a full-fledged venture, it's never become a full baked venture that really clearly defined a problem statement, really clearly defined a solution, that has been tested, piloted, scaled.

It hasn't gone through the full evolution of that. I'm almost… kind of still at the starting blocks, around really trying to figure out and refine what it can be… [I’m] figuring out, does it need to be, or are my experiences, my passion, my skill set, better served and more impactful if I just work with an existing organization that is also deeply invested in doing work around young black men?... There's lots of organizations doing collective impact work, doing community-based work. It could be its own thing, I could also be very happy just working with an existing organization that's invested in a similar group as I am.”

An alumnus suggested that 4.0 could create openings or opportunities following major personal or professional transitions in order to counsel alumni through pausing an idea, or pivoting, rather than stopping the venture. There is a lack of closure for alumni whose ventures have not continued because no one coached them away from the venture, no one did a “closeout interview.”

“At the end of the fellowship, we had our graduation. And a lot of it was focused on what are the next steps for your venture, how do you continue leveling up, and how do you expand. It was a lot of skill building for what's coming next... I wish that there had been an option for me, when I went back for my graduation, to really process and decide, just have somebody to be a sounding board about the idea of closing down an idea. What I liked about 4.0 is I felt like that was 48

always on the table. It was never like everybody is going to succeed and everyone's idea is going to become a million-dollar business. It was always like failure is common, failure is to be expected. It is okay. That was really great. But I felt like I could have used someone… It would have been really great if there was a way to decide, and not just me in my head feeling the lack of confidence, me just deciding on my own. And again… I had all these life things going on, so it's possible that nothing could have changed my mind, but it would have been nice to have been walked through it in a more Focus area way. How do you decide if it's the right time to close it down, or how do you adapt your idea to a new location?... For me I was just like there was enough things going on in my life that I was like I'm not going to pick it back up again. But I think that it's possible that I could have pivoted, I could have truly paused [instead of stopped]… I think that's what would have been helpful, is maybe some workshop or someone to walk me through in a more objective way, are you sure you're ready to give up on it entirely, or do you want to pivot it to just being for elementary [schools], do you want to pivot it, what would it look like to close down something in one community and restart it in another community? How would you even know that is necessary in the new community?”

Suggestions from alumni interviews echo this idea that the roles of 4.0 coaches could be expanded to: include advising fellows through a pivoting process, make decisionmaking more intentional, and provide closure if they ultimately decide to end or pause their ventures: “The other thing is you could think about maybe a coach or somebody who has ended their venture, who could help maybe talk a person through that who was... in that crossroads.”

Another interviewee described how coaching could be a “thought partnership” to help bring clarity to venture goals and their future course: “I think for me personally, I probably needed, and still need, which is why I'm still grappling, or would benefit from, I need probably just almost, I guess 49

masterminding, where it's a group of four of us, it's a group of six of us, somebody [who's] a more expert or coach, who's taking this group of six through the process over a year. Whether it's meeting once a month or whatever it is, but it's much more high touch and much more thought partnership, to get to the clarity of what it is you're trying to do.”

HOW HAVE EARLY-STAGE IDEAS GROWN AND PIVOTED? When asked in the Alumni Survey (2022), “Did you receive additional funding for your idea/venture beyond what you first received from 4.0?” 53.65% of respondents reported that yes, they had received additional funding from another source. This indicates substantial growth in more than half of the early-stage ideas funded by 4.0. The amount of funding received ranges from $1 to $15,000,000. Additionally, alumni expressed gratitude to 4.0 for their support, describing how it was vital for their ventures: “That your funding helped pay for my rent is no small feat and is what kept me going.”

Some also noted that they would be eager for additional funding opportunities for both early-stage and more developed ventures from 4.0.

Another way to assess idea development and pivoting through coaching. Coaching is a critical part of the 4.0 process, and when utilized to its full potential, it becomes a key asset for fellows to launch, adjust, and meaningfully transition their ventures.

Some ventures pivoted as part of the 4.0 process and coaching was critical to help fellows launch projects, adjust, and transition. The following excerpt explains how coaches are a key asset in helping each venture find its path and in supporting ventures to apply the mindset of approaching a problem in multiple ways: “The venture started off really broad… We didn't end up doing what we planned to do, but we just ended up doing what we could do and created some lesson plans. We work with teacher professional development, so we ultimately ended up creating some activities that we [still] use in our workshops. It was very helpful, but it was not what we planned to do, which I think is par for the course…

So we learned the meaning of the word pivot… we learned to pivot, and we 50

pivoted constantly… It probably wasn't accidental… I had a very good coach.

That was very helpful, I think we had, it was either weekly or maybe it was every other week coaching. I found that valuable.”

However, there is a critical balance between supporting projects to pivot/pause and to keeping them accountable. The value of the coach-fellow relationship was highlighted by other alumni that described lost opportunities in the coach-fellow relationship. Alumni suggested that coaches should actively facilitate the process of pivoting and “pushing” or challenging fellows. For example: “We had coaching. We had a coach who we really loved personally and could get a drink with, and all that, really, really great. The type of coaching that it was, though, if anything, it played into a lot of our insecurities. What I mean by that is, we saw a lot of failures. And they weren't really trying to keep that motivation, to keep pushing. It was positive, but it was kind of like, ‘Okay, that's just where it is.

All right. We'll just leave it.’ And it seemed kind of complacent… I think a stronger, more rigorous coaching opportunity where, when meeting with our coach, there are a lot of expectations, more expectations or more deliverables, or anything like that, to kind of keep us going, and if anything, to inspire us to pivot or find new ways to really grow within that venture… The coach wasn't somebody who you meet with once. We met with our coach, for a while, monthly, then biweekly, and then weekly. And you really get to know that person, and you really care about them. And I think, not that this person wasn't a great person. I highly admired them. I just think I needed someone to be like, ‘Let's pick ourselves up.

We got this. Let's take this idea and move it here.’ They don't have to give us the plan, the major plan, but maybe just give us a little push or nudge into that next step… [A]t the core of 4.0, I think they really cherish the autonomy of participants… And I don't think they want to set too much structure around everything, because then that might scare someone off, or it may stop a really great idea from blossoming into something new… I understand that 100%, and I know that's probably a really hard line to toe by any means. I just think for us, being so green to everything, I think we really needed to build up the grit or that resilience.”

51

The need for an expanded coaching role was echoed by another alumnus who ultimately never successfully launched their pilot: “I would credit them [4.0] with getting me to walk away from what I would call a training or workshop thinking about what I may need to do more to launch [the venture]. So I think that was positive, but ultimately, I think with those questions leaning out there and coinciding with a time in my life that was a bit tumultuous, it got left on the shelf… I think there was a follow-up call and maybe a coaching call a month or so later. I think perhaps having one or two additional touch points with 4.0 about my venture specifically, and perhaps helping me explore the tension in those questions that I raised and the reality of the space, etc., I wonder if it would have maintained a motivation for launch, because I came into the workshop or the fellowship Essentials geared up for, I think, a fall launch that year or whatever. I left feeling like I needed to do more work to launch. And so I think the gap between that and launch could have been just maybe one or two additional really substantive and robust touch points with them potentially would have kept me moving toward a launch, even if it was just a small pilot.”

A fellow whose venture is still continuing discussed how highly effective and impactful their coach was in helping them transition their venture into a nonprofit organization.

The match process of this fellow with a coach who had relevant firsthand experience was critical to achieving this success: “And I think that was possibly the most beneficial experience, just working so closely with someone who had done that before and started their own nonprofit.

And I think it was mainly through our conversations with [them] that we were like, ‘Okay, we can do this. We should do this, start our own thing.’”

Overall, these results speak to the success of 4.0 in supporting idea development through its fellows. The ideas are innovative, and fellows are able to pivot when necessary.

52

FOCUS AREA 3: ECOSYSTEM DEVELOPMENT HOW DOES 4.0 FOSTER NETWORKS AND COLLABORATION WITHIN THE ALUMNI NETWORK? This focus area centers on the increased development of promising ventures in key focus areas through facilitation of fellow-funder partnerships and alumni networks. The approach to networking has shifted in the past several years, in response to the COVID19 pandemic and in ability to effectively host in-person events. As a result, efforts have shifted from geographically centered efforts to thematic efforts, termed “4.0 communities”. While alumni universally support being part of 4.0 networks, responses to the alumni survey “How often do you keep in touch with other fellows you met or collaborated with through 4.0? And Have you participated in any of the following 4.0 Communities?” show that over the past year, participation in events was limited. While 75% reported participating in the alumni network, many, about one-third of those participated annually or a few times a year (“rarely”). Participation in communities was a bit more limited, where 40% of alumni in 2022 reported some level of participation while 60% did not participate at all.

As noted in the section related to lessons learned, alumni have also reflected in interviews that, due to a variety of factors, they been faced with the decision to pause, transition, continue, or end their venture and that this is a pivotal time for support.

Several interviewees shared that they would benefit from assistance navigating these decisions. Another interviewee described how coaching could be a “thought partnership” to help bring clarity to venture goals and their future course.

FOCUS AREA 4: RESEARCH DEVELOPMENT This focus area centers on increased sharing of lessons learned and best practices for entrepreneurship in education.

53

WHAT ARE 4.0’S KEY FOCUS AREAS AND HOW DO THEY ALIGN WITH BEST PRACTICES IN EDUCATION?

In order to address this question, we will present 2 areas of literature, both foundations of the 4.0 approach. The first is design thinking, and the second is the science of pivoting.

Design Thinking 4.0 is committed to advancing equity in education as well as to developing equity mindsets among their fellows. In order to do so, 4.0 has implemented an approach to curriculum development that merges design thinking with an equity framework. An overview of design thinking, including how it can be implemented within an equity framework, is presented below.

Design thinking, traditionally used in business, engineering, and architecture, is now increasingly being used in the fields of social entrepreneurship and education (Chou, 2018; Panke, 2019). It is both a creative process for problem solving and a humancentered approach that requires extensive exploration into the lives and problems of a community prior to generating solutions. Design thinking requires engaging in empathic conversations with community members and engaging stakeholders in co-creating solutions. Furthermore, design thinking as a process is both iterative and exploratory, as ideas are implemented in real time and as the original problem evolves over time (Liedtka et al., 2017). It provides opportunities to experiment, develop and prototype models, receive feedback, and redesign (Razzouk & Shute, 2012).

Owen (2007) has given considerable thought to the critical skills that should be embedded in teaching design thinking. He describes 13 critical design-thinker characteristics and ways of working, which are often required, but not always taught.

While some skills which are not explicitly taught are likely to transfer in the context of longer programs (such as advanced degrees), he argues that more formal approaches to instruction are needed. Among the skills discussed are topics like conditioned inventiveness, which emphasizes that what is created is novel, and inventive, but also 54

practical in the context of the environment in which it is placed. Similarly the concept of human-centered focus, calls for what is designed to continually respond to the needs of the people for whom the solution is targeted. Other concepts of note include recognizing the importance of sustainability and the best interests of the environment are considered (environment-centered concern) as well as recognition for a key foundation of design thinking, the visual ability to images to depict ideas across a range of medias.

Tempered optimism, bias for adaptivity, predisposition toward multifunctionality, systemic vision, view of the generalist, ability to use language as a tool, affinity for teamwork, facility for avoiding the necessity of choice, and ability to work systematically with qualitative information, round out the 13 key elements presented.

In recent years, there has been a call to reformulate design thinking to intentionally incorporate equity so that innovators and social entrepreneurs produce more equitable outcomes (Williams, 2019). For example, the EquityXDesign framework merges “the consciousness of racial equity work with the methodology of design thinking” (EquityXDesign, 2016). This framework rests on the assumption that in order for design thinking to lead to more equitable products, systems, and institutions, then design thinking processes, tools, and mindsets themselves must be redesigned. The human designers must work to uncover their explicit and implicit biases, as well as acknowledge the power of systematic oppression (EquityXDesign, 2016).

Another example of modifying design thinking practices is the Equity-Centered Community Design approach created by Creative Reaction Lab (Creative Reaction Lab, n.d.). Through this design process, individuals learn tools intended to help dismantle systemic oppression which in turn can create a more equitable society. These tools include skills which target a variety of areas including approaches to building humility and empathy, defining and assessing topic and community needs, recognizing history and healing, as well as acknowledging, sharing, and dismantling power constructs (Creative Reaction Lab, n.d.).

55

Such approaches both reflect a new generation of designers and respond to the current need to establish programs and practices that re-define norms and power structures to effectively enable entrepreneurs and oppressed communities to thrive. Such approaches are an emerging method of thinking about, and designing intervention approaches, and are increasingly important in the 4.0 curriculum and implementation strategies.

The Science of Pivoting Pivoting, which was traditionally used in business settings, is now increasingly being applied in the field of social entrepreneurship education (Crilly, 2018; Flechas Chaparro and de Vasconcelos Gomes, 2021). Pivoting is described by Reis (2011) as “a structured course correction designed to test a new fundamental hypothesis” (p. 149). A pivot comprises strategic decisions made either after a failure or in the face of potential failure and leads to changes in the course of action (Flechas Chaparro and de Vasconcelos Gomes, 2021). Four stages of the pivoting process were identified as critical to idea development (Flechas Chaparro and de Vasconcelos Gomes, 2021): 1) Recognition 2) Generating Options 3) Seizing and Testing; and 4) Reconfiguration

These four pivoting stages are mirrored across 4.0 mindsets/skills, which include the following strategies, among others: 1. Identifying assumptions (Recognition); 2. Sharing an idea with people who have not heard it before (Seizing; Testing); 3. Breaking down a problem and testing it in multiple ways (Generating Options; Testing); 4. Understanding how tiny tests apply to the bigger picture (Testing; Reconfiguration);

56

5. Failing and learning from the experience (Testing; Generating Options; Reconfiguration) 6. Trying again after failing (Reconfiguration).

An effective pivot relies on the abandonment of preconceptions and acknowledgment of resultant new ways of seeing. While the importance of pivoting is increasingly understood and accepted, it remains a challenging concept to embrace and implement.

Ries (2011) has been clear that the pivot is one of the hardest decisions an entrepreneur can face. Decisions to pivot at an individual level incorporate important cognitive aspects of awareness and change and are closely linked to examining biases.

This role of pivoting has been discussed in the disruptive innovation literature. Often the examination of one’s own or the dominant logic is critical to the process of recognizing and potentially rejecting assumptions (Chesbrough, 2010).

WHAT ARE 4.0S STRENGTHS AND NEEDS AROUND APPLIED DATA COLLECTION AND EVALUATION SKILL BUILDING, FOR BOTH VENTURES IT SUPPORTS AND AS AN AGENCY?

This research question is one that we expect to address moving into the final year of the partnership, however several significant activities have already been undertaken that we felt important to report. These include monitoring our RPP and the development of a logic model in concert with the program teams which reflects both evaluation objectives as well as current activities undertaken as part of fellow-programming.

MONITORING OUR RPP Between January and February of 2022, a brief survey was administered to members of the UD research team partners and 4.0 practice team partners. This survey was composed of 14 questions assessing the partnership based on the five dimensions of an RPP described above. The results from this quick check are presented below in Figures 4 and 5.

57

Figure 4: Personal Experience in the 4.0-Cresp RPP

The results of the Quick Check revealed that the overwhelming majority of the respondents felt positively about their personal experience in the 4.0-CRESP RPP. In response to the first statement, “my opinions are taken into account in this partnership,” 100% of respondents reported that yes, their opinions were taken into account “to a great extent.” In response to the second question, “our organization benefits from this partnership,” approximately 89% of respondents reported that their organization benefited from this partnership “to some extent” or “to a great extent.” Similarly, in response to the third question about voice, 100% of the respondents said that their voice was heard in the partnership “to some extent” or “to a great extent.” And lastly, 100% of the respondents said that their experience was valued in the partnership “to some extent” or “to a great extent.”

58

FIGURE 5: ASSESSMENT OF THE 4.0-CRESP RPP

The next group of questions in the Quick Check asked respondents to assess the partnership more generally. Overall, the feedback was positive. The areas where at least one respondent stated that “we could do a little better” were as follows: analyzing data, jointly interpreting findings, rigorous research methods, and original analyses.

The findings of the surveys were used as discussion points in subsequent RPP meetings. The survey is planned to be administered again in the summer of 2022, approximately six months after its first administration. Further, results generated from the Alumni Survey, which was completed in March 2022 and discussed below, also served as an important opportunity to jointly interpret findings and mutually discuss data analysis. As a result, an Alumni Survey report was prioritized for completion in the summer of 2022.

Developing, Refining and Using A Logic Model In order to clarify activities and outcomes and to help prioritize outcomes from measurement, considerable effort was spent during 2021 and 2022 to work with 4.0’s 59

Evaluation and Learning and Programs Teams to examine existing outcomes and measures, think about on the ground realities and practices, and ultimately to refine the logic model to better represent current practice. The model is structured in terms of the 4.0 key areas of talent development, idea development, ecosystem development, and research development. This logic model forms the basis of research and evaluation, as well as reporting and is now actively used to guide evaluation decisions (Figure 6).

Figure 6: 4.0 Logic Model

WHAT LESSONS LEARNED REGARDING SMALL-SCALE PILOT EDUCATION INNOVATIONS CAN BE SHARED WITH THE BROADER RESEARCH AND EDUCATION COMMUNITIES?

In the upcoming year, we plan to conduct interviews with 4.0 staff and leadership in order to understand the critical issues in supporting pilot education innovations, 60

including supporting alumni. Funding is an important component of this conversation including the extent to which funding needs are met.

In the past 2 years, we have shared recommendations and best practices through the production of the five reports noted in the introduction. Among these is the “New Normal” evaluation report, which reported on the ways in which that particular investment approach addressed the social-emotional learning needs during COVID-19.

Survey materials were developed by UD and supplied to the pilots who, in turn, administered the surveys to their participants. This yielded common data across New Normal programs. Over 250 youth completed a survey administered after participation in the pilot. Findings from the youth surveys revealed that youth are eager to help those in need, and feel close to their friend group, even during the pandemic. However youth disagreed that: •

“Adults in my town or city listen to what I have to say;”

•

“I am popular with others my age;”

•

“I feel like an important member of my local community;”

•

“I have a lot of friends;”

•

“I give time and money to make life better for other people.”

Regarding social awareness, about 40% of youth disagreed with the statement “I am comfortable describing my feelings.” This was echoed in post-survey comments, where respondents expressed reluctance and discomfort when discussing sensitive topics.

Youth participants provided several recommendations for future programs: include more time for in-person sessions; extend the length of the program; integrate more physical activities; expand the marketing and reach in their community; and structure the program as an outlet for socializing and stress release (and make it less like school).

FINAL THOUGHTS In the “Final Thoughts” section of programming feedback surveys from 2019-2021, fellows were asked what advice they would offer future 4.0 applicants or fellows. The main themes expressed were encouragement to apply, to keep going with the 61

challenge, to remain open and adapt ideas to community feedback, and to build relationships with cohort members and coaches. Following are select quotes that represent these themes that were expressed by fellows in response to this question over multiple survey years: “Apply, apply, apply!! It'll be one of the best things you've ever done.”

“Applicant: APPLY!! 4.0 ESSENTIALS Fellowship was such a liberating and empowering experience for me. As a Black woman I oftentimes experience imposter syndrome and self-doubt. Through ESSENTIALS I gained tangible skills, knowledge, and language to share the mission and purpose for creating my very own early childhood venture. Future Fellow: Be fearless and deliberate in this journey. Share your truths. Build genuine relationships with peers and 4.0 staff. Maintain the relationships post camp and most importantly ask for feedback, peer review, and support as needed.

“Come in ready to expand your horizons and lean into the community as much as possible, with a lens of reciprocity. That is 4.0's most incredible asset: the most diverse, highly dedicated community of passionate change-makers.”

“This is a great opportunity to connect with others who are supportive and passionate. Stick with the process; each part of it is meaningful and important in its own way.”

“Speak your truth about where you are with your venture/idea right now, and don't be ashamed if your idea feels small! Locate where you are, where you've been, and consider the next steps toward where you want to go. If you can say all of those things, you are more prepared than you realize, and 4.0 could be the place to help you on your journey. Own it!”

62

“Actively engage, test your ideas, and be vulnerable. It is an amazing experience.”

“4.0 teaches you the skills and the steps to connect with different stakeholders and transform an idea into concrete actions.”

“The fellowship is about learning, not necessarily succeeding. If you're willing to put in the effort, then you will learn a lot from executing your pilot.”

“Trust your gut! Even if your pilot doesn't pan out the way you wished it had, the spirit of your idea is valuable - stick with it, don't be afraid to pivot, and anchor the changes you make in community feedback.”

“If you get two months into the fellowship and realize your idea doesn't work...that is okay and means the fellowship works.”

“I would encourage applicants to be clear about the needs of the community they want to serve and to stay flexible about the ways they seek to provide a solution in a way that the community has expressed.”

“Do it if you want to have the support and structure to really take the first step towards bringing your business idea to fruition.

“Go into the Fellowship with an open mind and challenge your pre-existing idea. Receiving feedback can sometimes appear overwhelming. Don't be afraid to share your thoughts and ideas and work through your idea. Be willing to connect with others. It is one of the most beneficial supports you can have in forming an equitable and inclusive idea.”

63

“Go with your gut, don't overthink it too much and believe in yourself as much as everyone else believes in you! The first step is just as important as the last. Make everything and every day count.”

“Be open and trust the process. This fellowship will only strengthen you and your idea.”

“Be ready to explore areas that you haven't thought of while building your program. Be open-minded to learning new things and seeing different perspectives.”

“DO IT! Connecting to like-minded individuals is so necessary in the challenging line of work to disrupt the status quo. We, revolutionaries, need families to fall back on, to energize from, to hear stories for inspirations, to receive nourishment from. You are not alone in this work.

Apply to be a part of the family you've been looking for.”

“Really use your coach and cohort to challenge you and move your idea forward.”

“Trust the piloting process and reach out to your cohort members!”

“Embrace every moment with flexibility and give yourself grace. Take advantage of every opportunity to build human connection with your cohort. They are inspiring and just as nervous to go through a pilot as you!”

“Reach out to cohort members for one on one time to get to know them.

That was the most valuable part of the experience!”

“Be open to feedback and receive the coaching.”

64

“You get out of 4.0 what you put into it. The organization has a great network of past fellows and supporters, which can be a great resource if you take the initiative to tap into it.”

“As with most things in life, this fellowship is what you make it.”

“Dream big. Focus down.”

Open-ended comments from 4.0 fellows about their experiences and advice for future fellows reflect 4.0’s mindsets and skills such as trying again after failing, identifying assumptions, breaking down a problem and testing it, and evaluating connections with affected communities. 4.0’s core strategies (Curriculum, Coaching, Community, Cash, and Continuous Improvement), which are integral to the organization’s Logic Model and theory of change, are also exemplified in this feedback. These comments from 4.0 fellows demonstrate that lessons of self-reflection, resiliency, pivoting, and connection are integral to the 4.0 experience and beyond.

RECOMMENDATIONS Based on these findings, the researchers from CRESP propose several recommendations for consideration by 4.0: 1. Consider an in-person alumni meeting for those who completed the program during COVID. a. Alumni from this time would like to have some in-person experience. A gathering, follow-up refresher or similar attempt to invite them to network and celebrate their accomplishments would be welcome.

2. Assist alumni in connecting to 4.0 in new ways and through existing channels they might not be aware of. a. If not done already, consider informing graduates before they leave the program of the alumni opportunities available to them, and sign them up at that time. Additionally, other opportunities to increase outreach to alumni communities should be explored, as alumni are mixed in terms of their 65

familiarity with what communities exist and how they can get involved with them. b. Create additional supports for alumni around funding, which is an important reason that they would like to stay connected. Alumni would like ongoing opportunities to learn about where to find resources, how to apply for funds, and how to write grants and structure proposals.

3. Explore the ways in which 4.0 training could tailor efforts to best support entrepreneurs with similar interest areas, perhaps offering content expertise and evidence-based ideas, as well as examples of effective strategies to foster fellows’ understanding of current best practices. a. Additional needs described during the application stage may also be a framework for common instruction, such as app development and technology infrastructures. i. A venture self-assessment completed by fellows periodically throughout 4.0 participation, as well as the opportunity for alumni to pick workshops that are themed around where their idea is now, including topics such as pivoting may be of use. This could particularly support those that are re-designing ventures/pivoting. b. Furthermore, teambuilding during and after the fellowship for common venture ideas could be expanded upon. c. Finally, consider how fellows are educated about the variety of activities, theory and content options used in similar programs. For example, creating a mix of activities that include opportunities for physical activity and de-stressing, especially if programming extends the school day for youth.

4. Ascertain fellows’ incoming knowledge and utilization of mindsets and skills such as joy, anti-racism, equity and determination. This is critical so that programming is not redundant with their existing knowledge. Rather, programming should expand upon the areas where there are gaps or dive deeper into areas where they already have background knowledge.

5. Continue developing evaluation tools and measures that are as targeted and efficient as possible.

66

a. Consider a process that occurs annually to review survey tools from the prior year. As part of this strategy, reflect on the alignment of tools with program components and priorities. Maintain consistency whenever possible and update the tools as needed for the subsequent year, eliminating questions when they are not needed. b. Furthermore, a strategic review of the graduation survey in concert with the alumni survey and data collected on fellow applications may be of use.

Questions such as those that focus on participant records of their assumptions, for example, may be lengthy for respondents to complete and difficult to use in reporting.

67

REFERENCES Aram, D., & Bar-Am, O. C. (2016). Mothers helping their preschool children to spell words: A comparison between interactions using the computer vs. pencil and paper.

International Journal of Child-Computer Interaction, 7, 15-21. https://doi.org/10.1016/j.ijcci.2016.03.001 Bales, D., Dalsemer, K., Blagojevic, B., Hartle, L., Chung, N., Gardner, K., MaCleod, K., & Rodriguez-Vazquez, J. (2020). Using technology to enhance children’s learning at home and at school: Building relationships is key. https://www.naeyc.org/resources/blog/using-technology-enhance-childrens-learninghome-and-school Barnett, W. S., Epstein, D. J., Carolan, M. E., Fitzgerald, J., Ackerman, D. J., & Friedman, A. H. (2010). The State of Preschool 2010: State Preschool Yearbook.

National Institute for Early Education Research.

Becker, B. D., Gallagher, K. C., & Whitaker, R. C. (2017). Teachers' dispositional mindfulness and the quality of their relationships with children in Head Start classrooms.

Journal of School Psychology, 65, 40-53. https://doi.org/10.1016/j.jsp.2017.06.004 Bennett, S. V., Gunn, A. A., Gayle-Evans, G., Barrera, E. S., & Leung, C. B. (2018).

Culturally responsive literacy practices in an early childhood community. Early Childhood Education Journal, 46(2), 241-248. https://doi.org/10.1007/s10643-017-08399 Bierman, K.L., Morris, P.A., & Abenavoli, R.M. (2016). Parent engagement practices improve outcomes for preschool children. Edna Bennett Pierce Prevention Research Center, Pennsylvania State University https://www.rwjf.org/en/library/research/2017/02/parent-engagement-practices-improveoutcomes-for-preschool-child.html

68

Blewitt, C., Fuller-Tyszkiewicz, M., Nolan, A., Bergmeier, H., Vicary, D., Huang, T., ... & Skouteris, H. (2018). Social and emotional learning associated with universal curriculum-based interventions in early childhood education and care centers: A systematic review and meta-analysis. JAMA network open, 1(8), e185727-e185727. https://doi.org/10.1001/jamanetworkopen.2018.5727 Burchinal, M., Howes, C., Pianta, R., Bryant, D., Early, D., Clifford, R., & Barbarin, O. (2008). Predicting child outcomes at the end of kindergarten from the quality of prekindergarten teacher–child interactions and instruction. Applied Development Science, 12(3), 140-153. https://doi.org/10.1080/10888690802199418 Burgess, E., & Ernst, J. (2020). Beyond Traditional School Readiness: How Nature Preschools Help Prepare Children for Academic Success. International Journal of Early Childhood Environmental Education, 7(2), 17-33.

Byrd, C. M. (2016). Does culturally relevant teaching work? An examination from student perspectives. Sage Open, 6(3), 1-10. https://doi.org/10.1177/2158244016660744 Chesbrough, H. (2010). Business model innovation: Opportunities and barriers. Long Range Planning, 43, 354–363. doi:10.1016/j.lrp.2009.07.010.

Chou, D. C. (2018). Applying design thinking method to social entrepreneurship project.

Computer Standards & Interfaces, 55, 73-79. https://doi.org/10.1016/j.csi.2017.05.001 Cordiano, T. S., Lee, A., Wilt, J., Elszasz, A., Damour, L. K., & Russ, S. W. (2019).

Nature-Based Education and Kindergarten Readiness: Nature-Based and Traditional Preschoolers Are Equally Prepared for Kindergarten. International Journal of Early Childhood Environmental Education, 6(3), 18-36.

69

Cornelius-White, J. (2007). Learner-centered teacher-student relationships are effective: A meta-analysis. Review of Educational Research, 77(1), 113-143. https://doi.org/10.3102/003465430298563 Creative Reaction Lab. (n.d.). Our approach: Equity-centered Community Design. https://www.creativereactionlab.com/our-approach Crescenzi, L., Jewitt, C., & Price, S. (2014). The role of touch in preschool children's learning using iPad versus paper interaction. Australian Journal of Language & Literacy, 37(2), 86-95.

Crilly, N. (2018). ‘Fixation’ and ‘the pivot’: balancing persistence with flexibility in design and entrepreneurship, International Journal of Design Creativity and Innovation, 6:1-2, 52-65, DOI: 10.1080/21650349.2017.1362359. de Brey, C., Musu, L., McFarland, J., Wilkinson-Flicker, S., Diliberti, M., Zhang, A., ... & Wang, X. (2019). Status and Trends in the Education of Racial and Ethnic Groups 2018.

NCES 2019-038. National Center for Education Statistics. https://nces.ed.gov/pubs2019/2019038.pdf Dickinson, D. K., & Porche, M. V. (2011). Relation between language experiences in preschool classrooms and children’s kindergarten and fourth-grade language and reading abilities. Child Development, 82(3), 870-886. https://doi.org/10.1111/j.14678624.2011.01576.x Ece Demir-Lira, Ö., Applebaum, L. R., Goldin-Meadow, S., & Levine, S. C. (2019).

Parents’ early book reading to children: Relation to children's later language and literacy outcomes controlling for other parent language input. Developmental Science, 22(3), e12764.

70

Epstein, J. L. (2010). School, family, and community partnerships: Preparing educators and improving schools. Boulder, CO: Westview Press.

EquityXDesign. (2016). “Racism and inequity are products of design. They can be redesigned.” Medium. https://medium.com/equity-design/racism-and-inequity-areproducts-of-design-they-can-be-redesigned-12188363cc6a Flechas Chaparro, X.A. and de Vasconcelos Gomes, L.A. (2021), "Pivot decisions in startups: a systematic literature review", International Journal of Entrepreneurial Behavior & Research, Vol. ahead-of-print No. ahead-of-print. https://doi.org/10.1108/IJEBR-12-2019-0699 Flewitt, R., Messer, D., & Kucirkova, N. (2015). New directions for early literacy in a digital age: The iPad. Journal of Early Childhood Literacy, 15(3), 289-310. https://doi.org/10.1177/1468798414533560 Foulin, J. N. (2005). Why is letter-name knowledge such a good predictor of learning to read?. Reading and writing, 18(2), 129-155. https://doi.org/10.1007/s11145-004-5892-2 Gay, G. (2000). Culturally responsive teaching: Theory, research, and practice.

Teachers College Press.

Gay, G. (2002). Preparing for culturally responsive teaching. Journal of Teacher Education, 53(2), 106-116. https://doi.org/10.1177/0022487102053002003 Jennings, P. A. (2015). Early childhood teachers’ well-being, mindfulness, and selfcompassion in relation to classroom quality and attitudes towards challenging students.

Mindfulness, 6(4), 732-743. https://doi.org/10.1007/s12671-014-0312-4 Justice, L. M., & Ezell, H. K. (2001). Word and print awareness in 4-year-old children.

Child Language Teaching and Therapy, 17(3), 207-225.

71

Kermani, H., & Aldemir, J. (2015). Preparing children for success: integrating science, math, and technology in early childhood classroom. Early Child Development and Care, 185(9), 1504-1527. https://doi.org/10.1080/03004430.2015.1007371 Kuo, M., Barnes, M., & Jordan, C. (2019). Do experiences with nature promote learning? Converging evidence of a cause-and-effect relationship. Frontiers in Psychology, 10, 1-9. https://doi.org/10.3389/fpsyg.2019.00305 Ladson-Billings, G. (2014). Culturally relevant pedagogy 2.0: aka the remix. Harvard Educational Review, 84(1), 74-84. https://doi.org/10.17763/haer.84.1.p2rj131485484751 Lerkkanen, M. K., Kiuru, N., Pakarinen, E., Poikkeus, A. M., Rasku-Puttonen, H., Siekkinen, M., & Nurmi, J. E. (2016). Child-centered versus teacher-directed teaching practices: Associations with the development of academic skills in the first grade at school. Early Childhood Research Quarterly, 36, 145-156. https://doi.org/10.1016/j.ecresq.2015.12.023 Liedtka, J., Salzman, R., & Azer, D. (2017). Design thinking for the greater good: Innovation in the social sector. Columbia University Press.

Mann, V. A., & Foy, J. G. (2003). Phonological awareness, speech development, and letter knowledge in preschool children. Annals of Dyslexia, 53(1), 149-173.

The Mindfulness Initiative. (2016). Building the case for mindfulness in the workplace. https://www.themindfulnessinitiative.org/Handlers/Download.ashx?IDMF=46ef10fd4d64-41f9-91a6-163d52cd304c Mol, S. E., & Bus, A. G. (2011). To read or not to read: a meta-analysis of print exposure from infancy to early adulthood. Psychological Bulletin, 137(2), 267-296. https://doi.org/10.1037/a0021890

72

National Association for the Education of Young Children. (2020). Developmentally appropriate practice (DAP) position statement. https://www.naeyc.org/resources/position-statements/dap/contents North American Association for Environmental Education (NAAEE) (2017). Nature preschools and forest kindergartens: 2017 national survey. https://naturalstart.org/sites/default/files/staff/nature_preschools_national_survey_2017. pdf Office of Educational Technology. (2018). Guiding principles for use of technology with early learners. US Department of Education. https://tech.ed.gov/earlylearning/principles/ Owen, C. (2007). Design thinking: Notes on its nature and use. Design Research Quarterly, 2(1), 16-27.

Panke, S. (2019). Design thinking in education: Perspectives, opportunities and challenges. Open Education Studies, 1(1), 281-306. https://doi.org/10.1515/edu-20190022 Powell, D. R., Son, S. H., File, N., & San Juan, R. R. (2010). Parent–school relationships and children's academic and social outcomes in public school prekindergarten. Journal of School Psychology, 48(4), 269-292. https://doi.org/10.1016/j.jsp.2010.03.002 Radziwill, N. M., Benton, M. C., & Moellers, C. (2015). From STEM to STEAM: Reframing what it means to learn. The STEAM Journal, 2(1), https://doi.org/10.5642/steam.20150201.3 Razzouk, R., & Shute, V. (2012). What is design thinking and why is it important?

Review of educational research, 82(3), 330-348. https://doi.org/10.3102/0034654312457429 73

Ries, E. (2011). The lean startup: How constant innovation creates radically successful businesses. New York, NY: Penguin UK. Rowe, M. L., Denmark, N., Harden, B. J., & Stapleton, L. M. (2016). The role of parent education and parenting knowledge in children's language and literacy skills among White, Black, and Latino families. Infant and Child Development, 25(2), 198-220. https://doi.org/10.1002/icd.1924 Rymanowicz, K., Hetherington, C., & Larm, B. (2020). Planting the seeds for naturebased learning: Impacts of a farm-and nature-based early childhood education program.

International Journal of Early Childhood Environmental Education, 8(1), 44-63.

Tippett, C. D., & Milford, T. M. (2017). Findings from a pre-kindergarten classroom: Making the case for STEM in early childhood education. International Journal of Science and Mathematics Education, 15(1), 67-86. https://doi.org/10.1007/s10763-0179812-8 Torres-Crespo, M. N., Kraatz, E., & Pallansch, L. (2014). From Fearing STEM to Playing with It: The Natural Integration of STEM into the Preschool Classroom. SRATE Journal, 23(2), 8-16.

Wahyuningsih, S., Nurjanah, N. E., Rasmani, U. E. E., Hafidah, R., Pudyaningtyas, A. R., & Syamsuddin, M. M. (2020). STEAM Learning in Early Childhood Education: A Literature Review. International Journal of Pedagogy and Teacher Education, 4(1), 3344. https://dx.doi.org/10.20961/ijpte.v4i1.39855 Weiss, H. B., Kreider, H., Lopez, M. E., & Chatman-Nelson, C. M. (2014). Preparing educators to engage families: Case studies using an ecological framework. Thousand Oaks, CA: SAGE. Williams, L. (2019). The co-constitutive nature of neoliberalism, design, and racism.

Design and Culture, 11(3), 301-321. https://doi.org/10.1080/17547075.2019.1656901 74

Wynn, T., & Harris, J. (2012). Toward a STEM+ arts curriculum: Creating the teacher team. Art Education, 65(5), 42-47. https://doi.org/10.1080/00043125.2012.11519191 Yang, W., Datu, J. A. D., Lin, X., Lau, M. M., & Li, H. (2019). Can early childhood curriculum enhance social-emotional competence in low-income children? A metaanalysis of the educational effects. Early Education and Development, 30(1), 36-59. https://doi.org/10.1080/10409289.2018.1539557 Yoshikawa, H., Weiland, C., Brooks-Gunn, J., Burchinal, M. R., Espinosa, L. M., Gormley, W. T., ... & Zaslow, M. J. (2013). Investing in our future: The evidence base on preschool education. Society for Research in Child Development. https://files.eric.ed.gov/fulltext/ED579818.pdf

75