Real School Gardens Evaluation 2010-2013

REAL School Gardens Program Evaluation 2010 – 2013

Comprehensive Summary Report

Presented to:

REAL School Gardens

Prepared by: Michael Duffin and PEER Associates, Inc.

Fall 2013

 PEER Associates, Inc. is responsible for the quality and content of this report.

Principals Amy Powers and Michael Duffin can be contacted at Amy@PEERassociates.net or Michael@PEERassociates.net.

We wish to extend a particular thanks to the individual teachers and students who so graciously participated in this evaluation.

Â

Suggested citation: Duffin, M., & PEER Associates. (2013). REAL School Gardens, Program Evaluation, 2010-2013: Comprehensive Summary Report. Richmond, VT: Author. Downloaded [date] from www.PEECworks.org.

Table of Contents

Executive Summary ..………………………………………………………………………………..……………. 1

Overview ..……………………………………………………………………………………………….……………………… 3

Findings …………………………………………………………………………………………..……….………………………. 5

Concluding Reflections ………………………………………………………………………….……..……. 19

Appendices ……….…………………………………………………………………………………………….…….……. 20

Executive Summary REAL School Gardens, Program Evaluation, 2010-­‐2013 REAL School Gardens is a nonprofit organization that builds learning gardens in low-income schools, and then trains teachers to use them to improve academics.

REAL School Gardens has delivered its program to nearly 100 elementary schools in North Texas. In the spring of 2010, external evaluators PEER Associates, Inc. were invited to facilitate and implement a highly collaborative, multi-year, utilization-focused program evaluation process. Evaluation activities from 20102013 included building and revising a Logic Model, analyzing responses from over 5000 teacher surveys, and interviewing representatives from 10 schools, as well as implementing many other supporting processes, data collection, and reporting systems.

Most of the evaluation effort focused on the 20 most recent partner schools.

The four overarching findings that emerged from the evaluation work in 2010-2013 are summarized below and described in detail in the full report. Supporting claims are specified for the first two findings because they had the most robust evidence.

1. There was very strong evidence of REAL School Gardens helping teachers become more effective •

Teachers highly valued REAL School Gardens training

•

Individual teachers receiving multiple trainings reported steady, significant increases in effective teaching practices

•

Cohort averages of teacher effectiveness increased strongly and steadily over the first two program years

•

Teachers exceeding target levels of effectiveness received significantly more REAL School Gardens training

•

Preliminary findings from year one of the evaluation were consistent with subsequent analyses of teacher effectiveness after three full years of data collection

•

During interviews, all respondents spoke very highly of the REAL School Gardens Program

RSG Evaluation 2010-­2013

PEERassociates.net

1

2. Teachers unambiguously attributed student engagement in learning to the REAL School Gardens Program •

94% of teachers surveyed agreed with the statement “REAL School Gardens was a major contributor to increased engagement in learning for my students”

•

Positive student engagement in learning was a strong theme expressed in teacher interviews

•

A pilot study of student engagement in learning based on student and teacher survey self-reporting methodology did not support hypothesized correlations

•

A quasi-experimental study of student engagement in learning has been designed but not yet piloted

“[My students] are more alive, they are excited to learn, they are eager to learn. And since they’re excited and eager to learn they take ownership of [the garden].”

-- Teacher, REAL School Gardens Partner School

3. Some evidence suggested potential school-wide effects of the REAL School Gardens Program 4. REAL School Gardens staff honed the program model and its delivery based on evaluation outcomes Overall, evaluation evidence from 2010-2013 suggested that the REAL School Gardens Program met most of its targeted outcomes.

On average, teachers receiving REAL School Gardens training reported doing most of the target teaching practices before the end of the partnership. Meanwhile, when gardens were used as a tool for instruction, students were engaged in learning.

Looking forward, since the evidence showing program effects at the level of individual teachers is solid and growing more so each year, the big picture opportunity is to start a more rigorous exploration of the ways in which the REAL School Gardens Program does and does not translate into measureable effects at the level of a whole school.

RSG Evaluation 2010-­2013

PEERassociates.net

2

Overview REAL School Gardens builds learning gardens in low-income schools, and then trains teachers to use them to improve academics.

Formed officially as a nonprofit organization in 2007 after nearly a decade of successful projects, REAL School Gardens has delivered its program to nearly 100 elementary schools in North Texas and is offering its program to new regions in the fall of 2014.

The three-year partnership begins with parents, students, and funding partners participating in designing and then building the garden. Then REAL School Gardens staff provide professional development (PD) training in the form of whole school events at the beginning and end of the partnership, customized individual trainings and model teaching for selected teachers over the course of each school year, and ongoing supplies and support as needed.

In the spring of 2010, REAL School Gardens engaged external evaluators PEER Associates, Inc. to facilitate and implement a highly collaborative, multi-year, utilization-focused program evaluation process. This report is a formal summary of evaluation work completed 2010 through 2013. Evaluation activities and data are summarized in Figures 1 and 2 below.

Figure 1. Summary of Teacher Surveys Collected To Date

2010 Partner1

2011 Partner

20102012 PD2

Fall 2012 Partner

20122013 PD

Spring 2013 Partner

Fall 2013 Partner

283

243

250

176

86

204

75

1,317

205

160

222

107

191

80

965

12-13 Cohort (7 schools)

235

124

204

80

643

13-14 Cohort (4 schools)

165

91

109

365

10-11 Cohort3 (6 schools) 11-12 Cohort (7 schools)

Other (first pilot sample schools, old model schools from 08-09 and 09-10 Cohorts, FWISD) Total N =

351

634

186

634

824

1,234

1

381

3

799

698

693

Total N=

1,746

344

5,036

“Partner” surveys were administered to all teachers at a school at the beginning of a school year.

“PD” (Professional Development) surveys were administered to teachers as they finished individual training sessions during the course of a school year.

3 REAL School Gardens refers to the schools that start their partnership during the same year as a “cohort.”

1

2

RSG Evaluation 2010-­2013

PEERassociates.net

3

Figure 2. Summary of Key Evaluation Activities, By Contract Year

Spring 2010 • Conducted initial evaluation planning and prioritization • Drafted Logic Model • Designed online surveys to capture Logic Model outcomes using Stages of Change scale and to provide formative feedback for tailoring the content of training events • Began participation in the Place-based Education Evaluation Collaborative (PEEC) 2010 - 2011 • Piloted Partner survey with 18 schools representing a range of program involvement • Piloted PD surveys with teachers in 29 of 91 training events conducted this school year • Interviewed 12 REAL School Gardens “champions” from 10 schools representing a wide range of geography, duration of programming, and role within school • Generated baseline and initial findings executive summary report 2011-2012 • Refined surveys based on initial pilot results • Administered Partner surveys to 20 schools representing cohorts 10-11, 11-12, others • Administered PD surveys to teachers in 88 training events conducted this school year • Refined Logic Model, augmented by Theory of Change developed in parallel process • Built a tool to comprehensively track training dosage at the level of individual teachers as well as school and program levels • Successfully concluded participation in PEEC • Designed and implemented a pilot study of student engagement in learning based on student (N = 2,299) and teacher (N = 53) survey self-reporting methodology • Built dynamic “Reporting Book” tool for systematic, ongoing summary and tracking of school- and cohort-level survey results 2012-2013 • Added items to Partner survey for teachers to report on student engagement in learning • Administered Partner survey to 25 schools representing cohorts 10-11, 11-12, 12-13, and other potential partner schools in the “greenhouse” • Administered PD surveys to teachers at 112 training events this school year • Conducted three data mining sessions with program staff and evaluators • Designed a pilot quasi-experimental study of student engagement in learning • Re-administered Partner survey in fall 2013 to 7 schools that had not had high enough response rates for the spring 2013 Partner survey administration • Generated comprehensive three-year summary report

RSG Evaluation 2010-­2013

PEERassociates.net

4

Findings When considering REAL School Garden’s first three years of program evaluation work as a whole, four overarching findings emerged. Listed in order from most to least amount of supporting data, they are: 1. There was very strong evidence of REAL School Gardens helping teachers become more effective.

2. Teachers unambiguously attributed student engagement in learning to the REAL School Gardens Program.

3. Some evidence suggested potential school-wide effects of the REAL School Gardens Program.

4. REAL School Gardens honed the program model and its delivery based on evaluation outcomes.

These findings are described in detail in the following pages.

RSG Evaluation 2010-­2013

PEERassociates.net

5

Finding #1: There was very strong evidence of REAL School Gardens helping teachers become more effective.

The two main activities that REAL School Gardens provides to schools are building gardens and training teachers (see the Logic Model in the Appendix). Most of the evaluation effort to date has been aimed at tracking the outcomes of the professional development (PD) trainings offered to teachers. Surveys asked teachers to report the level at which they engaged in the six target practices described in the Logic Model (i.e. Content Knowledge, Beliefs About Gardens for Increasing Achievement, Teacher Engagement, Teaching Skills and Methods, Teaching Tools and Resources, and Time Spent Teaching Outdoors). The response scale offered choices corresponding to various “Stages of Change”4 that represent increasing attainment of a target behavior. The theoretical categories are “Disinterest,” “Deliberation,” “Designing,” “Doing,” and “Deepening.”

Six different types of analyses were used over the last three years to test the extent to which training efforts achieved target outcomes. With one exception in part of one of the analyses, all analyses suggested clear, often Figure 3. statistically significant, associations between REAL School Gardens training and increases in effective teaching practices. The convergence of multiple analyses strengthens the finding. The specific findings from each of the analyses are described in detail below.

•

Teachers highly valued REAL School Gardens training. On surveys administered after each training event, 93% of all survey responses (N = 1,599 for all cohorts from all years) agreed with the statements: “This PD connected directly and explicitly with the state curriculum standards I am accountable to” and “I will be able to apply the content of this PD

4 The Stages of Change model was adapted from psychological research on intentional behavior change (Prochaska,

DiClemente, & Norcross, 1992; Doppelt, 2009). The idea is that change happens in sequential increments, reversion to prior stages and temporary stretching to subsequent stages is typical, and success might be measured in terms of progression along a continuum of stages.

RSG Evaluation 2010-­2013

PEERassociates.net

6

to my regular work right away.” Fully 85% of responses indicated “strong” agreement (see Figure 3). This observed ceiling effect means that there is literally no room for improvement on this measure, and that this index should not be thought of as tracking change or improvement over time.

•

Individual teachers receiving multiple trainings reported steady, significant increases in effective teaching practices. Professional development at partner schools happens in two formats: events for the whole school offered at the beginning and end of the partnership, and a series of three or more model teaching sessions delivered to a selected group of four to six teachers at a given school over the Figure 4. course of the year.5 After three years of survey data it became possible to analyze how the responses of 269 individual teachers changed or not over the course of receiving multiple training sessions.

The pattern was clear, steady, statistically significant improvement from session to session on average.

The size of the increase from one training event to the next was small to medium, consisting of about a third of a standard deviation (between .4 and .7 on the five-point scale in this case), which corresponds roughly to 5-10 percentile points. Between the baseline survey and the fourth training event, the average response across all the target practices moved from Designing to Doing on the Stage of Change scale (i.e. from 3.0 to 4.0 on a five-point scale, see Figure 4 and Figure 5).

One of the single survey items (Teacher Engagement, item LM3) comprising the More Effective Educators Index is especially important because of its more prominent role in the REAL School Gardens Theory of Change. Data in Figure 5 show that after four or five or six training events the average teacher survey response corresponded to the Doing or Deepening level (i.e. between 4.0 and 5 on a five-point scale). That means teachers report that “Teaching outdoors/ in gardens/ in our schoolyard energizes me and makes teaching more fun” or “My

5 REAL School Gardens seeks to recruit the “fence sitters” rather than the “choir” in selecting who participates in the individual training/model teaching sessions delivered over the course of a year.

RSG Evaluation 2010-­2013

PEERassociates.net

7

career as a teacher is much better because of my experiences with students outdoors/ in gardens/ in our schoolyard” (see the PD Survey in the Appendix).

Thus the REAL School Gardens Program could be said to increase teacher job satisfaction.

Two other features of this change over “Thank you for giving me the tools to teach differently. You opened my time analysis warrant mention. First, eyes today to a whole new world I the practice that showed the biggest can explore with my students.” average gain over the course of multiple -- Teacher, REAL School training events (from 2.7 at baseline to Gardens Partner School

4.1 after the fifth training event, on a five-point scale) was Time Spent Teaching Outdoors. Second, the aggregate survey data set included 326 teachers who filled out multiple annual surveys but did not participate in any REAL School Gardens training events. This subset of teachers showed an increase after the first year of programming, but then no increases in subsequent years (see Figure 6 in the Appendix). Thus, teachers who did receive REAL School Gardens training reported significant practice change while those who did not receive training did not report practice change.

Figure 5.

RSG Evaluation 2010-­2013

PEERassociates.net

8

•

Cohort averages of teacher effectiveness increased strongly and steadily over the first two program years. At least once a year, all teachers at partner schools were surveyed about their level of the practices targeted in the REAL School Gardens Logic Model. Averaging those responses across the whole school, and then further aggregating those averages across the six or seven schools in a given cohort Figure 7. allowed for unique local differences to cancel each other out, leaving any residual changes over time more likely to be associated with the REAL School Gardens Program. Figure 7 shows upward sloping trends for each cohort.

The size of the annual average gain was fairly large, consisting of approximately a full standard deviation (.3 on the fivepoint scale in this case), which corresponds roughly to 15-30 percentile points. Further, the More Effective Educators Index represented in this analysis reflects the average of six survey items about effective teaching practices.

This suggests that the changes reported by teachers was fairly well rounded and likely not due to strong improvement in just one or two target outcomes.

The one exception to the pattern was the dip in the cohort 10-11 trend line for year three. Program staff pointed to unusual volatility in leadership and/or staff turnover for several schools in this cohort during year three of the program as a likely explanation for this observation. An alternative potential explanation is that enthusiasm and commitment for the program naturally wane after a couple of years. Future survey data should help determine whether cohort 10-11 was an anomaly or an early indicator of a typical pattern.

•

Teachers exceeding target levels of effectiveness received significantly more REAL School Gardens training. This program is conceptualized as a tool for helping teachers who are ready to embrace learning outdoors and in gardens gain the tools and support to make that change. One could say that ideal candidate REAL School Gardens partner schools are “fertile soil” awaiting “cultivation.” In the terms of the evaluation surveys, that translates as moving teachers from the Deliberation or Designing to the Doing or Deepening stage with respect to the target behaviors in the Logic Model (i.e. from

RSG Evaluation 2010-­2013

PEERassociates.net

9

a 2 or a 3 to a 4 or a 5 on the five-point Stage of Change scale). By the end of three years of program evaluation, it was possible to make solid statistical comparisons between the group of survey responses representing instances of passing the target threshold of Doing after at least one training session, i.e. greater than 4.0 on the five-point scale (N = 458), and those that had not (N = 2,042). The result was that the cases meeting the target averaged 2.0 trainings (SD = 1.4), whereas the cases not meeting the target averaged 1.7 trainings (SD = 1.1). The result was statistically significant (p < .001), meaning that the amount of REAL School Gardens training likely matters for achieving the target level of teacher effectiveness.

A secondary finding from this group comparison analysis was that many teachers reported at or above Doing or Deepening prior to receiving any training (37%, or 271 out of 729, for the More Effective Educators index). This suggests that there was a pre-existing capacity and/or interest in schools the program chose to partner with, i.e. that REAL School Gardens has, indeed, partnered with schools that are “fertile soil.”

•

Preliminary findings from year one of the evaluation were consistent with subsequent analyses of teacher effectiveness after three full years of data collection. During 2010-2011, the first year of survey data collection, options for testing the outcomes of the REAL School Gardens Program were more limited because there was scant data reflecting change over time. As an alternative approach, a “dose-response” analysis found that the amount of training a school had received positively and significantly predicted 11 of 14 outcome measures (ΔR2 = 0 to .09, p < .01, see Figures 8 and 9 in the Appendix for more details). Also, a matched pairs comparison of survey responses from after one or more training events to a baseline survey response (N = 64) yielded significant increases over time (see Figure 10 in the Appendix).

Even though these preliminary analyses were not specifically replicated in subsequent years of the evaluation, they strengthen the overall body of evidence about the outcomes of REAL School Gardens training by employing analytical triangulation, or different approaches converging on similar results.

RSG Evaluation 2010-­2013

PEERassociates.net

10

•

During interviews, all respondents spoke very highly of the REAL School Gardens Program. The first year of program evaluation included systematic analysis of transcripts from interviews with staff from 10 schools representing a range of levels of involvement with REAL School Gardens. The purpose of the interviews was to add context and depth to the data gathered from surveys. Direct expressions of overall satisfaction from staff at partner schools included statements such as “I think it’s a wonderful program that should continue and not lose its focus,” and “REAL School Gardens is an awesome organization that has truly fostered the love for learning.” There was also a theme in the qualitative data specifically appreciating the training component of the program. Representative statements in this regard included a teacher claiming that “The professional development they provide … really allows the teachers who do go to see how other people are using it and how we can integrate it into all other subject areas,” and a principal asserting that “I have no doubt in my mind that [without] REAL School Gardens … we would not be where we are now, the teacher education element is so important!” In sum, the survey and interview data findings largely reinforced each other.

“Since we've had the garden our overall [TAKS Science] score was like twenty-six percent passing and now we’re up to like eighty-seven percent, and it’s hard not to see the correlation between the rise in the data and having the garden.”

- Elementary Teacher, REAL School Gardens Partner School “Does it show up in a test score? I think it does, but it’s not going to be [directly attributable as] ... ‘because of the garden.’”

- Science/Math Specialist, REAL School Gardens Partner School

RSG Evaluation 2010-­2013

PEERassociates.net

11

Finding #2: Teachers unambiguously attributed student engagement in learning to the REAL School Gardens Program.

The ultimate purpose of the REAL School Gardens Program is to positively impact students, as reflected in the Logic Model (see Appendix). That said, this program is one relatively small intervention among the numerous other factors that influence student learning, attitude, and behavior over the course of their school and home lives. The organization’s Theory of Change (see Appendix) is consistent with educational research that describes engaged learners as an intermediate link between teacher practice and student achievement. This evaluation focused on engagement in learning rather than longer term measures of academic achievement that are influenced by a wider range factors in order to increase the likelihood of detecting effects more attributable to the program. Two of the four student engagement in learning measurement strategies described in more detail below generated clear evidence suggesting an association between the REAL School Gardens Program and high student engagement in learning. One measurement strategy did not yield sufficient variability to support or deny claims about student engagement in learning, and a final measurement strategy has not yet been implemented as of the writing of this report.

•

94% of teachers surveyed agreed with the statement “REAL School Gardens was a major contributor to increased engagement in learning for my students.” Starting in the spring of 2013, items were added to the Partner survey that asked teachers to report directly on the connection between REAL School Gardens and the level of engagement in learning they perceived in their students. The survey offered the following definition from the National Research Council: “Engaged students do more than attend or perform academically; they also put forth effort, persist, self-regulate their behavior toward goals, challenge themselves to exceed, and enjoy challenges and learning.”

Of the 924 responses collected to date, 313 (or 34%) “tend[ed] to agree” and 444 (or 60%) “strongly agree[d]” with the question of direct contribution reported above, whereas only 52 (or 6%) surveys reflected either the “strongly disagree” or “tend to disagree” response option. For additional survey questions about student engagement in various subject areas when the outdoor classroom was used for instruction, teachers reported the strongest student engagement for Science, with Math and Language Arts engagement only slightly less strong. Clearly, teachers saw the REAL School Gardens Program as helping to engage their students in learning.

RSG Evaluation 2010-­2013

PEERassociates.net

12

•

Positive student engagement in learning was a strong theme expressed in teacher interviews. Teachers were asked during interviews to specify which of the impacts in the REAL School Gardens Logic Model (i.e. academic achievement, healthy habits, pro-social skills and behaviors, connections to nature) that learning gardens had the biggest effect on. Rather than a consensus of one particular type impact being most prevalent, there were claims made about all of the impacts, usually in the form of stories of specific, often inspiring, cases of positive student results attributed to the program. The idea of engagement in learning was both an explicit and implicit theme weaving through teacher interviews. The following quotes were a few of the many claims expressed: o “Out in that garden they are able to connect it with something that’s going on in class, and they have a deeper understanding and appreciation or they can ... apply it in another situation.” (Math/Science Specialist) o “The garden has had just a phenomenal effect on them.” (Science Teacher) o “They are more alive, they are excited to learn, they are eager to learn. And since they’re excited and eager to learn they take ownership of it.” (Second Grade Teacher) o “[The garden] is central because we do reading, writing, and arithmetic out there. Every day we're collecting scientific data … They have a specific job to do when they go out to the garden… We get as complicated as the kids can handle…It's a very integrated curriculum with the math and science.” (Principal)

•

A pilot study of student engagement in learning based on student and teacher survey self-reporting methodology did not support hypothesized correlations. During 2011-2012, the second year of REAL School Gardens program evaluation, a substantial effort was made to devise additional tests of the relationship between the program and student engagement in learning. The design involved comparing change over time measures of specific teacher practices with student reports of their perceived engagement in several learning domains. The sample was large, representing retrospective post-test responses from a purposeful sample of 53 teachers and 931 matched pairs of responses from 2,299 of their 4th and 5th grade students. The measures of effective teaching practice did show sufficient variability and change from the fall

RSG Evaluation 2010-­2013

PEERassociates.net

13

to the spring (ΔX = +.8 on a five-point scale, p < .001, SD = .7), but student reports reflected a ceiling effect at both fall and spring measurement occasions (X = 3.5 on a four-point scale, SD = .5). Thus, no correlations were found. Either the specific intended teaching practices used in this pilot were not reliable predictors of student engagement, or the student self-report survey measures and/or the administration strategy used in this study were not sufficiently calibrated to describe variations in student engagement in learning. Or it could be that the high levels of student engagement with garden-oriented learning from the beginning mean that there is little room for improvement once the gardenoriented learning begins. Other programs in the Place-based Education Evaluation Collaborative (PEEC)6 have experimented with studies and instruments designed to reliably associate student engagement in learning with specific teacher practice, and have encountered similar methodological challenges.

•

A quasi-experimental study of student engagement in learning has been designed but not yet piloted. During the third year of REAL School Gardens Program evaluation, 2012-2013, work began on the next iteration of an attempt to find measures of student engagement that are more direct than teacher reports. This is part of the ongoing effort at REAL School Gardens (and also within the larger field of place-based education evaluation and research) to reliably apply concepts from Self-Determination Theory7 to the study of educational programs.

6

See http://www.peecworks.org

7 See http://www.selfdeterminationtheory.org/ for resources about the work spearheaded by Edward Deci and

Richard Ryan.

RSG Evaluation 2010-­2013

PEERassociates.net

14

Finding #3: Some evidence suggested potential school-­‐wide effects of the REAL School Gardens Program.

As the REAL School Gardens Logic Model and Theory of Change have become increasingly refined over the last three years, the primary focus on teacher practice and student impacts has come into sharper focus. Though the program works primarily with teachers, either individually or in groups, the possibility of effects spreading and taking root school-wide is implied. For instance, the Logic Model states a goal of 60% of teachers using the garden after three years, which suggests that school-wide results are important to consider. Also, there are dimensions of the REAL School Gardens partnership that directly involve the whole school, such as the Big Dig day to install the garden, whole school and cohort level training events, and the simple fact that the garden is available to and, presumably, a potential component of the identity of, everyone in the school.

Though not the central focus of either the program or the evaluation, there were peripheral pieces of evidence that emerged from the evaluation hinting that REAL School Gardens could be contributing to a larger cultural shift toward outdoor learning at partner schools. These are described below.

•

Figure 11.

Survey reports of school level context factors increased slightly during REAL School Gardens tenure. When the online surveys were first designed, one intended use was to assist program staff in the partner selection process.

To that end, questions about Culture, Climate, Leadership, Family Involvement, and Pedagogical Norms were included in the annual survey. Though it did not turn out to be particularly useful for the selection process, a School Level Outcomes Index, comprised of the average of the five topics described above, remained part of the survey analysis and reporting. Figure 11 shows upward sloping trend lines for each cohort, though not as strongly as for the More Effective Educators Index. The aforementioned dip in year three for cohort 10-11 is reflected in this figure as well, adding some support to the idea that school level factors can potentially affect (in this case, inhibit) teacher practice.

RSG Evaluation 2010-­2013

PEERassociates.net

15

•

The level of REAL School Gardens training a school received predicted positive school culture in the preliminary dose-response analysis. This finding was of secondary importance due to not being part of a larger pattern of year one findings, and due to the preliminary nature of the type of analyses conducted in the first year of the evaluation. Still, it does warrant passing mention as part of this collection of assorted evidence about potential school level effects. The size of this effect was statistically significant (p < .01) though not very large (ΔR2= .06).

•

Two sub-themes from interviews suggested evidence of school level dynamics. The interviews conducted during the first year of the evaluation were semistructured in format, and thus allowed for, even invited, respondents to share their own observations about the program. Nearly half of the interviewees made explicit mention of the large amount of “time and care” it takes to build and/or maintain a vibrant garden. Also, the topic of gardens as catalysts for parent and community involvement came up unsolicited in most interviews. Both of these sub-themes suggested that REAL School Gardens is not perceived as being only about teacher practice.

•

Teachers receiving no REAL School Gardens training reported a slight increase in effective teaching practices after year one of the program. Figure 6 in the Appendix shows the results of analyzing the subset of cases involving teachers who filled out the annual survey from year to year, despite not having participated in any of the intervening REAL School Gardens trainings. This analysis is also described in the second bullet under finding #1.

The presence of a slight bump up during the first year suggests a potential spread of effect from teachers receiving training to their colleagues who did not. It should be noted, however, that even if this did represent a genuine phenomenon, it was limited to the first year, since no further increases were observed in subsequent years.

•

Most schools averaged less than 60% of staff at Doing or Deepening.

The average percentage of respondents at a school who reported at the Doing (4.0) or Deepening (5.0) stage for the More Effective Educators index was 40% after one year, 45% after two years, and 44% after three years of REAL School Gardens programming (see Figure 12 in the Appendix). While these figures did not surpass the intentionally aspirational 60% target, they did represent substantial increases from the beginning of the program for most schools, often doubling or more the percentage of teachers initially reporting at the target level.

RSG Evaluation 2010-­2013

PEERassociates.net

16

Finding #4: REAL School Gardens staff honed the program model and its delivery based on evaluation outcomes.

This finding is qualitatively different from the preceding three. It was derived not from systematic, formal collection of participant and program data, but rather from overarching reflection and informal documentation of relevant conversations and processes by the evaluator. This finding is essentially a brief progress report on the commitment to a utilization-focused approach shared by both program staff and the evaluation team.

It is important to note that the observations below reflect results the evaluation work contributed to, which is not to claim that the evaluation was the only, or even most direct, cause.

•

The REAL School Gardens Program model evolved in meaningful ways over the course of three years of evaluation. Some examples of key changes made to the program along the way included moving from a 5-year to a 3-year partnership, and dropping the large, conference style professional development sessions in favor of focusing training on partner schools or other new school- or districtlevel configurations. Other examples included determining the number of teacher training sessions to provide each school, instituting cohort training sessions, and creating a “greenhouse” designation for schools that are potential partner school awaiting confirmed funding arrangements. Evaluation processes and data played some role in each of these decisions.

•

Data collection and reporting was substantially focused and streamlined. At the beginning of this program evaluation there were approximately a dozen different tools and templates for data collection, each focused on its own independent question and process. After three years, most data collection is now centralized via a couple of variations of online surveys that are explicitly and coherently tied to program theory. The pool of data is updated regularly and reported in a cloud-based “Reporting Book” that is structured to facilitate grant reporting and program staff reflection. More importantly, the

RSG Evaluation 2010-­2013

PEERassociates.net

17

purpose and use of data have become more tied to and embedded in strategic organizational decision processes. In the words of one program staff: “[This evaluation] took us from measuring outputs to measuring outcomes, and as a result we were able to demonstrate performance. As a result of demonstrating performance we were able to secure more funding, hire more staff, and achieve a broader, deeper impact on students and teachers.”

•

Funding was diversified and the program expanded, despite tough economic times. According to the Executive Director, “One of the key drivers of success over the last three years was developing a Logic Model, evaluating it through this multi-year program evaluation, and using it to clearly communicate our program’s outcomes and impacts.”

•

Program staff demonstrated unusually high appetite and aptitude for critical reflection and dealing with complicated data. Staff at all levels have been deeply involved in designing evaluation instruments, data collection protocols, reporting tools, and shared meaning making of data analyses. Throughout, program staff actively sought out disconfirming data to test their assumptions.

RSG Evaluation 2010-­2013

PEERassociates.net

18

Concluding Reflections Overall, evaluation evidence from 2010-2013 suggested that the REAL School Gardens Program has met most of its targeted outcomes. Program outcomes have become clearly specified, measures have been put in place to track progress toward them, and the resulting data has helped calibrate the amount of change that is reasonable to expect.

On average, the teachers who participated in REAL School Gardens trainings reported being ready to take on the challenges of using gardens for learning at the beginning of the partnership. Then, at some time before the end of the partnership, these teachers reported doing the effective teaching practices targeted by the program. Meanwhile, when gardens were used as a tool for instruction, students were engaged in learning.

Looking forward, since the evidence showing program effects at the level of individual teachers is solid and growing more so each year, the big picture opportunity is to start a more rigorous exploration of the ways in which the REAL School Gardens Program does and does not translate into measureable effects at the level of a whole school. This would likely invite strategic decisions about whether to invest more deeply in each school in order to move toward more of a “whole school change” model, or to remain content with primary effects on the subset of teachers at a school that are touched by REAL School Gardens programming, thus being able to reach more schools.

RSG Evaluation 2010-­2013

PEERassociates.net

19

Appendices Program Descriptions

Logic Model

Theory of Change

Figures

Figure 6. Paired samples analysis of outcomes for individuals with two or more Partner survey scores but WITHOUT ANY PD event scores, as of July 2013

Figure 8. Correlating Educator Outcomes with RSG Training, 2010-­‐2011

Figure 9. Summary of Data for 2010-­‐2011 RSG Partner Educator Surveys, Correlating RSG Dose and Implementation with RSG Logic Model Outcomes

Figure 10. Comparing Educator Stage of Change, Fall 2010 to Spring 2011, N = 64 Matched Pairs

Figure 12. Selection from “Reporting Book,” Oct 2013

Sample Instruments

Fall 2010 Interview Guide

PD Survey

RSG Evaluation 2010-­2013

PEERassociates.net

20

RSG Evaluation 2010-­2013

PEERassociates.net

21

RSG Evaluation 2010-­2013

PEERassociates.net

22

Figure 6.

Figure 8.

RSG Evaluation 2010-­2013

PEERassociates.net

23

Figure 9.

The ΔR2 statistic tells how much of the outcome can be predicted based on knowledge of the level of training received. Values of ΔR2 = .1 to .2 could be considered medium-sized effects in this context.

RSG Evaluation 2010-­2013

PEERassociates.net

24

Figure 10.

RSG Evaluation 2010-­2013

PEERassociates.net

25

Figure 12. Selection from “Reporting Book,” Oct 2013

RSG Evaluation 2010-­2013

PEERassociates.net

26

RSG Evaluation 2010-­2013

PEERassociates.net

27

RSG Evaluation 2010-­2013

PEERassociates.net

28

RSG Evaluation 2010-­2013

PEERassociates.net

29

RSG Evaluation 2010-­2013

PEERassociates.net

30