Camp Fire School Readiness & Early Education Apprenticeship Program Report 2022-2023

•

•

•

•

50

52

57

60 APPENDIX

61 APPENDIX

62 APPENDIX

63

64

65 APPENDIX J:

List of Figures

12 Figure 1: CFSRP Theory of Change

14 Figure 2: CFSRP Children Served, 20092010 (pilot) to 2022-2023

18 Figure 3: CFSRP Teacher Retention Rates from BOY to EOY, 2014-2015 to 2022-2023

19 Figure 4: CFSRP Student Retention Rates from BOY to EOY, 2015-2016 to 2022-2023

22 Figure 5: Percentage of Infants Demonstrating Age-Appropriate Skills by Assessment Area (ASQ ® -3), 2022-2023 (N=42)

22 Figure 6: Percentage of Toddlers Demonstrating Age-Appropriate Skills by Assessment Area (ASQ ® -3), 2022-2023 (N=105)

23 Figure 7: Percentage of 4- and 5-Year-Old Children Developmentally On-Target at BOY and EOY (CPALLS+), 2022-2023 (N=34)

24 Figure 8: Percentage of 4- and 5-Year-Old Children Improving or Making Acceptable Progress from BOY to EOY (CPALLS+), 2022-2023 (N=34)

25 Figure 9: Social Emotional Development at BOY and EOY (DECA), 2022-2023

27 Figure 10: CFSRP Classroom Environment and Management at EOY (CLASSTM), 2021-2022 and 2022-2023

28 Figure 11: Center Management Scores at BOY and EOY (PAS), 2022-2023 (N=15)

29 Figure 12: Comparison of Prekindergarten Early Literacy Skills (Fall 2023)

30 Figure 13: Comparison of Kindergarten Early Literacy Skills (TX-KEA, Fall 2023)

30 Figure 14: Comparison of Kindergarten Social Competence Skills (TX-KEA, Fall 2023)

33 Figure 15: Percent of Kindergarten – 2nd Grade Students Meeting Reading Growth ™ Targets Spring 2023

33 Figure 16: Percent of Kindergarten – 2nd Grade Students At or Above Average Reading Growth™ (MAP ® Reading Growth™, Spring 2023)

34 Figure 17: Percent of Students Meeting MAP Reading Growth Targets and Showing Average or Above Growth in Reading Skills

39 Figure 18: EEAP Theory of Change

List of Tables

13 Table 1: Description of CFSRP Professional Development Levels

13 Table 2: CFSRP Participating Centers, 2021-2023

14 Table 3: Number of Centers, Classrooms, Staff, and Children served by the Camp Fire School Readiness Program, 2009-2010 (pilot) to 2022-2023

15 Table 4: CFSRP Children Characteristics, 2022-2023 (N=822)

16 Table 5: CFSRP Staff Characteristics, 2022-2023 (N=196)

17 Table 6: CFSRP Mentor Characteristics, 2022-2023 (N=13)

20 Table 7: Mentor Activities, 2022-2023

21 Table 8: CFSRP Infant and Toddler Development Outcome Targets Versus Actual (ASQ ® -3), 2022-2023

23 Table 9: CFSRP Prekindergarten Development Outcome Goals versus Actual (CPALLS+), 2022-2023

26 Table 10: CLASS™ Domain Descriptions

31 Table 11: Comparison of Kindergarten Reading Fluency Skills (MAP ® Fluency™, Fall 2023)

31 Table 12: Comparison of Kindergarten Achievement (MAP ® Reading Growth™, Fall 2023)

32 Table 13: Percent of Students At or Above Average Reading Achievement Levels MAP ® Reading Growth™ (Spring 2023)

34 Table 14: 3rd Grade Reading Achievement: Longitudinal Cohorts

40 Table 15: EEAP Apprentice Characteristics (N=91), 2022-2023

41 Table 16: Apprentice Program Progression, by Cohort

41 Table 17: Apprentice Reasons for Leaving EEAP before Completion, 2022-2023

42 Table 18: Apprentice Self-Evaluation Results, 2022-2023

43 Table 19: Apprentice Appendix A Results, 2022-2023

Executive Summary

Background

Camp Fire First Texas has been committed to serving young children, their families, and the North Texas community for over 100 years. Two prominent avenues for strengthening early childhood services and supports that Camp Fire First Texas has pioneered are the Camp Fire School Readiness Program (CFSRP) and Camp Fire Early Education Apprenticeship Program (EEAP). Through these initiatives, children and educators access highquality programming and support. CFSRP aims to strengthen children’s school readiness (i.e., literacy, developmental, and social-emotional growth) through educator professional development and individualized mentoring. EEAP provides a career and educational pathway for early childhood educators by offering financial incentives, professional development, individualized incentives, and connection to higher educational institutions.

Report Structure

This report includes an evaluation of both CFSRP and EEAP. Each program is treated separately, allowing readers to examine the results of each program separately, or to consider their results jointly. Evaluation of each program addresses specific questions for the 2022-2023 school year:

School Readiness Program

To what extent did the CFSRP…

1. Implement professional development, stipend allocation, and mentorship activities as intended?

2. Improve the percentage of children demonstrating ageappropriate developmental, early literacy, and socialemotional skills during the 2022-2023 program year?

3. Impact children’s growth in developmental, early literacy, and social-emotional skills during the 20222023 program year?

4. Enhance the quality of teaching, classroom management, and centers’ family engagement practices during the 2022-2023 program year?

5. Impact CFSRP children’s school readiness as they enter prekindergarten and kindergarten and progress to the third grade?

Early Education Apprenticeship Program

To what extent did the EEAP…

1. Differences in participant retention and success become evident?

2. Components collectively support educators’ knowledge and skills?

3. Participants attain (and maintain) portable child care credentials and wage increases?

CFSRP Results

PROGRAM IMPLEMENTATION

Process evaluation describes the extent to which the CFSRP program implemented program activities as intended and how those activities resulted in the expected program outcomes. Findings from a process evaluation enable Camp Fire to determine which aspects of the program are working as expected, and which processes require additional support and clarification to be most effective. Data from the 2022-2023 program year indicates a high teacher retention rate of 79%, but a decreased student retention rate of 67% for the program year.

Child Outcomes (ASQ®-3, CPALLS+)

Data from the ASQ-3 screening tool show infants in CFSRP classrooms demonstrating developmental progress in three of the five domains (communication, gross motor and problem solving) from beginning to end of year but showing less developmental progress in fine motor and personal-social skills. Similarly, toddlers demonstrated developmental progress in communication, personal-social skills and problem-solving skills, but showed a decrease in gross and fine motor skills from beginning to end of year.

Preschool CPALLS+ results show 4- and 5-year-old children making acceptable progress from beginning to end of year across all domains, with Rhyming and Rhyming 2 increasing by 24% and 23%, respectively. Although children showed the lowest growth in math skills (9%), the end of year percentage of children making acceptable progress was an impressive 97%.

Social emotional development measures show that the number of infants showing need decreased over the program year, indicating that appropriate supports were offered to address developmental needs. Toddler children show a slight increase in need during the program year and preschool children show little change. From beginning to end of year, 46% of infants, 41% of toddlers, and 49% of prekindergarten students showed improvement in their DECA score.

Center Outcomes (CLASS™, PAS)

At the end of year 2022-2023, CLASS™ scores for infant classrooms showed higher average scores than seen in 2021-2022 (+0.7 points), while toddler classrooms showed similar end-of-year results across both years. CLASS™ scores for pre-K classrooms showed increased averages in emotional support compared to the prior year but showed lower average scores in classroom organization and instructional support. National quality thresholds are defined for preschool classrooms, and Camp Fire goals include meeting those thresholds. At the end of the year, average preschool CLASS™ scores met the quality threshold for classroom organization but fell short in emotional and instructional support.

PAS results show CFSRP-supported centers exhibited mixed results within all three assessment areas, with the greatest improvements in program evaluation. Small improvements were also made in strategic planning and family communication. Slight decreases were seen in screening and identifying special needs, assessment in support of learning, and family support and involvement.

School Readiness and Academic Progress

Using results from FWISD assessments already in place, the evaluation examines student performance comparing cohorts of CFSRP children currently in grades prekindergarten through third grade to demographically similar groups of children who did not attend a CFSRP center.

As in prior years, results indicate that the CFSRP continues to contribute to prekindergarten and kindergarten students’ early literacy skills, with mixed results relative to academic progress in first, second and third grades.

• Prekindergarten CFSRP students had higher ratings than the matched comparison group on the Letter Naming, Vocabulary and Listening Skills measures.

• Kindergarten students had higher ratings on all the TX-KEA Early Literacy measures as well as on the MAP Reading Fluency Decoding, Listening Comprehension and Overall Achievement measures.

• At the end of 2023, higher percentages of kindergarten CFSRP students met their MAP Reading Growth Targets, but the results were similar for both groups of first grade students, and, in third grade higher percentages of the comparison group met their MAP Reading Growth Targets.

EEAP Results

PARTICIPANT RETENTION AND SUCCESS

EEAP is a two-year program that admits new apprentices every year. Of the 91 apprentices served in 2022-2023, 77 (85%) were new apprentices and 14 (15%) were returning for their 2nd year of enrollment in the program. By August 2023, 34 of the 91 apprentices served graduated from the program (37%); 29 are continuing into their 2nd year of enrollment (32%); and 28 (31%) cancelled their participation in the program.

KNOWLEDGE AND SKILL DEVELOPMENT

The EEAP program continues to focus on increasing the knowledge and skill level of participants. Among the 20222023 apprentices, 35 had completed Child Development Associate Credentials (CDA) (38%), of which, 14 completed their CDA before program enrollment and 19 completed their CDA during the program. Additionally, 56 completed EEI*.

Glossary of Terms

APPENDIX A Assessment tool created by the National Association for the Education of Young Children (NAEYC) to evaluate early childhood educators core competencies in six standards representing the core body of knowledge, skills, values, and dispositions early childhood educators must demonstrate to effectively promote the development, learning and well-being of all young children.

ASQ®-3 Ages and Stages Questionnaires®, Third Edition

A developmental screening tool for use with children ages one month to 5½ years old. The ASQ®-3 highlights a child’s developmental strengths and areas of concern. Camp Fire uses ASQ®-3 to see where individual children need additional support, assess development over time, and identify developmental domains to emphasize in the classroom.

BOY Beginning of Year

Generally refers to measurements taken at the beginning of a school year (September through November).

BPOT Best Practices Observation Tool

A research-based checklist for classroom observers to measure the presence or absence of teaching practices that align with the CFSRP professional development curriculum. Checklists are tailored to classroom type (infant, toddler, Pre-K3 and Pre-K4).

CFSRP Camp Fire School Readiness Program

Camp Fire First Texas’ research-based initiative to strengthen school readiness through improving language and socialemotional skills among children age 0-5 and enhancing the quality of teaching and childcare centers.

CIRCLE Circle Progress Monitoring Tool

A criterion-referenced assessment used to identify prekindergarten students who are on track, need monitoring, or need support for development of early literacy and social emotional skills.

CLASS™ Classroom Assessment Scoring System

A quality improvement tool focused on teacher-student interactions that measures the teaching quality within a classroom. Camp Fire uses CLASS™ to gauge classroom quality and provide individualized feedback to improve educator practices.

CPALLS+ An assessment of prekindergarten learning (literacy, math, science, social studies), social development, and critical Head Start skills. Camp Fire uses CPALLS+ to assess older children’s language and math skill development over time.

DECA Devereux Early Childhood Assessment

A strengths-based social-emotional screening and assessment tool for infants and toddlers (ages one month to 36 months) and preschoolers (ages three to five years). Camp Fire uses DECA to assess children’s social-emotional wellbeing and level of protective factors.

Glossary of Terms Introduction

EEAP Camp Fire Early Education Apprenticeship Program

Camp Fire First Texas’ workforce development initiative to build a career pathway for early childhood educators by combining paid, on-the-job learning with coaching, professional development, and professional certification.

EEI Early Educators Institute

The Early Educator Institute (EEI) offers biweekly 16-week sessions in Infant/Toddler and Pre-K tracks that explore early childhood topics supporting theoretical knowledge and practical application of developmentally appropriate practices in early learning spaces.

EOY End of year

Generally refers to measurements taken at the end of a school year (April through June).

FWISD Fort Worth Independent School District

The K-12 public school system that serves most students after they age out of the CFSRP.

MAP® Reading Fluency™

An online screening and progress monitoring tool that assesses basic reading skills with an emphasis on oral fluency (e.g., listening comprehension, words per minute, accuracy, decoding). It is part of a standardized, normreferenced series of assessments that can be used to measure students’ performance against the performance of a national sample. Students receive a MAP® Reading Fluency™ rating of below, approaching, meeting, or exceeding grade level expectations.

MAP® Growth™

An online screening and progress monitoring tool that assesses early literacy skills, reading comprehension, and use of vocabulary. It is part of a standardized, norm-referenced series of assessments that can be used to measure students’ performance against the performance of a national sample. Students receive a score that places them at one of five levels (low, low average, average high average or high), based on the national sample.

MOY Middle of year

Generally refers to measurements taken in the middle of a school year (January or February).

PAS Program Administration Scale

A leadership and program management assessment tool designed for center-based early childhood education programs. Camp Fire uses the PAS to assess center quality and support continuous improvement.

TX-KEA Texas Kindergarten Entry Assessment

An assessment of entering Texas kindergarten students’ language, literacy, STEM, social emotional, executive function, and academic motor skill. Kindergarten teachers in Texas administer the TX-KEA to their students at the beginning, middle, and end of the school year.

Camp Fire First Texas’ commitment to youth development begins right from the start, with early childhood. By strengthening the systems, programs, and professionals that support infants and young children, Camp Fire invests in strong starts for children across Fort Worth and beyond.

Since 2005, the Camp Fire School Readiness Program (CFSRP) has helped children be ready to start school through improving quality of care at neighborhood childcare centers. CFSRP supports early education programs that feed into the Fort Worth Independent School District with professional development and individual coaching/mentoring focused on teaching practices that improve language and social-emotional skills among children age 0-5.

In 2020, Camp Fire began its Early Education Apprenticeship Program (EEAP), a 1–5-year program combining paid on-the-job learning, coaching, professional development, and professional certification to strengthen career and educational pathways for early childhood professionals. EEAP is an innovative approach to a critical need: stabilizing the workforce through improved wages and decreased turnover. Two additional occupations were added in the Fall of 2023 –ECE II (Associate of Applied Science in Child Development) and ECE III (Bachelor of Applied Science in Child Development and Family Studies). The first U.S. Department of Labor-certified early childhood apprenticeship program in the state of Texas, EEAP is supported by the Texas Workforce Commission, Workforce Solutions for Tarrant County, Tarrant County College, Tarleton State University, North Central Texas College, San Antonio College, and the T.E.A.C.H. Early Childhood Scholarship through the Texas Association for the Education of Young Children, as well as generous local foundations.

Each year, Camp Fire prepares an evaluation report to share with partners and the community. The evaluation provides opportunities to reflect on challenges and successes in the preceding year, adjust to more effectively support children’s education, and have transparent, productive conversations across the community. The report separately addresses the activities and achievements of CFSRP and EEAP, enabling readers to consider each program independently and jointly.

CFSRP

CAMP FIRE SCHOOL READINESS PROGRAM

The CFSRP evaluation summarizes literacy, developmental, and social-emotional growth among children attending childcare centers supported by CFSRP in 2022-2023, center and classroom quality measures, and academic outcomes through third grade. It addresses five central evaluation questions. To what extent did the CFSRP:

1. Implement professional development, stipend allocation, and mentorship activities as intended?

2. Improve the percentage of children demonstrating ageappropriate developmental, early literacy, and socialemotional skills during the 2022-2023 program year?

3. Impact children’s growth in developmental, early literacy, and social-emotional skills during the 2022-2023 program year?

4. Enhance the quality of teaching, classroom management, and centers’ family engagement practices during the 2022-2023 program year?

5. Impact CFSRP children’s school readiness as they enter prekindergarten and kindergarten and progress to the third grade?

EEAP

EARLY EDUCATION APPRENTICESHIP PROGRAM

The EEAP evaluation summarizes participation and outcomes for new and returning apprentices in the third-year of the EEAP program since EEAP was instituted in 2020. Central evaluation questions for EEAP include:

1. What differences in participant retention and success are evident?

2. How did EEAP components collectively support educators’ knowledge and skills?

3. To what extent are EEAP participants attaining (and maintaining) portable child care credentials and wage increases?

Camp Fire School Readiness Program

Healthy development in early childhood supports lifelong health, academic achievement, economic stability, and community strength. High-quality early childhood care and education are foundational for social, cognitive, and developmental growth that lead to kindergarten readiness.1 Nowhere else are the benefits of high-quality early care as pronounced as among children from low-income households, who are at increased risk of entering kindergarten less prepared than their peers,2 with lasting disadvantages,3 but with strong potential to close gaps when supported in the first years of life.4 The CFSRP addresses this need by partnering with childcare centers in targeted areas of Fort Worth to support educators with the knowledge and skills to provide quality care to neighborhood children. This solution meets families and neighborhoods where they are, enhancing quality care without displacing current providers or developing new childcare centers.

The CFSRP measures its success in terms of child learning and development, classroom quality, and center quality at the beginning and end of each school year, as shown in the CFSRP theory of change5 (Figure 1). The components of the program are outlined in detail in Appendix A. To understand how and why these outcomes came about, the evaluation includes program implementation indicators: teacher and student retention, teacher participation in professional development, and mentoring activities.

Camp Fire School Readiness Program

Program Characteristics

PROGRAM TIERS

The CFSRP is designed to operate over multiple years, bringing center partners in to first gain familiarity with program components (Level 2), then commit to intensive professional development and mentoring (Level 3), and then receive ongoing professional development support to maintain quality over time (Level 4). In 2022-2023, CFSRP matured to the point of having over half of supported centers and classrooms at the sustainability level and the capacity to bring on new centers for support (Table 1).

1 Snow, K. L. (2006). Measuring school readiness: conceptual and practical considerations. Early Educ. Dev. 17, 7–41. doi: 10.1207/s15566935eed1701_2

2 Reardon, S. F. & Portilla, X. A. (2016). Recent trends in income, racial, and ethic school readiness gaps at kindergarten entry. AERA Open 2(3), 1-18. doi: 10.1177/2332858416657343

3 Garcia, E. & Weiss, E. (2017). Education inequalities at the school starting gate. Economic Policy Institute. epi.org/132500

4 Magnuson, K.A., Meyers, M.K., Ruhm, C.J., & Waldfogel, J. (2004). Inequality in preschool education and school readiness. American Educational Research Journal. 41(1):115-157. doi:10.3102/00028312041001115

5 A theory of change provides an illustration of a program’s impact pathway—the logical causal change that is expected to occur as a result of program activities.

Center participation begins after the third, intensity-level year and continues as long as the center remains in the program

Table 1: Description of CFSRP Professional Development Levels

Figure 1: CFSRP Theory of Change

CENTER CHARACTERISTICS

In the 2022-2023 school year, the CFSRP served 15 child development centers, reaching 196 staff members and 822 children in targeted areas of Fort Worth. Most participating centers were engaged in CFSRP Level 4 (53%). Of the 15 child development centers, three participated in Level 2 programming, four in Level 3 intensive support, and eight in Level 4 sustainability work. Eleven centers (73%) also participated in Texas Rising Star (TRS) during the 2022-2023 school year; each of those centers rated either three or four stars (Table 2). Texas Rising Star is the quality rating and improvement system for Texas early childhood programs. All program receiving child care subsidy payments are required to participate in the TRS program.

Currently, CFSRP exclusively serves center-based providers, helping them to enhance the quality of the education they provide, strengthen classroom management and teacher-child interactions, and advance their Texas quality rating and improvement rating through TRS. The number of classrooms served in a year provides a strong indicator of CFSRP investment of time and money. The classrooms participating in the CFSRP in 2022-2023 were primarily single age group classrooms (84%). The number of classrooms participating in CFSRP in 2022-2023 increased somewhat, as three additional centers were supported, increasing the number of classrooms served. A trend from 2020-2021 and 2021-2022 showed declines in classrooms served, attributed to center closures due to teacher shortages and persistent low enrollment associated with the COVID-19 pandemic, as well as to shifting partner priorities. Seeing an increase in center support and classrooms participating show results trending back toward pre-pandemic counts.

CHILDREN SERVED Figure 2: CFSRP Children Served, 2009-2010 (pilot) to 2022-2023.

Table 2: CFSRP Participating

2022-2023

Table 3: Number of Centers, Classrooms, Staff, and Children served by the Camp Fire School Readiness Program, 2009-2010 (pilot) to 2022-2023.

CHILDREN CHARACTERISTICS

Center recruitment to CFSRP is designed to engage early education programs in targeted areas served by Fort Worth ISD. The CFSRP served a diverse set of 822 young children in the 2022-2023 school year (Table 4). Children are predominantly African American (58%), Caucasian (20%), or Hispanic/ Latino (15%), in keeping with prior years. Ages served are well balanced from infancy (27%) to prekindergarten (13%), with toddlers representing the greatest share (36%). The number of prekindergarten children continues to decline (down 8% since 2020-2021), most likely because of the number of options for free prekindergarten education provided through local school districts, including Fort Worth ISD.

MENTOR CHARACTERISTICS

Camp Fire assigns experienced mentors to support teachers in the program. These mentors provide teachers with guidance on best practices in the classroom, how to handle challenging situations, and how to support child development and provide high quality care in ageappropriate ways. Camp Fire had 13 mentors in 2022-2023, an increase to accommodate mentorship requirements with Camp Fire’s Early Education Apprenticeship Program. 62% of mentors had a master’s degree. Eight were African American, two were Caucasian, and three were either Hispanic/Latino, multi-racial, or other. Three had more than 10 years of childcare experience, and four had between seven and 10 years of experience. Additionally, in 2022-2023 there were two mentors with 7-10 years of experience with CFSRP (see Table 6).

Race/Ethnicity

Years of Childcare Experience

STAFF CHARACTERISTICS

The CFSRP engages a variety of early childhood professionals—including center directors, lead teachers, assistant float classroom teachers, and others—in professional development and support to enhance quality of care and education. In approximate alignment with student demographics, 51% of staff supported by the CFSRP in 2022-2023 are African American, 15% are Caucasian, and 18% are Hispanic/Latino (Table 5); 88% of staff are female.

In general, early childhood staff in childcare centers have less formal training and education than those affiliated with school districts. Current Texas Child Care Regulation requires that child care providers have a High School diploma or General Education Development Test (GED) to work in child care settings. (https://public.tecpds.org/about-us/state-of-theece-workforce-report/2022-report/) Staff supported by the CFSRP follow this trend: most have less than an associate degree: 44% have a high school diploma or GED as their highest educational attainment, and 19% have some college but less than an associate degree. 19% have some type of childcare credential, generally a CDA.

Years of experience in the field of childcare varied across staff, with 40% of staff having three years of experience or less and 20% having ten years or more. This division is generally consistent with prior years, with the most difference seen in the percentage of staff having ten years of more experience (32% in 2021-2022).

Table 4. CFSRP Children Characteristics, 2022-2023 (N=822)

Table 5. CFSRP Staff Characteristics, 2022-2023 (N=196)

Table 6. CFSRP Mentor Characteristics, 2022-2023 (N=13)

Program Implementation6

Process evaluation describes the extent to which the CFSRP program implemented program activities as intended and how those activities resulted in the expected program outcomes. Findings from a process evaluation enable Camp Fire to determine which aspects of the program are working as expected, and which processes require additional support and clarification to be most effective.

TEACHER RETENTION BEGINNING OF YEAR (BOY) TO END OF YEAR (EOY)

Low teacher retention is a persistent challenge to providing high-quality early childhood care and education. Turnover adds program expenses, increases staff burdens, and can harm child development.7 Children benefit from having a secure attachment to caregivers;8 thus, teacher turnover within a school year has been negatively associated with children’s language, literacy, and social-emotional development, while teacher retention is linked to better outcomes for children.9,10,11 Further, teacher turnover is highest among the types of centers CFSRP serves: centers that are not school district-sponsored or otherwise publicly funded.12

Teacher turnover rates in all areas nationwide have risen in recent years and Texas has experienced one of the highest turnover rates in early childhood education in the nation at 20%.13 Low wages are cited as the primary reason for high turnover, with challenging working conditions and lack of administrative support as additional factors. In many cases, early childhood educators can earn more money in an unrelated field, increasing the challenge of hiring staff.

In the context of state and national teacher retention trends, CFSRP teachers showed above-average employment stability. Of the 136 CFSRP teachers employed during the 2022-2023 school year, 108 remained at their childcare center from beginning to end of year, for a 79% retention rate (see Figure 3). 13 Teacher retention rates for the program are typical for the CFSRP in recent years and exceed those expected industry-wide in regular years as well as during the pandemic.

Upon further exploration into teacher retention rates during 2022-2023, CFSRP teachers showed higher retention rates among teachers with more years of childcare experience; 65% for teachers with less than 1 year of childcare experience, 78% for teachers with between 1-6 years, and 100% for teachers with 7+ years of childcare experience. Similarly, higher teacher retention rates were seen for teachers at centers and classrooms more developed through CFSRP support; 54% at Level 2 centers, 91% at Level 3 centers, and 87% at Level 4 centers. According to the end-of-year survey distributed to teachers, survey participants reported that their biggest motivators for staying with the center were relationships with children and families (74%) and relationships with staff (48%).

6 A description of the methods used for this report are provided in Appendix B.

7 Buffett Early Childhood Institute at the University of Nebraska. (2018). Early childhood teacher turnover in Nebraska. https://buffettinstitute.nebraska.edu/-/ media/beci/docs/early-childhood-teacher-turnover-in-nebraska-new.pdf8 Garcia, E. & Weiss, E. (2017). Education inequalities at the school starting gate. Economic Policy Institute. epi.org/132500

8 Folbre, N. (2012). For Love or Money. New York: Russell Sage Foundation.

9 Markowitz, A. J. (2019). Within-year teacher turnover in Head Start and children’s school readiness. EdPolicyWorks, University of Virginia. https://curry.virginia. edu/sites/default/files/uploads/epw/70_Teacher_Turnover_in_Head_Start.pdf

10 Hamre, B., Hatfield, B., Pianta, R., & Jamil, F. (2014). Evidence for general and domain-specific elements of teacher-child interactions: Associations with preschool children’s development. Child Development, 85(3), 1257-1274.

11 Hale-Jinks, C., Knopf, H., & Kemple, K. (2006). Tackling teacher turnover in childcare: Understanding causes and consequences, identifying solutions. Childhood Education, 82(4), 219–226.

12 Grunewald, R., Nunn, R., Palmer, V. (2022). Examining teacher turnover in early care and education. Federal Reserve Bank of Minneapolis. https://www. minneapolisfed.org/article/2022/examining-teacher-turnover-in-early-care-and-education

13 Retention rates during the 2019-2020 school year were artificially high due to centers keeping teachers on staff, in hiatus, during the coronavirus pandemic. Not all teachers on the staff lists were not actually present in childcare centers. This unique situation led to an artificially high retention rate on paper in spite of having fewer children to serve in spring 2020. Delohery, Caitlin. Turnover Rate for Early Childhood Education Jobs in Texas: Overview, texestest.org, 2024

14 Sabol, T.J. & Pianta, R.C. (2012). Recent trends in research on teacher-child relationships. Attachment & Human Development 14(3). doi: 10.1080/14616734.2012.672262

15 Curby, T.W., Grimm, K.J., & Pianta, R.C. (2010). Stability and change in early childhood classroom interactions during the first two hours of a day. Early Childhood Research Quarterly 25(3): 373-384. doi: 10.1016/j.ecresq.2010.02.004

CHILD RETENTION (BOY TO EOY)

Continuity of care—in terms of having a consistent learning environment as well as maintaining healthy relationships with a consistent adult—is a critical element of early childhood learning and development. 14,15 Additionally, because Camp Fire measures developmental change from beginning of year to end of year, stable class enrollment is imperative to understanding child and program outcomes. From fall 2022 to spring 2023, CFSRP centers retained 549 of 822 children for 67% retention, a lower rate than in recent years (Figure 4).

Upon further exploration into child retention rates during 2022-2023, CFSRP children trended from highest to lowest retention among toddlers (73%), 3-year-olds (67%), infants (64%), and lastly, preschoolers (55%). The lowest retention among preschoolers is expected since 4-year-olds are eligible for free, full-day prekindergarten programs at FWISD locations. Like teacher retention rates, higher child retention rates were seen for children at centers and classrooms more developed through CFSRP support; 55% at Level 2 centers, 79% at Level 3 centers, and 67% at Level 4 centers.

Figure 3: CFSRP Teacher Retention Rates from BOY to EOY, 2014-2015 to 2022-2023

Figure 4: CFSRP Student Retention Rates from BOY to EOY, 2015-2016 to 2022-2023

TEACHER AND DIRECTOR PROFESSIONAL DEVELOPMENT PARTICIPATION

A key element of the CFSRP is professional development for center teachers and directors. Teachers can increase their knowledge and skills related to classroom management and child development through foundational professional development and the Early Education Institute (EEI) courses the CFSRP provides. Directors and Assistant Directors can increase their knowledge of leadership practices, business management, and child development through the CFSRP Director’s Institute (DI). In 2022-2023, professional development was offered both virtually and in person. Of 196 staff people participating in the CFSRP, 178 (91%) participated in EEI and/or foundational professional development. Professional development participation for staff across center level shows the highest percentage of participation at Level 4 centers (91%); 53% at Level 2 centers and 85% at Level 3 centers.

TEACHER AND DIRECTOR STIPENDS

In the 2022-2023 school year, within CFSRP, directors of Level 4 centers and mentor teachers were eligible to receive stipends of $500 each at the end of the school year, based on completion of agreed upon leadership and mentoring duties. A total of 45 CFSRP mentor teachers and directors received stipends in 2022-2023 (87% of eligible educators).

ONE-ON-ONE MENTOR ACTIVITY

CFSRP mentors typically provide on-site, individualized coaching to teachers and directors. Mentors help integrate assessment results into teacher action plans, improve the use of best practices in teaching and classroom management, and coach teachers and directors on how to provide quality education.

One-on-one mentoring activities in the 2022-2023 school year continued to utilize strategies developed during the pandemic for virtual interaction, such as introducing the use of Bluetooth and tablets to provide real-time coaching in addition to side-by-side work. Table 7 summarizes the CFSRP’s key mentoring activities and the number of visits in which mentors focused on each activity. Consistent with 2021-2022, mentors were especially supportive of teacher reflective follow-up on situations encountered in the classroom with classroom observation. Trending over the last couple of years, the total number of visit activities continues to increase (93% more visit activities performed in 2022-2023 compared to 2021-2022) as mentors had greater ability to conduct visits in person and additional mentors were onboarded for CFSRP.

*Activity only relevant to director

Table 7: Mentor Activities, 2022-2023

Child Outcomes16

The CFSRP measures its success in helping children attain age-appropriate developmental, early literacy, and social-emotional skills through consistent use of validated assessments. For younger children, key outcomes are developmental: gross and fine motor skills, communication, problem solving, and personal-social skills. For prekindergarten students, key outcomes are academic and developmental: literacy, numeracy, and social-emotional skills. For all children, social-emotional outcomes are an additional outcome and dimension of childhood development that supports school readiness. Descriptions of the assessments used to measure child outcomes are available in Appendix C.

Each year, to determine the program’s level of success in each of these domains, CFSRP staff set targets for the percent of children developmentally on-target at the end of the year. This evaluation assesses the extent to which these targets are met and how much individual children maintain and/or improve their developmental skills.

INFANTS AND TODDLERS: CHILD DEVELOPMENT

Center teachers administered the Ages and Stages Questionnaire (ASQ®-3) to infants and toddlers ages 0-3 at the beginning and end of the program year to screen for potential developmental concerns. Developmental concerns are identified when a child scores below the cut-off in at least one of the ASQ®-3’s five developmental domains (i.e., problem solving, communication, fine motor, gross motor, and personal-social skills). When developmental concerns are identified, center staff use the screening results to implement individualized instruction in their classrooms and refer families to external support services if the concerns are more severe.

In the aggregate, infants in CFSRP classrooms demonstrated developmental progress in three of the five domains (communication, gross motor and problem solving) from beginning to end of year but showed less developmental progress in fine motor and personal-social skills (Figure 5). Similarly, toddlers demonstrated developmental progress in communication, personal-social skills and problem-solving skills, but showed a decrease in gross and fine motor skills from beginning to end of year (Figure 6).

Because the ASQ is used as a screening tool for developmental concerns, it is designed to be used in six-month increments. Teachers use the tool that is closest to the child’s current age, so not all children in an infant or toddler classroom receive the same screening. End-of-year targets were met in only one category – gross motor skills for infants (Figure 8).

*Shaded end of year figures met or exceeded the year’s target.

Table 8: CFSRP Infant and Toddler Development Outcome Targets Versus Actual (ASQ®-3), 2022-2023

Figure 5: Percentage of Infants Demonstrating Age-Appropriate Skills by Assessment Area (ASQ®-3), 2022-2023 (N=42)

Figure 6: Percentage of Toddlers Demonstrating Age-Appropriate Skills by Assessment Area (ASQ®-3), 2022-2023 (N=105)

PREKINDERGARTEN CHILDREN: MATH AND LITERACY

Center teachers administered the CPALLS+ assessment to children ages three-and-a-half through five to assess their listening skills, ability to determine if two words rhyme (rhyming 1), ability to independently present a word that rhymes with a given word (rhyming 2), and early math skills. Rhyming 2 is one of the most challenging skills to master, and many children may not master Rhyming 2 skills until they turn five or six years old. For the purposes of the evaluation, the results for four- and five-year-old children (i.e., prekindergarten) are considered.

The increase in the percentage of 4- and 5-year-old children making acceptable progress from beginning to end of year was significant across all domains, with Rhyming and Rhyming 2 increasing by 24% and 23%, respectively. Although children showed the lowest growth in math skills (9%), the end of year percentage of children making acceptable progress was an impressive 97% (Figures 7 and 8).

Figure 7: Percentage of 4- and 5-YearOld Children Developmentally OnTarget

ALL AGES: SOCIAL-EMOTIONAL DEVELOPMENT

Center teachers administered the Devereux Early Childhood Assessment (DECA) to infant, toddler, and prekindergarten children at CFSRP-supported centers participating at Level 3 and 4.17 The DECA identifies whether children’s social-emotional skills need intervention (need), are within typical range for their age (typical) or exceed the typical range (strength).

The percentage of infants and toddlers in the “strengths” category decreased (infants) or held steady (toddlers) from beginning to end of year, but the number of infants showing typical development increased and the number of toddlers showing typical development decreased only slightly. Percentages for preschool children remained relatively steady, with a small decrease of strength from beginning to end of year and a small increase of typical growth during the same period.

The number of infants showing need decreased over the program year, indicating that appropriate supports were offered to address developmental needs. Toddler children show a slight increase in need during the program year and preschool children show little change (Figure 9). From the beginning to end of year, 46% of infants, 41% of toddlers, and 49% of prekindergarten students showed improvement in their DECA score.

Classroom and Center Outcomes

CLASSROOM ENVIRONMENT QUALITY

The Classroom Assessment Scoring System (CLASS™) is a widely used measure of the quality of teacher-child interactions in center-based early childhood programs. CLASS observations were conducted by certified Camp Fire staff at the end of the program year in a portion of infant, toddler, and PreK classrooms.18 Different domains are captured based on the developmental stage of the children in the classroom. In infant classrooms, CLASS™ measures teacher ability to respond to and interact with infants during play, routine care, and other activities (responsive caregiving). In toddler classrooms, it measures teachers’ ability to promote intentional, prosocial interactions that encourage children’s capacity for self-regulation and social skills (emotional-behavioral support) and teachers’ ability to promote emerging, expressive language skills in children (engaged support for learning). In preschool classrooms, it measures teacher ability to foster positive relationships and respond to children’s emotions or interests (emotional support); set clear behavioral guidelines and maintain a classroom that supports children’s interactions with teachers and peers through the effective management of children’s time, behavior, and attention (classroom organization); and help children learn to solve problems, develop more complex language skills, and use feedback to deepen children’s skills and knowledge (instructional support). Empirical studies have shown positive outcomes for children in classrooms with high CLASS™ assessment ratings.19, 20 Studies have also provided evidence of a threshold effect indicating a minimal level at which classroom quality in preschool classrooms is met to achieve positive student outcomes.21, 22 A description of the teacher-student interaction domains CLASS™ captures and associated quality thresholds is available in Table 10.

In 2022-2023, Camp Fire relied on third party providers to participate in the Texas Rising Star (TRS) program recommendation that quality early childhood centers participate in CLASS assessments. They discovered that some centers did not choose to follow the TRS recommendations and did not welcome CLASS observers into centers for classroom assessment. Eight of the 15 centers participating in the CFSRP completed CLASS™ assessments at the end of year. Barriers to completion on schedule included teacher turnover, which delayed training, lack of assessments completed by TRS CLASS observers and administration timelines.23

CLASS™ provides ratings that place the quality of teacher-child interactions in a high (5.6 or above), mid (3.0 to 5.5) or low range. Camp Fire thresholds include scoring in the mid-quality range or higher in all domains. It expects an overall increase from beginning to end of year in each CLASS™ domain for infant, toddler, and preschool classrooms. Due to limited assessments results in the beginning of year 2022-2023, the end of year results for 2022-2023 will be directly compared with end of year results in 2021-2022 (Figure 10).

At the end of year 2022-2023, CLASS™ scores for infant classrooms showed higher average scores than seen in 2021-2022 (+0.7 points), while toddler classrooms showed similar end-of-year results across both years. CLASS™ scores for pre-K classrooms showed increased averages in emotional support compared to the prior year but showed lower average scores in classroom organization and instructional support. National quality thresholds are defined for preschool classrooms, and Camp Fire goals include meeting those thresholds. At the end of the year, average preschool CLASS™ scores met the quality threshold for classroom organization but fell short in emotional support and instructional support (Figure 11).

The small number of classrooms observed makes it difficult to draw significant conclusions on overall classroom quality. Camp Fire will restructure the CLASS observation process for the 2023-2024 program year.

18 In 2022 the decision was made to conduct CLASS observations only at EOY rather than BOY and EOY.

19 Carr, R., Mokrova, I., Vernon-Feagans, Burchinal, M. (2019). Cumulative classroom quality during prekindergarten and kindergarten and children’s language, literacy, and mathematics skills. Early Childhood Research Quarterly, 47, 218-228.

20 Vitiello, V. E., Bassok, D., Hamre, B. K., Player, D., & Williford, A. P. (2018). Measuring the quality of teacher–child interactions at scale: Comparing researchbased and state observation approaches. Early Childhood Research Quarterly, 44(3), 161-169.

21 Burchinal, M., Vandergrift, N., Pianta, R., & Mashburn, A. (2010). Threshold analysis of association between childcare quality and child outcomes for low-income children in prekindergarten programs. Early Childhood Research Quarterly, 25, 166-176.

22 For Pre-K CLASSTM, the quality threshold is set at 5 for the Emotional Support and Classroom Organization domains, and at 3.25 for the Instructional Support domain. For infants and toddlers, scores of 3 to 5 indicate a mix of effective teacher-child interactions, while scores of 6 to 7 indicate consistently effective teacher-child interactions.

23 The centers that were not able to complete CLASSTM assessments at the end of year in 2022-2023 include Amazing Creations, Faith Academy Learning Center, International Kids Academy, Kiddyland Childcare, Mother Goose, and Temple Days.

Table 10: CLASS™ Domain Descriptions

Figure 10: CFSRP Classroom Environment

CHILDCARE CENTER QUALITY

The Program Administration Scale (PAS) is a research-based instrument that captures childcare center leadership and management quality. Each CFSRP-supported center was assessed by the director and director’s mentor to identify areas of strength and improvement. CFSRP staff identified three focal areas: child assessments, program planning and evaluation, and family partnerships. Collectively, CFSRP-supported centers exhibited mixed results within all three areas, with the greatest improvements in program evaluation. Small improvements were also made in strategic planning and family communication. Slight decreases were seen in screening and identifying special needs, assessment in support of learning, and family support and involvement (Figure 12).

CFSRP Contributions to School Readiness

The CFSRP also measures its success in terms of children’s school readiness upon entering Fort Worth Independent School District (FWISD) prekindergarten or kindergarten and continued academic progress through third grade. Using results from FWISD assessments already in place, the evaluation examines student performance comparing cohorts of CFSRP children currently in grades prekindergarten through three to demographically similar groups of children who did not attend a CFSRP center. The comparison groups were randomly selected and matched for school, grade level, socio-economic status, ethnicity, and gender. For prekindergarten and kindergarten students, the evaluation compares results of assessments administered at the beginning of the year. For each subsequent grade level, the results are from assessments administered at the end of each year. Where enough students in the CFSRP group were available (≥40), statistical analyses were used to determine if any observed differences are statistically significant. Otherwise, differences of 5 percentage points or greater are noted. Additional information about each assessment is included in the sections below and in Appendix D. Appendix D also describes the matching process. Appendix E shows the demographic characteristics of the CFSRP students and their comparison groups.

PREKINDERGARTEN LITERACY SKILLS (FALL 2023)

FWISD administers the Circle Progress Monitoring Tool (CIRCLE)24 to prekindergarten students at the beginning of each year to identify specific early literacy skills that the children have (or have not) developed. Teachers use the CIRCLE results to plan instruction targeting individual student needs.

Figure 12 shows comparisons of the CIRCLE results for the children who attended a CFSRP center in the year prior to entering prekindergarten and their demographically similar comparison group. Although CIRCLE assesses multiple early literacy skills, the results reported in Figure 12 are for early literacy skills the CFSRP teachers focus on in their early childhood classrooms.

Descriptive analyses revealed higher percentages of CFSRP students on target for Letter Naming, Vocabulary and Listening skills. Appendix F shows results of the group comparisons of prekindergarten early literacy skills since 2018. Each year the CFSRP children had higher performance on measures of Letter Naming, Vocabulary and Listening Skills. The comparison group had higher ratings for Phonological Awareness.

*Smaller group sizes for this item (CFSRP n=20; Comp. n=64)

Figure 12: Comparison of Prekindergarten Early Literacy Skills (Fall 2023)

KINDERGARTEN READINESS (FALL 2023)

Kindergarten Assessments

FWISD administers the Texas Kindergarten Entry Assessment (TX-KEA)25 and the MAP® Reading Fluency™26 and Growth™27 assessments to kindergarten students at the beginning of each year. As with the prekindergarten assessments, these assessments are screening tools that help teachers plan instruction that targets individual student needs.

The MAP® Reading Fluency™ and Growth™ assessments each measure slightly different aspects of children’s reading ability. MAP® Reading Fluency™ assesses the extent to which students have mastered specific grade level reading skills. MAP® Reading Growth™ assesses children’s ability to use their reading skills for learning. For the current evaluation, the comparison group excluded children who attended FWISD prekindergarten in the prior year.

• The CFSRP group included kindergarten students who attended a CFSRP centers for prekindergarten in 2022-2023 or as 3-yearolds28, in 2021-2022, and

• The comparison group included students who did not attend FWISD prekindergarten in the prior year or a CFSRP center at any time.

For Language skills, descriptive analyses revealed that CFSRP children had higher ratings for Listening Comprehension measures relative to students who had never attended a CFSRP center or FWISD prekindergarten. The ratings for Overall Language and Vocabulary are similar for both groups. For Literacy skills, the CFSRP children had higher ratings for all measures.

For Social Competence skills, like the prior year results, descriptive analyses revealed higher ratings for the CFSRP students for Inhibition and Social Emotional skills. However, also like the prior year results, the comparison group received higher ratings for Emotional Management skills. See Appendix G for results of the analyses from prior years.

25 TX-KEA is a screening tool designed to assess whether kindergarten students are on-track with their literacy and social emotional skills. TX-KEA meets the Texas Educations Agency requirement for school districts to administer a Kindergarten assessment for all Kindergarten students. https://www.texaskea.org/.

26 MAP® Reading Fluency™ is an online screening and progress monitoring tool that assesses basic reading skills with an emphasis on oral fluency (e.g., listening comprehension, words per minute, accuracy, decoding). Students receive a rating of below, approaching, meeting, or exceeding grade level expectations. https://www.nwea.org/map-reading-fluency/.

27 The Kindergarten MAP® Reading Growth™ is a standardized, norm referenced assessment that also focuses on early literacy including reading comprehension and use of vocabulary. Students are rated as low, low average, average high average or high, based on the performance of a national sample. https://www.nwea. org/map-growth/.

28 Some CFSRP students entered prekindergarten in FWISD once they turned 4 years old while others remained at a CFSRP center for their prekindergarten year.

29 Although TX-KEA includes additional literacy measures as well as math measures, the evaluation focuses on overall language and literacy scores and on 5 specific skills identified by CFSRP staff as most important for kindergarten readiness.

Figure 13 shows the comparisons for the KEA language and literacy measures and Figure 14 shows the comparisons for the KEA social competence measures.

Figure 14: Comparison of Kindergarten Social Competence Skills (TX-KEA, Fall 2023)

Figure 13: Comparison of Kindergarten Early Literacy Skills (TX-KEA, Fall 2023)

MAP® Reading Fluency™ and Growth™ Results

(Fall 2023)

Table 11 shows the percentages of children in each group rated below, approaches, meets, or exceeds grade level expectations for the MAP® Fluency™ measures. CFSRP students had higher percentages of students at or above grade level expectations than the comparison group for three of the four measures (Phonological Awareness, Phonics and Listening Comprehension (highlighted in green). The differences were statistically significant for Phonological Awareness and Word Recognition. The ratings were similar for both groups (highlighted in yellow) on Picture Vocabulary measures. These results are similar to prior years when the CFSRP students had higher ratings on the majority of MAP® Fluency™ measures (see Appendix H).

MAP® Reading Growth™Results

MAP® Reading Growth™. Table 12 shows the percentage of children rated as low, low-average, average, high-average or high (compared to a national sample) on the MAP® Reading Growth™ measures. As in Fall 2022 (see Appendix H), the CFSRP higher percentages of students were at or above average for Overall Reading Achievement and for Reading and Writing Skills. The two groups showed similar performance on the comprehension measures.

CONTINUED ACADEMIC SUCCESS

The current evaluation uses the MAP® Reading Growth™ Assessment to measure student achievement at the end of each year and to measure student growth over time30. For kindergarten, first and second grade students, the MAP® assessments have replaced the ITBS. For third grade students, the MAP® Reading Growth™ assessment presents an alternative to STAAR for several reasons, primarily because it provides consistent measures from one grade level to the next as well as standardized measures and measures of academic growth.31

As with the analyses of the prekindergarten and kindergarten readiness assessments, analyses of the MAP® Reading Growth™ assessments compare results of the CFSRP students at each grade level with a demographically similar group of students who did not attend one of the CFSRP centers. The results for kindergarten, first, and second grade groups are shown in Table 13. For these groups, the comparisons are made using a cross-sectional design as in prior years. They are not longitudinal samples that follow the same students from year to year.

However, for third grade, the current evaluation uses a longitudinal approach. The students in the 3rd grade CFSRP and comparison groups were also in FWSID in their kindergarten year. With this approach, the CFSRP and Comparison Group students had likely all been in FWISD since kindergarten, providing even more similar groups for the comparisons. Figures 15 shows these results. For both the cross-sectional and longitudinal comparisons, the results identify:

• the percentage of students at or above average grade level achievement at the end of each year based on the national normed sample,

• the percentage of students who met their growth targets at the end of each year, and

• the percentage of students with average or above average growth by the end of each school year compared to the national sample.

30 RP also reviewed results of the MAP® Fluency™ and MAP Math Growth™ Assessments for internal use.

31 Differences include the following: (1) STAAR is a criterion referenced assessment measuring whether a student meets a designated grade level standard at a given time. (2) As a standardized assessment, MAP® Reading Growth™ provides measures of end-of-year student achievement relative to other similar students from a norm-referenced nationwide sample. (3) MAP® Reading Growth also includes measures at the end of the year that indicate the extent to which students meet projected growth targets. The targets set for each student are based on their baseline beginning of year scores and the observed growth for similar students in the norm-referenced sample. (4) MAP® Reading Growth™ assesses the amount of students’ growth from beginning to end of year, relative to the amount of growth observed in the norm-referenced sample.

End of Year Reading Achievement and Growth, Kindergarten-2nd Grade

As shown in Table 13, results of the end-of-year achievement level analyses for kindergarten, first, and second grade students are mixed, with higher ratings for the CFSRP students at the end of their

kindergarten year and at the end of second grade. The comparison group students had higher ratings at the end of their first-grade year on the Composition measures (highlighted in green).

When considering the extent to which students met their projected growth targets at the end of the 2021–2022 school year, and the amount of growth from the beginning to the end of the year, the results are also mixed. In kindergarten, higher percentages of CFSRP students met their growth targets (see Figure 15) and showed average or above average growth32in reading skills (see Figure 16). However, in 2nd grade the percentages were higher for the comparison group. The differences were statistically significant for the 2nd grade comparisons.

As mentioned above, the current evaluation used a longitudinal approach to compare CFSRP students’ reading achievement with even more similar groups of students who had not attended one of the CFSRP programs in their early childhood years. In addition to matching the two groups on demographic characteristics, the evaluator used FWISD enrollment data to create comparison groups who, like the CFSRP students, had been enrolled in FWISD in their kindergarten year. This new matching process was conducted for two cohorts of students: those in 3rd grade in 2021-2022 and those in 3rd grade in 2022-2023. Table 14 and Figure 17 show results of the analyses comparing the MAP Reading Achievement and MAP® Reading Growth™ results for these two cohorts of students.

End of Year Reading Achievement and Growth, 3rd Grade

The results of the third-grade comparisons are also mixed. In terms of reading achievement, the comparison group had higher ratings than the CFRP group, particularly in 2023 (see Table 14). However, these measures do not account for students’ earlier levels of achievement. When considering earlier achievement, higher percentages of the CFSRP students met their growth targets and showed higher levels of growth (see Figure 17). Academic growth was similar for both groups in 2023.

Table 14: 3rd Grade Reading Achievement: Longitudinal Cohorts

Figure 15: Percent

Conclusions and Discussion

PROCESS EVALUATION

Teacher retention from beginning to end of year remained high (79%), while lower retention rates were seen for children (67%). While teacher retention is high, the average years of employment are relatively low at 2.53 years. Teacher participation in professional development remained high at 91%. Mentor teacher and director stipends remained high for eligible participants at 87%. Education for early childhood staff continues to be a challenge, with only 14% of staff holding any early childhood certification.

Infants and toddlers made up 63% of enrollment (infants – 27% and toddlers – 36%) with three-year-olds making up another 23%. Only 12% of children in SRP programs were four- or five-years-old, in line with recent trends of preschool children leaving childcare centers for public PreK programs at local school districts or charter schools.

CHILD OUTCOMES

Overall, child outcomes were mixed: preschool children showed positive academic gains, but infants and toddlers showed mixed development and did not meet most target goals. Social emotional development among all ages remained steady or showed slight improvement.

Infants and toddlers showed some development in the areas of communication skills, gross motor skills and problem-solving skills (infants) and communication skills, personal-social skills, and problem-solving skills (toddlers), but did not meet targets in ASQ ®-3 domains except for infant gross motor skills.

Among prekindergarten students, academic results, as measured by CPALLS+, were positive. In all domains, the percentage of children improving or making acceptable progress increased over the year. Targets were met for mathematics and rhyming 2, but not for listening and rhyming 1.

Finally, results from the DECA assessment showed that social emotional skills increased among infants and toddlers and held constant among preschoolers from BOY to EOY.

CLASSROOM AND CENTER OUTCOMES

Classroom quality was variable during the 2022-2023 school year. Overall, CLASS™ scores were mixed among infant, toddler and preschool classrooms. Responsive Caregiving (infant), Engaged Support for Learning (toddler) and Emotional Support (preschool) domains increased or held steady year over year, but Emotional and Behavioral Support (toddler) and Classroom Organization and Instructional Support (preschool) decreased during the same period.

Center quality showed some improvement in family communications, program evaluation and strategic planning. The areas of family support and involvement, screening and identification of special needs and assessment in support of learning showed some decline during the program year.

SCHOOL READINESS OUTCOMES

Prekindergarten and Kindergarten Readiness

As in prior evaluations, the current evaluation suggests that CFSRP continues to contribute to prekindergarten and kindergarten early literacy skills. FWISD prekindergarten students who attended a CFSRP center had higher ratings than the comparison group in Letter Naming, Vocabulary and Listening. Of note are the higher ratings in Vocabulary, a particularly important early literacy skill for later success in reading (64% for CFSRP vs. 50% for the comparison groups).

FWISD kindergarten students with prior attendance at a CFSRP center had higher ratings on all of the TX-KEA literacy measures. Of note are the higher ratings (53.8%) for the CFSRP students’ blending skills relative to the percentages reported in the prior year evaluation (26.5%; see Appendix D). This finding suggests

that CFSRP successfully followed recommendations to increase their focus on developmentally appropriate activities to support early literacy blending skills.

On the TX-KEA Social Competence measures, the CFSRP students had higher ratings on the Social Emotional Competence measure. However, the comparison group had higher ratings for Emotional Management. Given that these results are like those of the prior year evaluation, CFSRP should continue their approach to supporting social emotional skills but also identify additional activities supporting specific behaviors related to Emotional Management.

The CFSRP kindergarten students also had higher ratings on 3 of the 4 MAP® Reading Fluency™ measures and Overall Reading Achievement on the MAP Reading Growth™ measures. These results highlight the importance of early education to help children be ready to learn in kindergarten and provide additional evidence that the CFSRP is successfully addressing early literacy development in their early childhood classrooms.

For CFSRP program improvement efforts, staff can continue to use results of the evaluation to better understand how their program impacts school readiness and to identify specific areas that need additional focus.

CONTINUED ACADEMIC SUCCESS

The findings related to CFSRP students’ success as they continue through school to the 3rd grade are not as clear and difficult to interpret. As in the prior year evaluation (see Appendices E and F), the analyses of the Spring 2023 MAP® Reading Growth™ measures for kindergarten, first, second and third grade students produced mixed results. The CFSRP children had higher ratings in some areas and the comparison group students had higher ratings in other areas. Even with the new approach to compare results for longitudinal samples of third grade students, the results are mixed. On the measures of Reading Growth, higher percentages of the 2022 cohort of CFSRP students met their growth targets but for the 2023 cohort, higher percentages of the comparison group met their growth targets.

2022-2023 CFSRP Evaluation Recommendations

The following recommendations are based on the findings of the 2022-2023 evaluation.

CHILD OUTCOMES:

• Continue targeting personal-social skill development in infant and toddler classrooms through teacher training, modeling, and classroom practices.

• Consider additional developmentally appropriate classroom teaching strategies for listening and introductory rhyming skills for preschool classrooms.

• Camp Fire and other community organizations should continue their efforts to support early childhood programs and to encourage families to enroll their children in quality public or private centers.

• Camp Fire and other community organizations should consider ways to support the expansion of infant and toddler classrooms as more families enroll preschool children in public preschool programs.

• Reevaluate ASQ-3 target goals based on trend data for the past several years.

CENTER OUTCOMES:

• Strengthen center quality supports, particularly in the areas of identifying and serving children with special needs and conducting and using child assessments.

• Identify opportunities to maintain or enhance classroom quality from beginning to end of year.

SCHOOL READINESS OUTCOMES:

• Continue the successful focus on early literacy skills with an increased focus on phonological awareness.

• Identify additional developmentally appropriate activities to address children’s social-emotional management skills.

EVALUATION PROCEDURES:

• Work with childcare centers to ensure complete and accurate records for educators and students.

• Increase training for childcare staff in administering the ASQ-3 screening tool.

• Meet with FWISD staff to discuss potential factors that may be contributing to the mixed results related to the end-of-year measures of continued academic success. Consider new or modified evaluation methods to address these mixed results including potential value of continued longitudinal analysis, additional child data to include in the analyses and/ or a primary focus only on the prekindergarten and kindergarten school readiness measures.

Early Education Apprenticeship Program

A key component of high-quality early childhood education that has received increasing attention in recent years is a stable, wellqualified workforce of educators and caregivers. Research has made clear that teacher-child relationships are an important element of early childhood learning and development33 and that centers with lower wages have higher teacher turnover rates.34 Further, support in early childhood has potential to reduce disparities and support the economic and academic success of children and families from lower-income and minority populations,35making wage shortcomings a threat to educational equity and achieving universal kindergarten readiness.

The child care industry has long been challenged by wages and provider turnover; the issue has been compounded in recent years by pandemic-related employment declines and relative wage increases in non-childcare industries that compete for workers.36 Nationwide, median hourly wages for childcare workers (excluding preschool teachers) are $13.22,37 placing childcare workers in the second percentage of all occupations.38 Wage levels in early education are lower still in Texas; in 2019, the median hourly wage among Texas teachers in early learning programs was $10.15, compared to $32.41 among kindergarten teachers. Despite the importance of the early childhood workforce, structural forces limit educators’ ability to attain living wages and supportive career pathways.

Camp Fire Early Education Apprenticeship Program (EEAP)— the first early childhood apprenticeship program in Texas to be registered with the U.S. Department of Labor—works to address these issues. Through EEAP, apprentices have paid positions at host site child development centers, where Camp Fire negotiates wage increases and provides a program stipend in addition to coaching from experienced educators, professional development through Camp Fire’s Early Education Institute and foundational professional development programs, and the opportunity to obtain a Child Development Associate (CDA) credential before graduation. Through the program, apprentices can expand their education and earning potential beyond a CDA by earning up to 33 hours of college credit and being directly connected to local college and university partners for advanced education.

EEAP enrolls apprentices in cohorts each fall and is designed to support progress at an individual pace, with completion at either 12 months (for those with an existing CDA or related credential) or 24 months (for those with limited prior work experience and/or no current CDA; see Appendix K). The theory of change39 below outlines the causal model for the program (Figure 23), with the expectation that through strengthening educators within host sites, educators and centers are both strengthened, ultimately to the benefit of child development.

33 Markowitz, A. J. (2019). Within-year teacher turnover in Head Start and children’s school readiness. University of Virginia EdPolicyWorks Working Paper. https:// curry.virginia.edu/sites/default/files/uploads/epw/70_Teacher_Turnover_in_Head_Start.pdf. Hamre, B., Hatfield, B., Pianta, R., and Jamil, F. (2013). Evidence for general and domain-specific elements of teacher-child interactions: Associations with preschool children’s development. Society for Research in Child Development 85(3). doi: 10.1111/cdev.12184.

34 Grunewald, R., Nunn, R., and Palmer, V. (2022). Examining teacher turnover in early care and education. Federal Reserve Bank of Minneapolis. https://www. minneapolisfed.org/article/2022/examining-teacher-turnover-in-early-care-and-education.

35 Iruka, I. U., Oliva-Olson, C., & Garcia, E., (2021) Research to practice brief: Delivering on the promise through equitable polices. SRI International. https:// childcareta.acf.hhs.gov/sites/default/files/public/pdgb5ta_equitablepractices_rtp_acc.pdf

36 Ibid.

37 Bureau of Labor Statistics. (2022). Table 39-9011 Childcare workers, Occupational employment and wages, May 2021. https://stats.bls.gov/oes/current/ oes399011.htm.

38 Whitebook, M., Phillips, D., and Howes, C. (2014). Worthy Work, Still Unlivable Wages: The Early Childhood Workforce 25 Years after the National Child Care Staffing Study. University of California, Berkeley: Center for the Study of Child Care Employment. https://cscce.berkeley.edu/wp-content/uploads/2014/ ReportFINAL.pdf.

39 A theory of change provides an illustration of a program’s impact pathway—the logical causal change that is expected to occur as a result of program activities.

Camp Fire Early Education Apprenticeship Program

Program Components PD Classes On-site Individualized Coaching

FIRST ORDER

Apprentice outcomes (teacher or director)

SECOND ORDER Center outcomes

THIRD ORDER Child outcomes

Improved teaching knowledge and classroom experience Improved teaching practices

Increased staff wages and qualifications (years of experience, credentials)

Improved child-teacher interactions

Increased teacher retention and staff stability

Handoff to Higher Education Partners for Further Career Progression

Educational and career progression, increased wage capacity

Increased classroom and center quality

Improved child learning and development Children enter pre-K and kindergarten ready

In the 2022-2023 school year, Camp Fire launched the third year of the apprenticeship program. The program continued to expand hybrid work to allow for participation beyond Tarrant County and saw approximately double the number of centers supported since the previous year (16 centers in 2021-2022 and 33 in 2022-2023). Overlap in apprenticeship participation of those also supported by Camp Fire’s School Readiness Program dropped 20% showing the apprenticeship program is increasing its reach outside of those centers supported by CFSRP. The program continued to increase engagement with Spanish-speaking educators, and to be inclusive of many educator backgrounds while addressing childcare deserts and workforce weaknesses. This evaluation report summarizes participant characteristics, program progress, and workforce outcomes for apprentices served in 2022-2023.

Figure 18: EEAP Theory of Change

Participant Characteristics

APPRENTICES

Fall 2022 started the 3rd year of the Early Education Apprenticeship Program with 91 total apprentices participating. Of the 91 apprentices, 14 (15%) were continuing apprentices from 20212022, enrolled in their 2nd year in the program and 77 apprentices were new to the program (85%). New apprentice enrollment spiked in 2022-2023 compared to the previous year and the initial pilot cohort; 23 in 2020-2021, 34 in 2021-2022, and 77 in 2022-2023.

Apprentices in 2022-2023 were overwhelmingly female and came from varied age, racial/ethnic, and educational backgrounds upon program entry (Table 11). These participants had higher hourly wages at entry ($13.31 compared to $11.79 for the second-year participants) and a higher proportion of participants had no prior credentials at apprenticeship entry (64% compared to 56%).

HOST SITES

Thirty-three early childhood centers served as host sites for apprentices during the 2022-2023 school year (more than double the early childhood centers served in 20212022). These host sites support apprentice professional development, credential attainment, and annual wage increases through providing time, space, and financial support for early childhood educator career opportunities. Host sites had from one and nine participating apprentices in their classrooms.

*Race/ethnicity sums to more than 100% due to selection of multiple characteristics.

Program Participation

APPRENTICE RECRUITMENT

In EEAP’s first and second years (2020-2022), Camp Fire First Texas recruited apprentices in collaboration with partners, particularly centers participating in the School Readiness Program. EEAP introduced potential apprentices to the opportunity through virtual interest sessions and onsite recruitment with childcare center partners. It prioritized admission for teachers who have been in the field at least one year and who work at centers with a Texas Rising Star rating of 3 or 4.

All apprentices agreed to full program participation, with the support of their center director. They agreed to attend program professional development courses, complete associated coursework, meet with an assigned mentor on a regular basis, and maintain employment at their current center (i.e., host site) for a minimum of twelve months after graduation from the program. For this commitment, apprentices would receive up to $2,000 in stipends, assistance with wage increase negotiations, a non-expiring CDA credential, professional development, mentoring support, and assistance with entering or continuing in higher education opportunities.

APPRENTICE PROGRESS

EEAP is a two-year program that admits new apprentices every year. Of the 91 apprentices served in 2022-2023, 77 (85%) were new apprentices and 14 (15%) were returning for their 2nd year of enrollment in the program. By August 2023, 34 of the 91 apprentices served graduated from the program (37%); 29 are continuing into their 2nd year of enrollment (32%); and 28 (31%) cancelled their participation in the program (Table 13).

Among the 2022-2023 apprentices, 35 had completed CDAs (38%), of which, 14 completed their CDA before program enrollment and 19 completed their CDA during the program. Additionally, 56 completed EEI* during enrollment (62%) and 15 apprentices weren’t required to completed EEI.

Within the 2022-2023 apprentices, a portion of apprentices left the apprenticeship before completing it (31%). The top reasons for leaving the program were “voluntarily quit” and “left to accept other employment,” 32% and 29%, respectively (Table 14).

Table 15. EEAP Apprentice Characteristics (N=91), 2022-2023

Apprentice Outcomes

EEAP is an educational and career pathway building program designed to have a lasting, positive effect on the careers of educators. In addition to near-term goals of apprentice program completion and demonstration of classroom best practices, success for EEAP means wage increases and educational certifications for early childhood professionals.

In 2022-2023, Camp Fire staff implemented a new self-evaluation assessment and Appendix A assessment (see below for description of Appendix A). The evaluation assesses the extent to which these newly implemented assessments measure the apprentice’s confidence and outcomes throughout program participation.

APPENDIX A

Appendix A is a tool developed by the National Association for the Education of Young Children (NAEYC) that assesses the core body of knowledge, skills, values, and dispositions early childhood educators must demonstrate to effectively promote the development, learning, and well-being of young children. It covers six standards as outlined in NAEYC’s Professional Standards and Competencies for Early Childhood Educators, including:

• Child Development and Learning in Context

• Family-Teacher Partnerships and Community Connections

Understanding of child development for the age group I work with

83 -3

Understanding each child has own needs, strengths, interests 90 85 -5

Understanding that learning doesn’t just happen at school 95 86 -9

Understanding the value of diversity in families

Collaborating with families to meet the needs of children

86 -2

85 -7

Understanding the purpose of child assessments 88 86 -2

Understanding different types of child assessments

Demonstrating positive, caring, supportive relationships

Incorporating play in the classroom

80 -1

86 -9

83 -9

Creating relevant and engaging lesson plans 88 85 -3

Understanding the content that I need to teach 93 85 -8

Understanding different ways to teach concepts

Reflecting on my teaching to improve more and more over time

88 85 -3

95 85 -10

Making educational decisions based on children’s backgrounds

• Child Observation, Documentation, and Assessment

• Developmentally, Culturally, and Linguistically Appropriate Teaching Practices

• Knowledge, Application, and Integration of Academic Content in the Early Childhood Curriculum

• Professionalism as an Early Childhood Educator

EEAP Apprentices are assessed with the tool four times over the two-year program. (Table 16).

SELF-EVALUATION

Apprentices completed the self-evaluation assessment at the middle and end of the school year to identify areas where they have changed their confidence levels throughout program participation. Fifty-nine apprentices completed the self-evaluation in the middle and at the end of the year.

At mid-year, apprentices have varying high confidence levels from 73%-95% across all areas. At the end of 2022-2023, apprentices show increased confidence in “connecting families with community resources they may need” (+8%) and “involving families with assessments” (+5%). The biggest decline in confidence was seen in “reflecting on my teaching to improve more and more over time” (-10%). Decreases in confidence levels may indicate an over-estimation of skills by apprentices before they were exposed to additional support and training. (Table 15)

Cultivating Growth

In 2022-2023, the EEAP built on the success of its initial two-year pilot and guided a large new cohort of apprentices in pursuing increased education and professional development. When educator stability within a role has been particularly tenuous, EEAP brought stability and opportunity to apprentice and host site participants. Initial apprentice outcome results are promising, with graduates showing wage growth, commitment to ongoing educational pursuits, and the stability associated with earning a non-expiring professional credential. A key area for increasing focus and improvement is building and measuring the quality of teacher-child interactions in the classroom and increasing support designed to decrease attrition and increase completion of the CDA credential.

Table 18. Apprentice Self-Evaluation Results, 2022-2023

Table 19. Apprentice Appendix A Results, 2022-2023

2022-2023 EEAP Evaluation Recommendations

The following recommendations are based on the findings of the 2022-2023 evaluation.

APPRENTICE SELECTION

• Clearly communicate to potential apprentices recruited through current CFSRP partners the differences between CFSRP and EEAP

PROGRAM SUPPORT

• Identify barriers for participant completion

• Provide increased support to apprentices at risk for cancelling participation in the program

• Provide increased support to apprentices working towards the CDA credential

POST-PROGRAM SUPPORT

• After connecting graduates with institutes of higher education, utilize data sharing agreements to understand graduates’ postprogram academic outcomes

• Follow up with graduates to track post-program outcomes including career and educational progression, wage increases, and duration with their center and the field

EVALUATION PROCEDURES

• Consistently measure and report on all apprentices’ classroom quality at beginning and end of year, with the capacity for rapid response and improvement before end of year or end of program

• Standardize wage reporting on a consistent, hourly basis

• Collect standardized reasons for leaving the program in apprentice exit interviews

Conclusion

Camp Fire First Texas’ early childhood programs address fundamental family and community needs while pursuing high-quality educational experiences. The Camp Fire School Readiness Program (CFSRP) prepares children for school by supporting teacher and classroom quality through professional development and mentoring. The Early Education Apprenticeship Program (EEAP) strengthens the early education workforce in Tarrant County and surrounding areas through rigorous training and support. Each program has continued to evolve with community circumstances to best support the well-being of children, families, communities, and school districts. Overall, Camp Fire's programs to support early childhood educators and children in 2022-2023 demonstrated a commitment to growth, with innovative solutions to complex issues facing the field of early childhood education today.

List of Appendices

APPENDIX A

CFSRP Program Components

APPENDIX B

Evaluation Methods

APPENDIX C

Child Development and Center Quality: Assessment Tools and Data Analysis

APPENDIX D

Data Analysis and Assessment Tools for FWISD Data

APPENDIX E

CFSRP and Comparison Group Demographics (FWISD, Fall 2023)

APPENDIX F

Year-to-Year Comparisons of Prekindergarten Readiness Results

APPENDIX G

Year-to-Year Comparisons of Kindergarten Readiness Results

APPENDIX H

Year-to-Year Comparisons of Kindergarten Fall MAP® Reading Fluency™ Results (Fall 2020 - Fall 2023)

APPENDIX I

Year-to-Year Comparisons of Kindergarten, 1st and 2nd Grade MAP® Reading Growth™ Results (Fall 2020 - Fall 2023)

APPENDIX J

Early Education Apprenticeship Tracks

LEVELS OF SUPPORT

The CFSRP supports 64 classrooms in 15 licensed, non-residential child development centers that provide care services and early education. The CFSRP engages participating child development centers at four levels of professional development intensity. A child development center’s movement from a lower intensity level (Level 1: Initial Relationship Building) to a higher intensity level (Level 4: Sustainability) is determined by factors such as length of participation in the program, class participation requirements, and center performance and capacity. The table below shows the four professional development levels, three of which include professional development support.

PROFESSIONAL DEVELOPMENT (PD) AND STIPENDS

Professional development is offered to teachers through Foundational PD and the Early Education Institute (EEI). Both are designed to increase knowledge and skills in techniques that promote child development and classroom management. In addition, the EEI specifically addresses five components of reading science cited as critical for effective early literacy instruction.40 Directors attend many teacher sessions and participate in the Director’s Institute (DI). The DI is designed to increase knowledge of child development and center business management and leadership practices.

Historically, to promote teacher retention and engagement, full-time teachers who have completed foundational professional development and directors have been eligible for a stipend based on attendance and

demonstrated competency in Camp Fire professional development. The incentive pay is distributed at the end of the EEI at the program year's end. The teachers’ and directors’ fulfilment of the requirements is reviewed prior to payment distribution. In 2022-2023, stipends for professional development participation were targeted toward EEAP apprentices, directors of CFSRP Level 4 centers, and CFSRP mentor teachers were eligible for stipends of up to $2,000.

MENTORING

On-site individualized coaching is provided by CFSRP mentors41 who hold associate, bachelor’s, or master's degrees in child development or early childhood education; have three to five years of experience coaching, mentoring and/or working in early childhood settings; and hold certifications in child development assessments. CFSRP mentors provide coaching visits to Centers based on Center level, with Level 2 and 3 Centers receiving more visits than Level 4 Centers. The practice-based coaching sessions with teachers consist of creating Teacher Action Plans with SMART (Specific-MeasurableAttainable-Realistic-Timely) goals based on needs identified from assessments (e.g., using teaching best practices, improving classroom management) and supporting the attainment of the identified goals. The coaching sessions with directors also consist of setting SMART goals based on needs identified from assessments related to center business management and leadership practices and supporting the attainment of the identified goals.

FAMILY ENGAGEMENT

The CFSRP family engagement component is an ongoing collaboration between directors and center staff, which consists of a focus on reciprocal communication between families and center staff, as well as family support and involvement-- a range of activities that allows a childcare center to be responsive to family needs, including Parent Cafés42 and Playgroups.

40 National Council on Teacher Quality (2020). Program Performance in Early Reading Instruction retrieved from https://www.nctq.org/publications/home.

41 The CFSRP established a Mentor Professional Pathway framework which categorizes mentors into four levels based on their existing level of training and experience. For example, Level 1 (Beginning) mentors may need support with close guidance, Level 2 (Developing) mentors may need support with increasing independence, Level 3 (Proficient) mentors may need limited support and can independently enhance the knowledge and skills of others in the profession, and Level 4 (Exemplary) mentors can develop program policies and practices and enhance the knowledge and skills of others in the profession. The CFSRP Director uses the Mentor Professional Pathway framework to monitor mentor needs and promote professional development opportunities. There were 5 mentors during the 2018-2019 school year: one at Level 1, one at Level 2, and 3 at Level 3.

42 Parent Cafés are a type of family meeting/support group that CFSRP has supported partner centers in offering to their parents. Parent Cafés are carefully designed, structured discussions that use the principles of adult learning and family support to help participants explore their strengths, learn about Protective Factors, and create strategies to help strengthen their families. CFSRP has also encouraged the use of play groups as a family support. Play groups provide opportunities for parents and their children to interact together in a planned ‘play activity’ that aligns learning opportunities between school and home. The play groups promote social-emotional development, support parent/child relationships, and encourage parents to interact with other parents in the group. CFSRP also provides information and presentations about community resources to center directors who can then use this information to refer families to supportive services regarding family issues, which is another form of family engagement. The CFSRP uses the Family Engagement Measure from the Program Administration Scale (PAS) to set programmatic goals in this area.

PROFESSIONAL DEVELOPMENT LEVEL NUMBER OF CENTERS (CLASSROOMS) DESCRIPTION

RELATIONSHIP BUILDING BETWEEN CFSRP AND THE CENTER

LEVEL 1 0 (0)

Does not include professional development

LEVEL 2 3 (13) BASIC

Center participates for one year

LEVEL 3 4 (21) INTENSE

Center participates for three years

SUSTAINABILITY

LEVEL 4 8 (30)

Center participation begins after the third, intensity-level year and continues as long as the center remains in the program

Description of CFSRP Professional Development Levels

APPENDIX B: EVALUATION METHODS43

The CFSRP evaluation consists of both a process and outcome evaluation. The process evaluation component provides a clearer picture of how the CFSRP was being implemented in practice and determines to what extent the program was operating as designed in the theory of change. Findings from process evaluation can inform program improvement and help explain why a program achieved or failed to achieve the intended outcomes. The CFSRP process evaluation included a specific focus on professional development participation, stipend allocation, and mentor activities.

The outcome evaluation included a focus on child outcomes (developmental, academic, and social-emotional) and center outcomes (classroom environment and management, Center leadership and management). The CFSRP outcome evaluation also includes a comparative analysis of outcomes for CFSRP children and demographically similar groups non-CFSRP children enrolled in Fort Worth Independent School District (FWISD).

This part of the evaluation uses assessments already in place to compare the performance of the CFSRP children and the comparison groups when they enter prekindergarten and at the end of their kindergarten through third grade years.

The EEAP evaluation includes both process and outcome evaluation components. It examines characteristics, motivations, and barriers/supports for success of apprentices in the initial cohort, and a summary of apprentice progress to completion. Its focus is on identifying participant characteristics and needs to fully formulate and scale the program for future cohorts. Additionally, outcome measures are collected through a self-evaluation survey and Appendix A, which measures best classroom practices.

CHILD DEVELOPMENT AND CENTER QUALITY:

ASSESSMENT TOOLS AND DATA ANALYSIS

CFSRP contracts with CNM to provide program evaluation consulting services and CNMpact outcomes services. CNM created secure, web-based online data entry spreadsheets for each CFSRP child development center. Directors at CFSRP-supported child development centers entered student and teacher enrollment information, classroom information, and assessment data. The Classroom Assessment Scoring System (CLASS™) was used by trained CLASS™ observers and submitted the final data to the CNM evaluation team to analyze. Other assessment data were provided by the CFSRP Director. Prior to data analysis, CFSRP staff reviewed and cleaned final data. The table below presents each assessment and its associated assessment areas.

Classroom/Center Outcomes

ASSESSMENT AREA ASSESSED DATA ANALYSIS

Child Outcomes

AGES AND STAGES QUESTIONNAIRE, VERSION 3 (ASQ®-3)

A standardized, screening tool designed to identify infants and young children who are and are not displaying typical age-appropriate development. CFSRP recommends that children ages three years and five months or younger receive the ASQ®-3 assessment.

CIRCLE PHONOLOGICAL AWARENESS, LANGUAGE AND LITERACY SCREENER PLUS MATH (CPALLS+)

A standardized, criterion-referenced assessment designed to measure children’s literacy and language skills. CPALLS+ recommends that children ages three years and six months or older receive the CPALLS+ assessment.

DEVEREUX EARLY CHILDHOOD ASSESSMENT (DECA)

A strengths-based, standardized assessment and planning system that supports educators in promoting children’s social and emotional development, thus promoting resilience.

Cognitive and Physical Development

Language and Literacy Development

The evaluation team calculated the percentage of children meeting the cut-off for developmental skills in five domains at the beginning and end of the year. The results were disaggregated by age group.

The percentage of children demonstrating improvement in developmental skills from the beginning to the end of the year was also calculated.

Language and Literacy Development

Cognitive Development (Math)

The evaluation team calculated the percentage of children meeting the cut-off for language and literacy skills at the beginning and end of the year. Separate analyses were conducted for three-year-old children (MOY-EOY comparisons) and four-and five-year-old children (BOYEOY). The results were disaggregated by age group.

The percentage of four- and five-year old children demonstrating improvement in developmental skills from the beginning to the end of the year was also calculated.

SocialEmotional Development

The evaluation team calculated the percentage of children who scored in the Typical or Strength category in social-emotional/resilience at the beginning and the end of the year. The percentage of children demonstrating improvement in their scores was also calculated.

CLASSROOM ASSESSMENT SCORING SYSTEM (CLASS™)

A standardized, observation-based assessment designed to assess classroom management and quality on a 7-point scale. The Infant CLASS™ measures the quality of responsive caregiving in infant classrooms. The Toddler CLASS™ measures the quality of emotional and behavioral support and engaged support for learning in toddler classrooms. The Pre-K CLASS™ measures the quality of emotional support, classroom organization, and instructional support. For Pre-K CLASS™, the quality threshold is set at 5 for the Emotional Support and Classroom Organization domains, and at 3.25 for the Instructional Support domain.

BEST PRACTICES OBSERVATION TOOL (BPOT)

A research-based observational checklist that measures the presence or absence of research-based teaching practices that align with CFSRP professional development curriculum. This tool is intended for professional development purposes. Teachers in infant classrooms are rated on 105 best-practice teaching strategies, and teachers in toddler classrooms are rated on 110 bestpractice teaching strategies. The BPOT for three-year-old classrooms and four-year-old classrooms includes 110 and 120 best-practice teaching strategies, respectively.

PROGRAM ADMINISTRATION SCALE (PAS)

A 25-item research-based instrument that measures the quality of leadership and management practices of early childhood programs. PAS measures quality on a 7-point scale (1 = inadequate, 3 = minimal, 5 = good, 7 = excellent).

Classroom Management and Quality

The evaluation team included only teachers with matched pre- and post-assessment scores in the analysis. The average CLASS™ pre- and post-assessment scores were compared. In addition, for Pre-K classrooms, analyses examined the percent of teachers who met a researchbased quality threshold for each domain.

Quality in Teaching Practices

Based on the design of the BPOT assessments, the evaluation team calculated the total observations and created a weighted system that categorized scores as ‘needs support’, ‘emerging’, and ‘consistently meets’. The results were disaggregated by domain.

This assessment is used internally to assist mentors with identifying target areas for teacher development.

Center

Leadership and Management Quality

The evaluation team used each center’s individual score to calculate an overall average for each of the 10 domains.

DATA ANALYSIS AND ASSESSMENT TOOLS FOR FWISD DATA

Camp Fire partners with FWISD to gain access to data from student assessments currently used in the school district. Through this partnership, the CFSRP evaluation team has been able to assess the impact of the program for eight years with analyses of the children’s assessment scores not only as they enter school but also through their kindergarten, 1st, 2nd, and 3rd grade years. Camp Fire contracted with Aware Research Solutions Inc. to provide evaluation and data analysis services.

Each year, using a random selection process44, the evaluation team compares the assessment scores of children who attended a CFSRP-supported center in one of the prior six years and children in -demographically similar comparison groups who did not attend one of the centers. This technique allows the evaluation team to create a random, non-biased sample of children who are like the sample of CFSRP children and, in turn, make valid comparisons between the two groups. Any statistically significant differences identified in the results provide evidence that the differences between CFSRP children’s scores and the comparison group’s scores can be attributed, in part, to the CFSRP program rather than to random chance.

ASSESSMENT DOMAINS

Kindergarten Readiness

The groups were matched on the following characteristics:

• School location

• Grade level

• Ethnicity

• Free/Reduced Lunch Status

• Gender

• Limited English Proficiency (LEP) status

• Special Education status

To get the best possible match, the comparison groups are necessarily larger than the CFSRP groups. Students with Special Education and/ or LEP designations included in the analyses only if the CFSRP group included 5 or more with the designation (see Appendix E for the demographic descriptions of the CFSRP and comparison groups). Depending on the grade level for each set of comparison groups, the analyses were conducted with the FWISD assessment data described in the table on the next page, “FWISD Assessments Used in the Evaluation.” Because very few CFSRP children were assessed in Spanish only the English versions of each assessment were included in the analyses.

Prekindergarten Readiness

CIRCLE is similar to the CPALLS+ assessment used in the CFSRP three and four-year old classrooms. It is a criterion-referenced assessment based on the Texas Education Agency (TEA) prekindergarten guidelines for literacy, math, and social skills. Teachers use CIRCLE at the beginning of the school year to help identify children who meet or do not meet developmental benchmarks so they can plan individualized instruction.

44 For the current evaluation, the random selection process used SAS Statistical Analysis Software, SAS/STAT | SAS to select the comparison group students.

45 This description is taken from the Texas Kindergarten Entry Assessment User Guide at: https://cliengage.org/user-guides/User_Guide_TX-KEA_8.13.2018.pdf.

46 CLI Engage (2017). CIRCLE Progress Monitoring System. https://cliengage.org/public/tools/assessment/circle-progress-monitoring/.

47 A variety of sources provided the information included in these descriptions. 1Brief_PhonologicalAwareness.pdf (d1yqpar94jqbqm.cloudfront.net) Phonological Awareness LEARN - Children's Literacy Initiative (cli.org), Kindergarten_TEKS_0820 (texas.gov), Phonics LEARN - Children's Literacy Initiative (cli.org), Reading Curriculum Ladders (fortheteachers.org)

48 Texas Kindergarten Entry Assessment (TX-KEA).https://www.texaskea.org/.

TEXAS KINDERGARTEN ENTRY ASSESSMENT (TX-KEA)48

TX-KEA is a screening tool designed to assess kindergarten children’s skills in seven areas of school readiness: Language, Early Literacy, Math, Science, Executive Functioning, Social Emotional Skills, and Physical Skills. For each measure, students receive a score indicating if they are on-track, need monitoring or need additional support/ intervention. Teachers use the TX-KEA is to identify children who may need additional support and to plan individualized instruction.

Texas school districts and charter schools are required to administer a Kindergarten assessment for all Kindergarten students. The TX-KEA is on the Commissioner’s approved list of assessment Instruments for meeting this requirement.

The TX-KEA includes four domains: language, literacy, executive functioning, and social emotional competence and emotion management.

LANGUAGE DOMAIN

VOCABULARY is a foundational language skill that supports learning in all content domains. Knowing a student’s vocabulary abilities helps teachers adjust their own vocabulary usage during instruction to levels that are most beneficial for individual children.

LISTENING COMPREHENSION, a foundational language skill that supports learning in all areas, assesses students’ ability to understand verbal information and follow directions. Knowing students’ receptive language abilities allows teachers to adjust the complexity of their own language during instruction.

LITERACY DOMAIN

The LETTER NAMES subtest assesses children’s knowledge of the names associated with various letters of the alphabet. Letter names is one component of letter knowledge which can predict reading achievement

The LETTER SOUNDS subtest assesses children’s knowledge of letter sounds. Letter knowledge at kindergarten entry is a strong predictor of literacy achievement

The BLENDING SOUNDS subtest assesses children’ phonological awareness, or sensitivity to the sound structure of oral language. Phonological awareness is necessary for learning to read and write and is predictive of literacy achievement.

The SPELLING subtest assesses children’s early spelling abilities, which is the ability to use sound-symbol relationships to write words. TX-KEA assesses spelling because it is highly related to later literacy achievement. Attempting to spell words requires children to apply multiple literacy skills simultaneously, such as alphabet knowledge and phonological awareness.

EXECUTIVE FUNCTIONING

The Executive Functioning domain addresses the cognitive skills used by children to plan, problem solve, and follow classroom rules.

INHIBITION. Students are asked to respond accurately to a specific stimulus (e.g., butterfly), and withhold, or inhibit, a response to a different stimulus (e.g., bee). Scores reflect the student’s ability to respond accurately while inhibiting a response

WORKING MEMORY. Students are assessed on their ability to hold in memory 1 – 3 pieces of information in an increasingly complex setting. In this subtest, children recall where cars are parked in a garage. The number of cars and the number of parking spaces increases as the student progresses.

ATTENTION. Students are assessed on their ability to focus their attention, stay on task, as well as quickly and accurately focus on relevant features of the task. They are provided 2 minutes to make as many correct matches as possible between the target object, a flower, and five answer choices, other flowers.

SOCIAL EMOTIONAL COMPETENCE AND EMOTION MANAGMENT

The SOCIAL/EMOTIONAL COMPETENCE subtest focuses on children’s social and emotional skills within a classroom setting. The subtest evaluates children’s pro-social skills, approaches to learning, and emotion understanding.

The EMOTION MANAGEMENT subtest focuses on children’s ability to manage their emotions and respond appropriately to an emotional experience. They are evaluated on whether they can adapt to the demands of a classroom and school environment.

Kindergarten-3rd Grade Reading Fluency49

LANGUAGE DOMAIN

MAP® READING FLUENCY™50 is an online screening and progress monitoring tool. The assessment establishes a benchmark oral reading fluency level for students, and depending on the level, assesses foundational or advanced reading skills. For kindergarten students, MAP® Reading Fluency™ assesses Language Skills (Listening Comprehension & Vocabulary) and Decoding Skills (Phonological Awareness & Phonics). For each skill, students are rated at, below, approaching, meets or exceeds grade level.

LISTENING COMPREHENSION includes the ability to listen to a passage that is read aloud and ask and answer questions about the passage, as well as the ability to repeat and follow sequenced, multi-step directions.

PICTURE VOCABULARY refers to the students’ ability to understand the meaning of spoken words (receptive vocabulary) by correctly identifying pictures related to the spoken words.

DECODING DOMAIN

PHONOLOGICAL AWARENESS is the ability to recognize and work with sounds in the spoken language, including skills such as: identifying rhyming words, recognizing words with same initial sound, blending syllables, and segmenting multisyllabic words.

PHONICS/WORD RECOGNITION refers to students’ ability to recognize the sounds of the spoken language and the letters and combinations of letters (words) that represent those sounds.

Kindergarten-3rd Grade Reading Growth51

MAP® READING GROWTH™52 is a standardized assessment that provides categorical (quintiles) and continuous scale (RIT) scores designed to measure achievement at a given point in time as well as growth over the school year and from one year to another. The quintile scores identify students as low, low average, average, high average, or high, relative to a national norm-referenced group, for overall reading achievement, foundational skills (Reading/Writing & Vocabulary) and Literal Comprehension skills (Analyzing Text & Composition.). MAP® Reading Growth™ also includes measures of each student’s projected growth (from fall to spring), identifies whether the student met that projection and provides a growth quintile based on their baseline beginning of year scores and the observed growth for similar students in the norm-referenced sample.53

FOUNDATIONAL SKILLS

BEGINNING READING AND WRITING skills include phonological awareness and phonics (described above) as well as correct use of oral language.

USE OF VOCABULARY refers to the students’ ability to understand how words can be used such as the ability to identify words with the same meaning or finish an incomplete sentence.

LITERAL COMPREHENSION

ANALYZING TEXT refers to students’ ability to understand key ideas, implied meaning, and details in an oral or written passage.

COMPOSITION: Inquiry and Research refers to the students’ ability to use critical thinking to analyze and summarize text.

49 A variety of sources provided the information included in these descriptions. 1Brief_PhonologicalAwareness.pdf (d1yqpar94jqbqm.cloudfront.net) Phonological Awareness LEARN - Children's Literacy Initiative (cli.org), Kindergarten_TEKS_0820 (texas.gov), Phonics LEARN - Children's Literacy Initiative (cli.org), Reading Curriculum Ladders (fortheteachers.org)

50 https://www.nwea.org/map-reading-fluency/

51 A variety of sources provided the information included in these descriptions. 1Brief_PhonologicalAwareness.pdf (d1yqpar94jqbqm.cloudfront.net) Phonological Awareness LEARN - Children's Literacy Initiative (cli.org), Kindergarten_TEKS_0820 (texas.gov), Phonics LEARN - Children's Literacy Initiative (cli.org), Reading Curriculum Ladders (fortheteachers.org)

52 https://www.nwea.org/map-growth/

53 NWEA. (2020). 2020 NWEA MAP Growth normative data overview. https://teach.mapnwea.org/impl/MAPGrowthNormativeDataOverview.pdf

AND COMPARISON GROUP DEMOGRAPHICS (FWISD, FALL 2023)

YEAR-TO-YEAR COMPARISONS OF PREKINDERGARTEN READINESS RESULTS (FALL 2018 – FALL 2023)

Comparison of Beginning of Year Pre-K Skills: CIRCLE Assessment Fall 2018-Fall 2023

For all comparison charts, Yellow = stat.

YEAR-TO-YEAR COMPARISONS OF KINDERGARTEN READINESS RESULTS (FALL 2018 – FALL 2023)

FWISD Kindergarten Entry Assessment (TX-KEA) Ratings: CFSRP Students and Comparison Group (Fall 2018 - Fall 2023)54 For all comparison charts, Yellow = stat. significant, p<.05 ; Green = 5 point or higher difference

APPENDIX H, CONTINUED: YEAR-TO-YEAR COMPARISONS OF KINDERGARTEN FALL MAP READING GROWTH RESULTS (FALL 2020 – FALL 2023)

YEAR-TO-YEAR COMPARISONS OF KINDERGARTEN FALL MAP READING FLUENCY

RESULTS (FALL 2020 – FALL 2023)

MAP Reading Fluency Scores: CFSRP Students and Comparison Group (Fall 2020 - Fall 2023)55

OVERALL ACHIEVEMENT

MAP READING FLUENCY ITEM LEVELS

YEAR-TO-YEAR COMPARISONS

OF KINDERGARTEN,

1ST AND 2ND GRADE MAP READING GROWTH RESULTS

ANALYZING TEXTS

APPENDIX K:

EARLY EDUCATION APPRENTICESHIP TRACKS

EARLY EDUCATION APPRENTICESHIP TRACKS

No prior work experience

Expired or no CDA or other qualifying related instruction

Complete EEI and concentration classes

Complete EEI and concentration classes

Prior related work experience

Active CDA or other related instruction

Obtain valid CDA

Obtain valid CDA

Request credit for prior work experience

Request credit for prior instruction

Complete remaining on-the-job learning (4,000 hours)

Complete EEI and concentration classes

Complete EEI and concentration classes

Complete remaining on-the-job learning (2,000 or 4,000 hours)

Receive credit for up to 2,000 hours of prior work experience

Complete CDA, remaining education and on-the-job learning

Receive credit for qualifying related instruction

Complete remaining education and on-the-job learning

EARLY CHILDHOOD EDUCATOR CERTIFICATE

Camp Fire School Readiness Program

Since 2005, the Camp Fire School Readiness Program (CFSRP) has helped children be ready to start school through improving quality of care at neighborhood childcare centers. CFSRP supports early education programs that feed into the Fort Worth Independent School District with professional development and individual coaching/mentoring focused on teaching practices that improve language and social-emotional skills among children age 0-5.

Our Students

Our Teachers & Directors

Early Education Apprenticeship Program

Camp Fire School Readiness Program & Early Education Apprenticeship Program Evaluation Report 2022-2023 School Year