2018-2019 Camp Fire School Readiness Program Evaluation Report

SCHOOL Readiness 2018-2019 Program Evaluation Report

Prepared by: CNM • AWare Research Solutions

Camp Fire First Texas + 817.831.2111 + CampFireFW.org

TABLE OF CONTENTS LIST OF FIGURES ������������������������������������������������������������� 3 LIST OF TABLES ��������������������������������������������������������������� 4 EXECUTIVE SUMMARY ������������������������������������������������� 5 Evaluation Objectives ������������������������������������������������������� 5 Key Findings: CFSRP Program Implementation ��������� 5 Key Findings: Child Outcomes ����������������������������������������� 6 Key Findings: Center Outcomes ��������������������������������������� 7 INTRODUCTION ��������������������������������������������������������������� 8 Program Components ������������������������������������������������������� 8 Levels of Support ������������������������������������������������������� 8 Professional Development (PD) and Stipends ����� 9 Mentoring ������������������������������������������������������������������� 9 Family Engagement ��������������������������������������������������� 9 Program Theory of Change ��������������������������������������������� 10 PROCESS EVALUATION ��������������������������������������������� 11 Professional Development and Stipend/Incentives � 11 Mentors ������������������������������������������������������������������������������� 11 One-on-One Mentoring ����������������������������������������� 11 Director Experiences ������������������������������������������������������� 12 CHILD OUTCOMES: DEVELOPMENT AND ACADEMIC READINESS ����������������������������������������������� 13 Age-Appropriate Development: Infants and Toddlers ������������������������������������������������������������������������������� 13 Age-Appropriate Development: Preschool ����������������� 15 4- and 5-Year-Olds ��������������������������������������������������� 15 Children’s Social-Emotional Development ����������������� 15 School Readiness: Academic and Social-Emotional � 16 Pre-Kindergarten and Kindergarten Early Literacy Skills ����������������������������������������������������������� 16 Social-Emotional Skills ����������������������������������������� 17 Academic Success Over Time ��������������������������������� 18

CENTER OUTCOMES ����������������������������������������������������� 23 Improved Quality in Classroom Environment and Management ��������������������������������������������������������������������� 23 CLASS™ Results ������������������������������������������������������� 23 Improved Quality in Center Leadership and Management Practices ��������������������������������������������������� 25 CFSRP CHARACTERISTICS ����������������������������������������� 26 Center Characteristics ����������������������������������������������������� 26 Student Characteristics ��������������������������������������������������� 27 Teacher Characteristics ��������������������������������������������������� 28 Teacher Wages and Retention ������������������������������� 28 Mentor Characteristics ����������������������������������������������������� 29 CONCLUSION ����������������������������������������������������������������� 30 Areas of Strength ������������������������������������������������������������� 30 Process Evaluation ��������������������������������������������������� 30 Child Outcomes ��������������������������������������������������������� 30 Center Outcomes ����������������������������������������������������� 31 Areas for Growth ��������������������������������������������������������������� 31 Process Evaluation ��������������������������������������������������� 31 Child Outcomes ��������������������������������������������������������� 32 Center Outcomes ����������������������������������������������������� 33 Recommendations ����������������������������������������������������������� 33 APPENDICES ������������������������������������������������������������������� 35

Camp Fire School Readiness Program | 2018-2019 Program Evaluation Report

2

LIST OF FIGURES Figure 1: CFSRP Theory of Change ������������������������������������������������������������������������������������������������������ 10 Figure 2: Age-Appropriate Infant Development, Change from BOY to EOY, 2018-2019 (N=89) ������������������������������������������������������������������������������������������������������������������������������������������������������ 14 Figure 3: Age-Appropriate Toddler Development, Change from BOY to EOY, 2018-2019 (N=180) ���������������������������������������������������������������������������������������������������������������������������������������������������� 14 Figure 4: CPALLS+, Percentage of 4- and 5-Year-Old Children Improving or Making Acceptable Progress from BOY to EOY, 2018-2019 (N=56) ������������������������������������������������������������ 15 Figure 5: Social-Emotional Development at BOY and EOY, 2018-2019 ���������������������������������������� 16 Figure 6: Fall 2019 Comparisons of Pre-Kindergarten Literacy Ratings (CIRCLE Assessment) �� 17 Figure 7: Fall 2019 Comparisons of Kindergarten Language and Literacy Ratings (KEA) ���������� 17 Figure 8: Fall 2019 Comparisons of Kindergarten and Pre-Kindergarten Children’s Social Competence ������������������������������������������������������������������������������������������������������������������������������������������ 18 Figure 9: Comparisons of Kindergarten Reading Achievement (CFSRP and Comparison Groups; ITBS 2016-2019) ���������������������������������������������������������������������������������������������������������������������� 19 Figure 10: Comparisons of Kindergarten Math Achievement (CFSRP and Comparison Groups; ITBS 2016-2019) ���������������������������������������������������������������������������������������������������������������������� 19 Figure 13: Comparisons of Second Grade Reading Achievement (CFSRP and Comparison Groups; ITBS 2016-2019) ���������������������������������������������������������������������������������������������������������������������� 21 Figure 14: Comparisons of Second Grade Math Achievement; ITBS 2016-2019) ������������������������ 21 Figure 15: Comparisons of 3rd Grade Reading Achievement; STAAR 2017-2019 ������������������������ 22 Figure 16: Comparisons of 3rd Grade Math Achievement; STAAR 2017-2016 ������������������������������ 22 Figure 17: Classroom Environment and Management at BOY and EOY, 2018-2019 ������������������� 24 Figure 18: Percentage of Classrooms at or Above the Preschool Quality Threshold, 20182019 ��������������������������������������������������������������������������������������������������������������������������������������������������������� 24 Figure 19: Percentage of Classrooms at or Above the Preschool Quality Threshold by Level, 2018-2019 ������������������������������������������������������������������������������������������������������������������������������������������������ 25 Figure 20: Center Leadership and Management Practices at BOY and EOY, 2018-2019 ����������� 26 Figure 21: Comparison of Age Groups over Time ������������������������������������������������������������������������������ 27 Figure 22: Student Retention from BOY to EOY ������������������������������������������������������������������������������� 28 Figure 23: Teacher Retention ��������������������������������������������������������������������������������������������������������������� 29

LIST OF TABLES Table 1: Description of CFSRP Professional Development Levels ����������������������������������������������������� 8 Table 2: EEI and DEI Participation ��������������������������������������������������������������������������������������������������������� 11 Table 3: One-on-One Mentoring Activities ����������������������������������������������������������������������������������������� 12 Table 4: CFSRP Infant and Toddler Development Outcome Goals Versus Actual ������������������������� 13 Table 5: CFSRP Preschool Development Outcome Goals Versus Actual ����������������������������������������� 15 Table 6: PAS Performance – Score Increase and Quality Threshold ������������������������������������������������ 26 Table 7: Number of Centers, Classrooms, Staff, and Children served by the Camp Fire School Readiness Program, 2009-2018 ������������������������������������������������������������������������������������������������������������� 27

Camp Fire School Readiness Program | 2018-2019 Program Evaluation Report

3

EXECUTIVE SUMMARY

School readiness is a multi-faceted concept that refers to the early skills, knowledge, behaviors, and developmental milestones that prepare children to enter and succeed in school. 1,2 Empirical studies suggest that children enrolled in high-quality early education and development programs tend to be more prepared with the skills and knowledge needed to enter and succeed in school.3, 4 The Camp Fire School Readiness Program (CFSRP) was established in 2005 and is designed to improve children’s school readiness by improving the quality of child development centers in target neighborhoods within the Fort Worth Independent School District (FWISD). Using rigorous, comprehensive, research-based programs, the CFSRP provides professional development, coaching and mentoring for directors and teachers, as well as support for family engagement activities. Although comprehensive, the program’s primary focus is on promoting children’s social-emotional and early language/literacy development.

Evaluation Objectives The evaluation includes four objectives that address both the process and the outcomes of implementing the CFSRP in the targeted child development centers.

• To assess the extent to which the CFSRP is implemented with fidelity. This ‘process’ portion of the evaluation was added in the 2018-2019 school year to help Camp Fire staff identify program elements that might need continued and/or increased focus and support.

• To assess the program’s effectiveness in improving children’s developmental skills, particularly early literacy and social-emotional skills.

• To assess the program’s effectiveness in improving center and classroom quality.

• To assess the impact of the program on children’s school readiness and academic success in FWISD schools.

Key Findings: CFSRP Program Implementation • Professional Development enhances teacher and director performance, but participation showed room for improvement. Teachers’ use of Professional Development was below expected levels, but the majority of those who did participate engaged at levels that made them eligible for a full stipend. Directors reported that the professional development opportunities provided by the CFSRP improved their leadership and management practices.

• Mentorship was a valuable support for teachers and directors and CFFT is working to ensure that mentor focus remains on target. Directors rated their mentoring experiences with the CFSRP positively, most

1

Karoly, L.A., Kilburn, M.R., Cannon, J. (2005). Children at risk: Consequences for school readiness and beyond (Research Brief). Santa Monica, CA: RAND Corporation. Retrieved from http://www.rand.org/pubs/research_briefs/RB9144.html.

2

Texas Early Learning Council. (2011). Defining school readiness: National trends in school readiness definitions. Retrieved from http:// earlylearningtexas.org/media/10138/trends%20in%20school%20readiness%20final%2011-1.pdf.

3

Pears, K.C., Fisher, P.A., Kim, H.K., Bruce, J., Healey, C.V., & Yoerger, K. (2013). Immediate effects of a school readiness intervention for children in foster care. Early Education and Development, 24(6), 771-791.

4

Barnett, W.S., & Yarosz, D.J. (2007). Preschool policy brief: Who goes to preschool and why does it matter? Retrieved from http://nieer.org/ policy-issue/policy-brief-who-goes-to-preschool-and-why-does-it-matter-updated.

Camp Fire School Readiness Program | 2018-2019 Program Evaluation Report

4

specifically helping with challenges they may face with staff and helping in areas in which they want support.

When providing support to teachers, mentors spent the majority of Center visit time conducting observations and providing support with instructional planning. Mentors also spent time helping teachers with assessments; while important, the amount of time spent on this activity was more than expected. Anecdotal evidence suggests that mentors were spending time teaching teachers how to administer assessments and then mentors were spending time entering data from assessments and managing the data collection process. The CFSRP will investigate this finding further to determine if more mentor development is needed.

Key Findings: Child Outcomes • The results of the outcome evaluation demonstrated that the CFSRP is having positive results for both infant and toddler groups, but showed some areas for growth with preschool-aged children.

• The majority of infants either improved or continued to perform on target in each domain (i.e., Communication, Fine Motor, Gross Motor, Personal-Social, and Problem Solving).

• Toddlers performed equally well, with all end-of-year goals met for development and a large majority of toddlers improving or continuing to perform on target in each area.

• Preschool children (ages 4 and 5 years old) did not meet goals for development at the end of the year; however, the percentage of 4- and 5-year-olds making acceptable progress at the beginning- and end-ofyear increased across all domains.

• All children, infant through preschool, showed an improvement in social-emotional well-being with an increase in the percentage of children in the “strengths” category, which means the child displayed social and emotional strengths. Infants and toddlers also showed a decrease in the percentage of children in the “needs” category, Children in the “needs” category are in need of instruction to further develop social and emotional skills. Preschool children had a small increase in the “needs” category.

• The CFSRP continues to contribute positively to children’s kindergarten readiness. As in prior years, prekindergarten and kindergarten children entered FWISD with significantly higher ratings than a demographically matched comparison group on language and literacy skills assessed by the Texas Kindergarten Entry Assessment (TX-KEA).

• The CFSRP continues to contribute to children’s academic success through the first grade. In each of the past seven years. FWISD kindergarten and first grade children who had attended one of the CFSRP centers had higher reading and math scores on standardized tests than the demographically similar comparison groups.

• The differences fade as the children progress to the second and third grade. Second grade CFSRP children have had similar math and reading scores as the comparison group on standardized tests four years and third grade CFSRP and comparison group students have also similar reading and math scores on the for the past three years on the State of Texas Assessment of Academic Readiness (STAAR).

Camp Fire School Readiness Program | 2018-2019 Program Evaluation Report

5

Key Findings: Center Outcomes • Classroom-level caregiving, support, and environment varied by age group and Center level. When measuring classroom-level caregiving, support, and environment, infant and toddler classrooms showed growth from beginning- to end-of-year. Preschool classrooms struggled in these areas, showing decreases in classroom scores from beginning- to end-of-year. In some cases, Centers with four or more years of participation in the CFSRP (Level 4 Centers) performed at higher levels than Centers with fewer years of participation.

• Leadership and management quality at CFSRP Centers improved from beginning- to end-of-year. A majority of CFSRP Centers met the performance goals for Staff Development, Program Evaluation, and Family Support and Involvement domains. There was room for growth in the area of Staff Orientation, with a minority of Centers meeting performance goals, and Family Partnerships, with scores coming in just below the “Good Quality” threshold. Again, as expected, Level 4 Centers had the strongest performance and Level 2 had the lowest; but Level 2 Centers also showed growth from beginning- to end-of-year.

Camp Fire School Readiness Program | 2018-2019 Program Evaluation Report

6

INTRODUCTION Program Components The Camp Fire School Readiness Program (CFSRP), established in 2005, is comprised of school readiness programs for infants, toddlers, and preschool-aged children offered at child development centers; professional development for Center teachers and directors; on-site, individualized coaching and mentoring; and family engagement.

The CFSRP is a rigorous, comprehensive, research-based program designed to improve children’s school readiness by improving the quality of child development centers in target neighborhoods that feed into Fort Worth Independent School District (Fort Worth ISD) schools. Although comprehensive, the program’s primary focus is on promoting children’s social-emotional and early language development. In accordance with research documenting the positive impact of quality early education and development programs, the CFSRP is meant to mitigate the social and economic challenges many families living in these communities face which contribute to lower school readiness (e.g., limited transportation, residential transience, and limited access to language-rich environments).

Without the CFSRP, it is estimated that many of the children served would be 12-14 months behind in language and pre-reading skills when they enter school.5 These children are also at risk for being held back as early as kindergarten because they have not mastered important skills necessary for academic success in school.

Levels of Support The CFSRP supports 87 classrooms in 19 licensed, non-residential child development centers that provide care services and early education. The CFSRP engages participating child development centers at four levels of professional development intensity. A child development center’s movement from a lower intensity level (Level 1: Initial Relationship Building) to a higher intensity level (Level 4: Sustainability) is determined by factors such as length of participation in the program, class participation requirements, and center performance and capacity. Table 1 shows the four professional development levels, three of which include professional development support.

Table 1: Description of CFSRP Professional Development Levels Professional

Number

Development Level

of Centers

Description

(Classrooms) Level 1

N/A

Relationship building between CFSRP and the center (does not include

Level 2

5 (25)

Basic (Center participates for one year)

Level 3

10 (45)

Intense (Center participates for three years)

Level 4

4 (17)

Sustainability (Center participation begins after the third, intensity-level

professional development).

year and continues as long as the center remains in the program)

5

Lee, V.E. & Burkam, D.T. (2002). Inequality at the starting gate: Social background differences in achievement as children begin school.

Washington, DC: Economic Policy Institute.

Camp Fire School Readiness Program | 2018-2019 Program Evaluation Report

7

Professional Development (PD) and Stipends Professional development is offered to teachers through Foundational PD and the Early Education Institute (EEI), Both are designed to increase knowledge and skills in techniques that promote child development and classroom management. In addition, the EEI specifically addresses 5 components of reading science cited as critical for effective early literacy instruction.6 Directors attend many teacher sessions, and also participate in the Director’s Institute (DI). The DI is designed to increase knowledge of child development and center business management and leadership practices.

In an effort to promote teacher retention and engagement, full-time teachers who have completed Foundational PD and directors are eligible for a stipend based on attendance and demonstrated competency.

The incentive pay is distributed at the conclusion of the EEI at the end of the program year. Teacher and director fulfillment of the requirements is reviewed prior to payment distribution.

Mentoring On-site individualized coaching is provided by CFSRP mentors7 who hold Bachelor’s or Master’s degrees in child development or early childhood education, have three to five years of experience coaching, mentoring and/or working in early childhood settings, and hold certifications in child development assessments. CFSRP mentors provide coaching visits to Centers based on Center level with Level 2 and 3 Centers receiving more visits than Level 4 Centers. The practice-based coaching sessions with teachers consist of creating Teacher Action Plans with SMART (Specific-Measurable-Attainable-Realistic-Timely) goals based on needs identified from assessments (e.g., using teaching best practices, improving classroom management) and supporting the attainment of the identified goals. The coaching sessions with directors also consist of setting SMART goals based on needs identified from assessments related to center business management and leadership practices and supporting the attainment of the identified goals.

Family Engagement The CFSRP family engagement component is an ongoing collaboration between directors and center staff, which consists of a focus on reciprocal communication between families and center staff, as well as family support and involvement-- a range of activities that allows a child care center to be responsive to family needs, including Parent Cafés8 and Playgroups.

6

National Council on Teacher Quality (2020). Program Performance in Early Reading Instruction retrieved from https://www.nctq.org/ publications/home.

7

The CFSRP established a Mentor Professional Pathway framework which categorizes mentors into four levels based on their existing level of training and experience. For example, Level 1 (Beginning) mentors may need support with close guidance, Level 2 (Developing) mentors may need support with increasing independence, Level 3 (Proficient) mentors may need limited support and can independently enhance the knowledge and skills of others in the profession, and Level 4 (Exemplary) mentors can develop program policies and practices and enhance the knowledge and skills of others in the profession. The CFSRP Director uses the Mentor Professional Pathway framework to monitor mentor needs and promote professional development opportunities. There were 5 mentors during the 2018-2019 school year: one at Level 1, one at Level 2, and 3 at Level 3.

8

Parent Cafés are a type of family meeting/support group that CFSRP has supported partner centers in offering to their parents. Parent Cafés are carefully designed, structured discussions that use the principles of adult learning and family support to help participants explore their strengths, learn about Protective Factors, and create strategies to help strengthen their families. CFSRP has also encouraged the use of play groups as a family support. Play groups provide opportunities for parents and their children to interact together in a planned ‘play activity’ that aligns learning opportunities between school and home. The play groups promote social-emotional development, support parent/child relationships, and encourage parents to interact with other parents in the group. CFSRP also provides information and presentations about community resources to center directors who can then use this information to refer families to supportive services regarding family issues, which is another form of family engagement. The CFSRP uses the Family Engagement Measure from the Program Administration Scale (PAS) to set programmatic goals in this area.

Camp Fire School Readiness Program | 2018-2019 Program Evaluation Report

8

Program Theory of Change A theory of change provides an illustration of a program’s impact pathway—the logical causal change that is expected to occur as a result of program activities. As shown in Figure 1, the CFSRP provides classes focused on increasing knowledge and skills in child development to directors and teachers throughout the program year. In addition, CFSRP mentors, with specialized training in child development and business management, provide on-site coaching for directors and teachers. Finally, the program’s family engagement component focuses on reciprocal communication between families and center staff, as well as family support and involvement focusing on their child’s development. The intended first-order outcomes include improvements in teachers’ classroom practices, directors’ business management and leadership practices, and overall improved center quality. With improvements in classroom and center quality, the program expects to observe improvements in children’s language and socialemotional development, ensuring they are prepared for school.

Figure 1: CFSRP Theory of Change

Camp Fire School Readiness Program

Program Components

Professional Development Classes

On-Site, Individualized Coaching

Family Engagement

Center/Teacher Outcomes (First-Order)

Teachers improve teaching practices

Directors improve leadership and management practices

Improved center and classroom quality

Child Outcomes (Second-Order)

Improved language and social-emotional skills in CFSRP children

CFSRP children are prepared to enter school

CFSRP children demonstrate academic success in school

Camp Fire School Readiness Program | 2018-2019 Program Evaluation Report

9

PROCESS EVALUATION

9

An important component of the CFSRP theory of change is professional development and on-site, individualized mentoring. In order to get a clearer picture of how these components were being implemented in practice and to better understand how these efforts might contribute to student outcomes CNM and the CFSRP implemented new process evaluation procedures to collect additional information about professional development, stipend provision, and mentoring. Process Evaluation determines whether a program is implementing activities as intended and whether those activities resulted in the expected program deliverables or products. The results of the process evaluation will aid Camp Fire in determining which aspects of the program are working as expected, what differences in implementation are appropriate and should be folded into accepted program procedures, and which processes require additional support and clarification in order to operate most effectively.

Professional Development and Stipend/Incentives In 2018-2019, approximately 62% of teachers and 85% of directors participated in required professional development sessions (Table 2). They attended a total of 736 class sessions for a total of 1,447 hours. Staff completed 505 assignments. Of the teachers and directors participating, 65% and 45% received either a full or partial stipend, respectively (Table 2). CFFT plans to continue to evaluate professional development participation moving forward.

Table 2: EEI and DEI Participation PD Type

Number Eligible for PD

Number Attended PD

Percent Participation

Number Receiving Stipend

Percent Receiving Stipend

EEI

111

69

62%

45

41%

DEI

20

17

85%

9

45%

Mentors One-on-One Mentoring Mentor visits were primarily designed to be spent on classroom observation, instructional planning, and reflective follow-up. For the spring semester of the 2018-19 year, mentors performed 410 visits to Centers and classrooms with an average of 1.66 hours per visit for a total of almost 500 hours. Multiple activities occurred during each classroom visit. Table 3 displays one-on-one mentoring activities completed during Center visits.

The most common activities performed during visits were observation (31%), instructional planning (22%), helping with child assessments (16%), and reflective follow-up (13%). The amount of time mentors spent on assisting with assessments was unexpected. The time spent helping with child assessments is a reflection of mentors both spending time teaching teachers how to administer assessments and then spending time entering data from assessments and managing the data collection process. This time is also a reflection of the complex nature of test administration and a need for additional support.

9

A description of the process and outcome evaluation methods is available in Appendix B.

Camp Fire School Readiness Program | 2018-2019 Program Evaluation Report

10

Table 3: One-on-One Mentoring Activities Type of Visit Activity

Number of Visit Percent of Visit Activities Activities

Observation

128

0.31

Instructional Planning

94

0.23

Help with Child Assessments

64

0.16

Reflective Follow Up

54

0.13

Modeling

27

0.07

Side-by-Side Coaching

23

0.06

Problem Solving Generate Options 18

0.04

Total

1.00

408

Director Experiences Directors reported a positive experience with the CFSRP related to professional development and mentoring and the results showed some interesting areas for further investigation. All directors (N=10) reported that the CFSRP invited them to participate in professional development, 6 out of 10 reported that they were eligible for the stipend.

Directors believed teachers were more motivated from the stipend than they were themselves. When the CFSRP administers the teacher survey again in the 2019-2020 school year, this will be a particular area of focus. Directors reported that their Centers were highly engaged with the program and that professional development sessions helped them improve their leadership and management practices.

When asked to rate how much their Director Mentor helped them in certain areas, directors rated all areas of assistance highly. They identified receiving assistance in the areas in which they want support and receiving help with challenges they face with staff as the top two areas of support. Directors rated their level of comfort with their interactions with their mentor and teacher interactions with their mentors highly, with teacher interactions with mentors being rated slightly higher. Charts displaying survey results are available in Appendix H.

Camp Fire School Readiness Program | 2018-2019 Program Evaluation Report

11

CHILD OUTCOMES: DEVELOPMENT AND ACADEMIC READINESS

CFSRP encourages children's skill development, builds literacy, and prepares children to be school ready. This section outlines the development, social-emotional, and academic readiness outcomes for the program.

Age-Appropriate Development: Infants and Toddlers Infants and Toddlers enrolled at CFSRP-supported child development centers are administered the Ages and Stages Questionnaire (ASQ-3) to screen for potential developmental concern. Center staff can then address those concerns with individualized instruction in the classrooms or, for more serious concerns, refer the family to support services.

A description of the ASQ-3 can be found in Appendix C. Developmental concern is defined as scoring below the cut-off in at least one of the five ASQ-3 developmental domains (i.e., problem solving, communication, fine motor, gross motor, and personal-social).

To assess the extent to which the CFSRP centers are successfully improving children’s language/literacy and social emotional skills, CFSRP staff set targets for the percent of children developmentally on-target at the end of the year. The evaluation assesses the extent to which these targets are met. The evaluation also examines the extent to which individual children maintain and/or improve their developmental skills Infants and Toddlers Infants and toddlers were developmentally on target10 across the majority of CFSRP goals (Table 4). When looking at individual change from BOY11 to EOY, the majority of infants and toddlers either improved or continued to perform above their “real” age in each domain (Figures 2 and 3).

Table 4: CFSRP Infant and Toddler Development Outcome Goals Versus Actual ASQ Developmental Domain

% of Children Developmentally On-Target at End-of-Year Infants (N=89) Toddlers (N=180) Target

EOY Actual

Target

EOY Actual

Problem Solving Skills

70%

78%

80%

83%

Communication Skills

65%

63%

80%

84%

Gross Motor Skills

75%

76%

85%

89%

Fine Motor Skills

70%

72%

70%

76%

Personal-Social Skills

70%

75%

80%

88%

10

All CFSRP targets are outlined in Appendix L.

11

Definitions of terms can be found in Appendix B.

Camp Fire School Readiness Program | 2018-2019 Program Evaluation Report

12

Figure 2: Age-Appropriate Infant Development, Change from BOY to EOY, 2018-2019 (N=89)12 Improved/Maintained Age Level Declined

100%

Maintained Below Age Level

90%

Percent of Children

70%

80%

79%

80%

78%

76%

67%

60% 50% 40% 30%

21%

20% 10% 0%

19%

17%

11% 4% Communication

18% 6%

1%

Fine Motor

15% 8%

Gross Motor

Personal-Social

Problem Solving

Figure 3: Age-Appropriate Toddler Development, Change from BOY to EOY, 2018-2019 (N=180) Improved/Maintained Age Level Declined

Maintained Below Age Level

100% 90%

90%

89%

86%

86%

81%

80%

Percent of Children

70% 60% 50% 40% 30% 20% 10% 0%

12

16% 6%

8%

Communication

10% 3% Fine Motor

9%

1%

2%

Gross Motor

Personal-Social

10% 4% Problem Solving

It is expected that some infants and toddlers experience a decline in development in certain areas from BOY to EOY. The term is called “development regression�. When an infant or toddler is developing a new skill, they may regress on a skill they have already developed. https:// babyology.com.au/toddler/learning-and-development/development-regression-in-young-kids-what-is-it-and-when-to-be-concerned/

Camp Fire School Readiness Program | 2018-2019 Program Evaluation Report

13

Age-Appropriate Development: Preschool The CFSRP administered the CPALLS+ assessment to preschool-age children to assess their Listening skills, Rhyming 1 skills (i.e., children’s ability to determine if two words rhyme), Rhyming 2 skills (i.e., children’s ability to independently present a word that rhymes with a given word), and Early Math skills. Rhyming 2 is one of the most challenging skills to master, and many children may not master Rhyming 2 skills until they turn five or six years old.

A description of the CPALLS+ assessment is found in Appendix C.

4- and 5-Year-Olds While percentages for preschool-age children did not reach the expected goals (Table 5), 4- and 5-year-old preschool children showed great gains when looking at scores from BOY to EOY (Figure 4), particularly in Listening and Math with 84% and 94% of children either showing improvements or maintaining acceptable progress.

Table 5: CFSRP Preschool Development Outcome Goals Versus Actual % of Children Developmentally On-Target at End-of-Year

CPALLS+ Developmental Domain

Preschool 4- and 5-Year-Olds (N=56) Target

EOY Actual

Listening

90%

84%

Rhyming I

80%

70%

Rhyming II

60%

48%

Math

95%

94%

Figure 4: CPALLS+, Percentage of 4- and 5-Year-Old Children Improving or Making Acceptable Progress from BOY to EOY, 2018-2019 (N=56) 90%

79%

Percent of Children

80% 66%

70% 60%

48%

50%

43%

40% 30% 20%

32% 18% 9%

10% 0%

Listening Improved

21%

15% 7%

4%

23%

2%

Math

No Change: Making Acceptable Progress

16% 7%

Rhyming No Change: Needs More Assistance

9%

Rhyming 2 Declined

Children’s Social-Emotional Development Center teachers administered the Devereux Early Childhood Assessment (DECA) to infant, toddler, and 3- and 4-year-old (preschool) classrooms at CFSRP-supported centers participating at Level 3 and 4. The DECA identifies whether children’s social-emotional skills need intervention (need), are within typical range for their age (typical), or exceed the typical range (strength). In addition, CFSRP piloted administering the assessment at one Level 2 school.

A description of the DECA is found in Appendix C.

Camp Fire School Readiness Program | 2018-2019 Program Evaluation Report

14

For each age group there was an increase in the percentage of children in the “strengths” category. For infants and toddlers, there was a corresponding decrease in the percent in the “needs” category while preschool children showed a small increase in the “needs” category. The greatest increase in the “strengths’ category was for the infant group (see Figure 5). When looking at change from BOY to EOY, 43% of infants, 32% of toddlers and 23% of preschoolers had an improvement in their DECA score.

Figure 5: Social Emotional Development at BOY and EOY, 2018-2019 100% 90% 80%

10.0% 26.7%

17.6%

21.3%

10.1%

11.3%

70% 56.7%

60% 50% 40%

39.8% 47.2%

71.4%

63.1%

63.3%

30% 20% 10% 0%

33.3%

42.6%

31.5%

18.5%

10.0% BOY

EOY

Infants (N=30)

BOY

EOY

Toddlers (N=108) Strength

Typical

BOY

25.6% EOY

Preschool (N=168)

Need

School Readiness: Academic and Social-Emotional The CFSRP goal is for children to be “ready to learn” when they enter school and to experience continued academic success as they progress from grade to grade. To determine the extent to which the program accomplished this, the CFSRP-FWISD study used results from assessments already in place in the district to follow the CFSRP children as they entered FWISD in either pre-kindergarten or kindergarten and progressed to the 3rd grade.

FWISD assesses pre-kindergarten and kindergarten children at the beginning of each year to identify specific skills that they have (or have not) developed and to plan instruction accordingly. The district continues to assess children’s academic progress with the Iowa Test of Basic Skills (ITBS) at the end of the year for kindergarten, first, and second grade children. For third grade children, academic performance is assessed using the State of Texas Assessment of Academic Readiness (STAAR). Appendix D provides more detail about the assessments and the evaluation method.

For each grade level, the CFSRP-FWISD Study compares the results of these assessments for the children who attended a CFSRP Center in an earlier year and a demographically similar group of children who did not. Appendix I shows the demographic characteristics of the CFSRP children and their comparison groups at each grade level. In each of the graphs presented below, the shaded columns indicate statistically significant differences between the two groups.

Pre-Kindergarten and Kindergarten Early Literacy Skills Children who attended CFSRP Centers in prior years entered pre-kindergarten and kindergarten with higher ratings than their counterparts on beginning-of-year assessments of language and literacy skills (see Figures Camp Fire School Readiness Program | 2018-2019 Program Evaluation Report

15

6 and 7).The textured bars on the graph indicate statistically significant differences. In prior years, the CFSRPFWISD evaluation revealed similar results for both age groups, indicating consistently positive results of the CFSRP on children’s early literacy skills (see Appendix J).

Figure 6: Fall 2019 Comparisons of Pre-Kindergarten Literacy Ratings (CIRCLE Assessment)

Figure 7: Fall 2019 Comparisons of Kindergarten Language and Literacy Ratings (KEA)

Social-Emotional Skills The beginning-of-year assessments for FWISD Pre-Kindergarten and Kindergarten children also rate the children in one or more areas of social competence (see Appendix D for a description of the KEA assessment measures). As shown in Figure 8, the kindergarten CFSRP children had higher ratings than their counterparts in three of the five areas assessed. For pre-kindergarten children, the ratings were high for both groups though higher for the CFSRP children.

Camp Fire School Readiness Program | 2018-2019 Program Evaluation Report

16

Figure 8: Fall 2019 Comparisons of Kindergarten and Pre-Kindergarten Children’s Social Competence

Academic Success Over Time FWISD children who attended one of the CFSRP Centers had higher reading and math scores at the end of their kindergarten and first grade years than the matched comparison group (Figures 9 through 12) These results continue a seven-year trend13 of higher reading achievement for the CFSRP children in their first few years of school.

By second and third grade, however, the differences begin to fade. FWISD 2nd grade children who had attended a CFSRP Center showed similar reading and math scores with their counterparts for the past four years in reading and for the past two years in math. (Figures 13 and 14). FWISD 3rd grade CFSRP children and the comparison groups showed similar STAAR reading and math scores for the past three years (see Figures 15 and 16).

13

The seven-year trend includes three years when the district used the Stanford 10 assessment, rather than the ITBS.

Camp Fire School Readiness Program | 2018-2019 Program Evaluation Report

17

Figure 9: Comparisons of Kindergarten Reading Achievement (CFSRP and Comparison Groups; ITBS 2016-2019)14

Figure 10: Comparisons of Kindergarten Math Achievement (CFSRP and Comparison Groups; ITBS 2016-2019)

14

CFSRP kindergarten children included in these analyses would have attended a CFSRP center either one year earlier as 4-year olds or two years earlier as 3-year olds. (CFSRP N=150 in 2016, 107 in 2017, 108 in 2018,105 in 2019; COMP N=552 in 2016, 739 in 2017, 553 in 2018, 523 in 2019).

Camp Fire School Readiness Program | 2018-2019 Program Evaluation Report

18

Figure 11: Comparisons of First Grade Reading Achievement (CFSRP and Comparison Groups; ITBS 2016-2019)15

Figure 12: Comparisons of First Grade Math Achievement (CFSRP and Comparison Groups; ITBS 2016-2019)

15

CFSRP 1st grade children included in these analyses would have attended a CFSRP center either two years earlier as 4-year olds or 3 years earlier as 3-year olds. (CFSRP N= 105 in 2016; 103 in 2017, 98 in 2018, 74 in 2019. COMP N=463 in 2016, 746 in 2017, 431 in 2018, 393 in 2019).

Camp Fire School Readiness Program | 2018-2019 Program Evaluation Report

19

Figure 13: Comparisons of Second Grade Reading Achievement (CFSRP and Comparison Groups; ITBS 2016-2019)16

Figure 14: Comparisons of Second Grade Math Achievement; ITBS 2016-2019)

16

CFSR 2nd grade children included in these analyses would have attended a CFSRP center either three years earlier as 4-year olds or four years earlier as 3-year olds. (CFSRP N=61 in 2016, 80 in 2017, 126 in 2018, 93 in 2019; COMP N=227 in 2016, 473 in 2017, 648 in 2018, 497 in 2019).

Camp Fire School Readiness Program | 2018-2019 Program Evaluation Report

20

Figure 15: Comparisons of 3rd Grade Reading Achievement; STAAR 2017-201917

Figure 16: Comparisons of 3rd Grade Math Achievement; STAAR 2017-2016

17

CFSRP 3rd grade children included in these analyses would have attended a CFSRP center either four years earlier as 4-year olds or five years earlier as 3-year olds. CFSRP N=50 in 2017; 94 in 2018, 79 in 2019; COMP N=290 in 2107, 568 in 2018, 393 in 2019.

Camp Fire School Readiness Program | 2018-2019 Program Evaluation Report

21

CENTER OUTCOMES

CFSRP’s focus on developing teachers and Center directors amplifies the outcomes for children enrolled. Improving teachers’ understanding of child growth and development and improving directors’ ability to support teachers in their growth leads to children who are better prepared to succeed in school. This section outlines the Center-level outcomes for classroom environment and management and Center leadership and management practices.

Improved Quality in Classroom Environment and Management The Classroom Assessment Scoring System (CLASS™) assessment was administered at the beginning and end of the program. The CLASS™ is conducted by trained observers and focuses primarily on the quality of teacher-child interactions in the classroom. In infant classrooms, it is used to measure responsive caregiving (i.e., their ability to respond to and interact with infants during play, routine care and other activities). In toddler classrooms, it measures teachers’ ability to promote intentional, prosocial interactions that encourage children’s capacity for selfregulation and social skills (Emotional-Behavioral Support) and teachers’ ability to promote emerging, expressive language skills in children (Engaged Support for Learning). Finally, the CLASSTM assessment was administered in three- and four-year-old classrooms to measure three domains. The first domain focuses on teachers’ ability to foster positive relationships and respond to children’s emotions or interests (Emotional Support). The second domain focuses on teachers’ ability to set clear behavioral guidelines and maintain a classroom that supports children’s interactions with teachers and peers through the effective management of children’s time, behavior, and attention (Classroom Organization). The third domain focuses on teachers’ ability to help children learn to solve problems, develop more complex language skills, and use feedback to deepen children’s skills and knowledge (Instructional Support). Empirical studies have shown positive child outcomes for children in classrooms with high ratings on the CLASS™ assessment.18, 19Studies have also provided evidence of a threshold effect indicating a minimal level at which classroom quality is met to achieve positive student outcomes.20 For Pre-K CLASS™, the quality threshold is set at 5 for the Emotional Support and Classroom Organization domains, and at 3.25 for the Instructional Support domain. For infants and toddlers, scores of 3 to 5 indicate a mix of effective teacher-child interactions, while scores of 6 to 7 indicate consistently effective teacher-child interactions.

A description of the CLASS™ assessment is found in Appendix C.

CLASS™ Results The CFSRP expected an overall increase from beginning to end of year in each of the CLASS™ domains for infant, toddler and preschool classrooms. As seen in Figures 17 and 18, this expectation was met for infant and toddler classrooms, but not for preschool classrooms.

Center Level differences in infant classrooms were evident. Level 2 Centers showed a decrease in scores (5.03 to 4.89), Level 3 mostly maintained with an increase (5.52 to 5.58), and Level 4 showed an increase (4.96 to 5.75). Center Level differences were more mixed for the toddler age group.

18

Carr, R., Mokrova, I., Vernon-Feagans, Burchinal, M. (2019). Cumulative classroom quality during pre-kindergarten and kindergarten and children’s language, literacy, and mathematics skills. Early Childhood Research Quarterly, 47, 218-228.

19

Vitiello, V. E., Bassok, D., Hamre, B. K., Player, D., & Williford, A. P. (2018). Measuring the quality of teacher–child interactions at scale: Comparing research-based and state observation approaches. Early Childhood Research Quarterly, 44(3), 161-169.

20

Burchinal, M., Vandergrift, N., Pianta, R., & Mashburn, A. (2010). Threshold analysis of association between child care quality and child outcomes for low-income children in pre-kindergarten programs. Early Childhood Research Quarterly, 25, 166-176.

Camp Fire School Readiness Program | 2018-2019 Program Evaluation Report

22

Figure 17: Classroom Environment and Management at BOY and EOY, 2018-2019

Figure 18: Percentage of Classrooms at or Above the Preschool Quality Threshold, 2018-2019

Of concern, of course, are the decreases in average CLASS™ scores for preschool classrooms for the Classroom Organization, Emotional Support, and Instructional Support domains (see Figure 22). Classroom Organization scores decreased to below the quality threshold. Scores in the Instructional Support domain did not reach the quality threshold at BOY or EOY. When considering the percentage of classrooms that met the quality threshold at BOY and EOY, Classroom Organization had the largest decrease (81% to 48%). On a positive note, the percent of classrooms at/above the quality threshold for Instructional Support increased by 10 percentage points (see Figure 18). In reviewing these scores, CFSRP staff discussed the possibility that the decreases in Emotional Support and Classroom Organization could be due to a number of factors. In particular, the increased focus on instructional strategies in the 2018-2019 PD could have resulted in reduced support for other areas of classroom quality. In addition, many centers had to combine age groups (e.g., one class for 3 to 5 year-olds) toward the end of the year. The combined age groups present additional challenges for teachers in terms of age appropriate classroom management. Finally, the largest decrease occurred with Level 2 centers who are teachers new to CLASS™ and not attending EEI classes. Camp Fire is reviewing PD and mentoring strategies to address these issues in the coming years.

Camp Fire School Readiness Program | 2018-2019 Program Evaluation Report

23

The evaluator conducted additional analyses to examine the extent to which these patterns in preschool CLASS™ scores differed by Center Level. Level 2 and 3 Centers had decreases in scores from BOY to EOY, with Level 2 showing larger decreases. Level 4 Centers, those participating in the CFSRP for the longest time, showed improvements or maintained across the three domains (see Figure 19).

Figure 19: Percentage of Classrooms at or Above the Preschool Quality Threshold by Level, 2018-2019

Improved Quality in Center Leadership and Management Practices At the beginning and end of the 2019-2019 program year, a CFSRP mentor used the Program Administration Scale (PAS) to measure leadership and management quality in centers. PAS measures center quality on a 7-point scale and has the following benchmarks: 1 = inadequate, 3 = minimal quality, 5 = good quality and 7 = excellent quality.

Although centers are assessed across ten domains, the CFSRP places an emphasis on two family partnership measures. The Family Communications domain assesses how families are oriented to the center, the variety of methods used to communicate with families, and the frequency of formal conferences. The Family Support and Involvement domain measures the variety of ways that staff support enrolled families and involve them in the life of the center. A description of the PAS assessment can be found in Appendix C. Depending on the PAS measures directors were expected to demonstrate at least a .5- or 1-point increase from the beginning to the end of the year. For all measures, they were expected to achieve or maintain a rating of 5 or higher by the end of the year. See Table 6 and Figure 20 for results.

Camp Fire School Readiness Program | 2018-2019 Program Evaluation Report

24

Table 6: PAS Performance – Score Increase and Quality Threshold PAS Domain

Staff Orientation Staff Development Program Evaluation Family Support and Involvement

Target Increase 0.5 0.5 1.0 0.5

Overall Actual Increase 0.75 2.25 1.09 1.12

Score of 5 or Above No Yes No Yes

Center-Level Performance At Least 1-Point Score of 5 or Increase Above 3 out of 7 (43%) 2 out of 7 (29%) 6 out of 7 (86%) 6 out of 7 (86%) 8 out of 11 (73%) 6 out of 11 (55%) 7 out of 9 (78%) 8 out of 9 (89%)

NOTE: Green shading indicates that the CFSRP met that goal. Two Centers did not have both a BOY and EOY score for comparison for Staff Orientation and Staff Development.

Figure 20: Center Leadership and Management Practices at BOY and EOY, 2018-2019

CFSRP CHARACTERISTICS Center Characteristics Table 7 shows the number of Centers, Center classrooms, staff, and children the CFSRP served from 2009 to 2019.

In Table 3, the number of classrooms is highlighted below because it is the best indicator of the level of investment each program year in terms of both time and money. Mentor capacity and impact on teacher development is influenced by the number of classrooms on their caseload. During the 2018-2019 program year, 19 child development centers participated in the program. Of the 19 child development centers, 5 were Level 2 centers, 10 were Level 3 centers, and 4 were Level 4 centers. See Appendix A for a list of participating centers.

Camp Fire School Readiness Program | 2018-2019 Program Evaluation Report

25

Table 7: Number of Centers, Classrooms, Staff, and Children served by the Camp Fire School Readiness Program, 2009-2018

Student Characteristics 1,177 students Age Group: 24% infants, 36% toddlers, 23% preschool (3 year olds), 17% prekindergarten (4 and 5 year olds), 1% were six years old.

Figure 21: Comparison of Age Groups over Time

Race/Ethnicity: 56% African American, 1% Asian, 21% Caucasian, 14% Hispanic/Latino, 1% Other, 8% Multi-Racial 51% female, 49% male Classroom Level: Of the 87 classrooms in 2018-19, 84% were single-age classrooms. Classroom level was divided into Level 2: Basic (29%), Level 3: Intense (52%), and Level 4: Sustaining (20%).

Retention from BOY to EOY: 74% retained (874 or 1,177)

Camp Fire School Readiness Program | 2018-2019 Program Evaluation Report

26

Figure 22: Student Retention from BOY to EOY

Teacher Characteristics 168 teachers Education: 60% High School Diploma or GED; 26% Some College; 12% Associate’s or Bachelor’s Degree; 2% Not Reported Gender: 98% female, 2% male Race/Ethnicity: 63% African American, 18% Caucasian, 15% Hispanic/Latino, 3% Multi-Racial Center Level: 29% Level 2: Basic, 52% Level 3: Intense, 20% Level 4: Sustaining Child Care Experience: 15% Less than 1 Year, 36% 1-3 Years, 11% 4-6 Years, 12% 7-10 Years, 25% Over 10 Years, 1% Not Reported Length of Employment: Average length of employment was just over 3 years.

Years with CFSRP: 80% of teachers having three years of experience or less at an SRP Center, 16% 4-10+ years, 4% not reported.

Years of Child Care Experience: About 15% had less than one year, 36% had one to three years of experience, 11% 4-6 years, 12% 7-10 years, 25% of teachers had 10 years of experience or more, 1% not reported.

Teacher Wages and Retention A 2013 Texas Early Learning Council report found that child care professionals are paid wages lower than the national average and have few benefits, which has been directly associated with high staff turnover.21 During the 2018-2019 program year, 84% of childcare staff (141 out of 168) earned hourly wages of $7.25 to $13.00 per hour.22 The remaining teachers were paid a salary (15%), were volunteers (1%), or did not have pay information recorded (1%). Percentages total to more than 100% due to rounding.

The turnover rates of early childhood teachers at child development centers are four times higher than the rates observed among elementary school teachers.23 The annual retention rate for center-based child care

21

Texas Early Learning Council. (2013). Texas Early Childhood Workforce Compensation Study. Retrieved from https://www.earlylearningtexas. org/media/23683/texas%20early%20childhood%20workforce%20compensation%20study.pdf.

22

The average wage for a childcare worker in Texas is around $10.74 per hour. https://www.careerexplorer.com/careers/childcare-worker/salary/ texas/.

23

Whitebook, M. & Sakai, L. (2003). Turnover begets turnover: An examination of job and occupational instability among child care center staff.

Early Childhood Research Quarterly, 18, 273-293.

Camp Fire School Readiness Program | 2018-2019 Program Evaluation Report

27

professionals is estimated to be between 60% and 70%.24, 25Of the 168 CFSRP teachers employed during the 2018-19 school year, 132 worked from BOY to EOY for a 79% retention rate (see figure 3).

Figure 23: Teacher Retention

Mentor Characteristics 5 mentors 100% female 80% African American/Black, 20% Caucasian 60% Bachelor’s Degree, 40% Master’s Degree Years of Child Care Experience: 2 out of 5 had 7-10 years, 3 out of 5 had 10 years of childcare experience.

Years of CFSRP Experience: 2 out of 5 had less than one year, 3 out of 5 had one to three years.

24

Whitebook, M. & Sakai, L. (2003). Turnover begets turnover: An examination of job and occupational instability among child care center staff.

Early Childhood Research Quarterly, 18, 273-293.

25

Baumgartner, J.J., Carson, R.L., Apavaloaie, L., & Tsouloupas, C. (2009). Uncovering common stressful factors and coping strategies among child care providers. Child and Youth Care Forum, 38, 239-251.

Camp Fire School Readiness Program | 2018-2019 Program Evaluation Report

28

CONCLUSION

During the 2018-2019 program year, the Camp Fire School Readiness Program served 1,177 children, with 168 Center teachers, 19 directors and 5 assistant directors, in 87 classrooms, across 19 Centers. Twenty-four percent of children served were infants (0-18 months), 36% were toddlers (19-36 months), 23% preschool (3-year-olds), 17% prekindergarten (4- and 5-year-olds). Half (56%) of the children served were identified as African American.

Areas of Strength Process Evaluation Mentor Activities Mentors spent the majority of their time on observation and instructional planning. Directors also had positive feedback regarding their interactions with mentors and the level of type of support they received. Specifically they said that mentors supported them in the areas in which they wanted assistance and helped them deal with staffing challenges they may encounter.

Director Ratings of Professional Development and Stipend Assistance Directors reported the quality of their leadership and management practices improved as a result of professional development sessions and that the professional development stipend was a moderate motivator for participation. Directors especially believed that the stipend motivated teachers to participate in professional development. The CFSRP will get additional feedback from teachers during the 2019-2020 school year.

Child Outcomes Infant and Toddler Development Infants and toddlers displayed age-appropriate developmental skills at EOY and showed growth from BOY to EOY. The majority of infants and toddlers were developmentally on-target in Problem Solving, Communication Gross Motor, and Personal-Social skills. From BOY to EOY, infants and toddlers either improved or continued to perform above their “real” age in each domain. Infants and toddlers also showed growth in social-emotional development. Both infant and toddler groups had more students showing social and emotional strengths at EOY and fewer students showing a need for support with social and emotional skills.

Pre-Kindergarten and Kindergarten: Early Literacy Skills The results of the analyses of the FWISD assessment data indicate that the CFSRP is positively contributing to children’s school readiness, particularly in early literacy. On entry into pre-kindergarten in Fall 2019, CFSRP children had higher ratings than the matched comparison group in all the early literacy skills assessed.

Similarly, kindergarten children who had attended a CFSRP Center had higher ratings than their counterparts in five of the six literacy skills assessed. These findings have been consistent for the past 6 years and strengthen the conclusions about the impact of the CFSRP. Academic Success Over Time: Kindergarten and First Grade The CFSRP also appears to contribute to children’s continued academic success, as measured by standardized assessments, through the first grade. In each of the past seven years, kindergarten children who attended a CFSRP Center had higher reading and math scores on standardized assessments than their Fort Worth ISD peers. In first grade, the CFSRP had higher ratings in alternate years.

Camp Fire School Readiness Program | 2018-2019 Program Evaluation Report

29

Center Outcomes Director Focus on Staff Development When evaluating Human Resources Development at the Centers, Staff Development was a strength of the program. The majority of Centers showed improvement in this area from BOY to EOY and average scores at EOY increased into the “Good Quality” score range.

Family Support and Involvement CFSRP center directors also showed improvement in their efforts to support and engage families. In addition, by the end of the year scores increased into the “Good Quality” range. This is an encouraging finding given the CFSRP focus on the family partnership measures.

Demographics Student and Teacher Retention Student and teacher retention (74% and 79%, respectively) remained high with a large majority of both groups retained from BOY to EOY. Student retention continues to remain relatively stable over time. Teacher retention has increased over time and remains above the national average (60%-70%) for Center-based childcare professionals.

Areas for Growth Process Evaluation Professional Development Participation There was a moderate level of Professional Development participation among teachers and a relatively high level among directors. A smaller group of teachers and directors were eligible for either full or partial scholarships indicating a need for increased participation. These results suggest that the CFSRP should review several aspects of the Professional Development plan including the organization and description of Foundational PD, EEI, and DEI curriculum, eligibility requirements and payout schedule, and accountability.

The CFSRP professional development program is multi-level and it can be difficult to determine what any given Center, director, or teacher should be doing to actively participate. The program may benefit from a simpler model for participation. During the 2018-2019 school year, all teachers were welcome to participate in EEI, while full-time teachers were eligible for the stipend. According to the CFSRP theory of change, strengthening teaching practices improves child outcomes; thus, it may be appropriate for all teachers to benefit from EEI. Foundational PD is available to all teachers; however, more support may be needed. In addition, there are no stated accountability measures for participation. Centers and teachers who are not participated may be counseled by CFSRP staff, but there is not clear procedure for how to hold teachers and directors accountable for a lack of participation in professional development. Anecdotally, CFSRP is aware that transportation and access to childcare is a barrier to regular attendance.

Mentoring While mentors were spending the majority of their Center-visit time observing and assisting with instructional planning, they were also spending an unexpected amount of time helping with assessments. Mentors were spending time teaching teachers how to administer assessments and then spending time entering data from assessments and managing the data collection process. This speaks to the complex nature of test administration and a need for additional support.

The complexity of the mentoring program can lead to a disconnect between CFSRP staff, teacher, and director expectations and how the program operates in reality. Some preliminary findings from the director surveys Camp Fire School Readiness Program | 2018-2019 Program Evaluation Report

30

indicated that some directors were confused about the amount of time and support to expect from mentors.

The CFSRP may benefit from more clearly outlining the purpose and procedure of mentoring to ensure that directors and teachers have a clear understanding of what the mentorship program hopes to accomplish and adding some accountability measures to ensure that each Center receives the prescribed amount of Mentoring support for their level of participation (e.g., 4 hours per classroom per month for Level 2 and Level 3; 2 hours for Level 4).

To address this issue, and to better support teachers and provide additional assistance outside of CFSRP mentor visits, the CFSRP added a teacher mentor position at each Level 4 Center in 2019-2020. A teacher mentor is a teacher who has gone through all three years of EEI and graduated. In addition to teaching their own classes, they support other Center teachers. The CFSRP mentor continues to visit the Center once a month for 2 hours. In 2018-2019 the CFSRP piloted the teacher mentor component at one Center and expanded it to three Centers in 2019-2020. Each mentor teacher has set responsibilities that are developed with CFSRP and Center director input. The Center directors monitors mentor teacher performance and reports whether or not the teacher mentor met the requirements. Mentor teachers receive an incentive for their service.

Child Outcomes Preschool Outcomes (4- and 5-year-old classrooms) The CFSRP did not meet their stated goals for preschool on-target development at EOY for Listening, Rhyming I, Rhyming II, and Math skills for this group. The large majority preschool-age children showed growth or maintained acceptable progress in early literacy and math skills from BOY to EOY, but not to the expected levels. One reason for this may be that in 2017-2018, the CFSRP saw lower infant and toddler scores than expected. In response, the CFSRP increased professional development focus on those classrooms and shifted focus away from the preschool classrooms. This change in infant and toddler performance is suggestive that the professional development the CFSRP provides is effective and that if participation increased, child outcomes would also improve. It also suggests that the CFSRP needs to identify some strategies for increasing a focus on one area while still supporting another.

When assessing social-emotional development, preschool children did show an increase in the percentage displaying social and emotional strengths; however, there was also a small increase in the percentage of preschool-aged children showing a need for further support in these areas.

Pre-Kindergarten and Kindergarten: Social-Emotional Skills The results were mixed in terms of pre-kindergarten and kindergarten children’s school readiness in the area of social-emotional development with the CFSRP children showing higher ratings than their comparison counterparts in some areas (e.g., kindergarten social-emotional skills) but not in other areas (e.g., kindergarten emotion management). CFSRP staff can use these findings to learn more about the social skills that kindergarten teachers consider to be good indicators of school success and to plan their professional development accordingly.

Academic Success Over Time: Second and Third Grade By second and third grade, the differences between the CFSRP children and the comparison group fade.

CFSRP children in the second grade had similar Reading Iowa Test of Basic Skills (ITBS) scores as the comparison group over the last four years. The ITBS math scores have also been similar for the past two years.

Significant differences are also not evident in the third-grade student performance on the State of Texas Assessment of Academic Readiness (STAAR) reading and math assessments. CFSRP and FWISD staff will continue discussions to identify and address potential factors that could be contributing to this apparent fade effect.

Camp Fire School Readiness Program | 2018-2019 Program Evaluation Report

31

Center Outcomes Classroom Environment and Management The CFSRP met its goals for infants and toddler classrooms with these groups showing improvements in CLASS scores from BOY to EOY in the Responsive Caregiver (infants), and Emotional and Behavioral Support and Engaged Support for Learning (toddlers) domains; however, scores indicated a mix of effective teacherchild interactions rather than consistently effective interactions. In preschool classrooms, CFSRP did not meet their goal with CLASS scores decreasing from BOY to EOY for all three domains, Classroom Organization, Emotional Support, and Instructional Support. Classroom organization scores decreased to below the quality threshold and instructional support scores remained below the quality threshold at BOY and EOY. These results could be attributed to the same shift in professional development focus and staff experience as described Preschool Outcomes section above.

Director Focus on Staff Orientation The CFSRP met its goal of having a .5-point increase in PAS scores from BOY to EOY but the average score at EOY did not reach the “Good Quality” threshold. A small group of Centers showed improvement on Staff Orientation procedures from BOY to EOY and few Centers scored in the “Good Quality” range at EOY. The areas assessed included orientation policy, procedures, and process. Ramping up to and beginning each school year is an incredibly busy time. These findings suggest that CFSRP Centers are making progress with regards to Staff Orientation but have more work to do.

Program Planning and Evaluation The CFSRP met its goal of having a 1-point increase in PAS scores from BOY to EOY but the average score at EOY did not reach the “Good Quality” threshold. This domain includes staff evaluation of the program, parent evaluation of the program, and use of evaluation information. Program evaluation is a time consuming and difficult task that requires some training. It is possible that CFSRP Center directors did not have the resources necessary to adequately accomplish this task.

Recommendations The following recommendations are based on the findings of the 2018-2019 evaluation.

Professional Development: • Simplify the professional development program design so that it is easier for Center teachers and directors to understand and follow requirements.

• Continue to investigate the influence of a stipend of teacher and director professional development participation.

• Increase the requirements of professional development participation – move all teachers (both FT and PT) through PD. • Add accountability measures to ensure teacher and director participation.

• Add additional supports such as transportation and childcare assistance to increase participation.

Mentoring: • Formalize the mentoring requirements and implement accountability measures (i.e., further investigating the amount of mentoring and type of mentoring activities completed and what guidelines the CFSRP should adopt to reflect best practices in the area).

• Provide additional training and support for teacher-mentors. For example, providing targeted mentoring for teachers in multi-age classrooms.

Camp Fire School Readiness Program | 2018-2019 Program Evaluation Report

32

Child Outcomes: • Target preschool outcomes in professional development sessions. In addition, mentors should pay particular attention to preschool classrooms while out doing Center visits. Take caution that the increased focus on preschool outcomes does not detract from any necessary focus on infant and toddler outcomes.

• Further investigate Center Level differences in child outcomes and apply professional development and mentoring support where needed to maintain positive results over time.

• Continue use of social emotional tools and professional development that inform both teacher and family interactions with children (i.e., DECA and related support activities).

Center Outcomes: • Continue to focus on classroom environment and management in infant, toddler, and preschool classrooms.

In particular, mentors can focus on the specific skills addressed in the CLASS™ observation during their Center visits. The CFSRP should determine whether specific professional development sessions focused on each domain would be beneficial.

• Provide professional development and oversight to Center directors related to staff orientation, program evaluation, and family communication. Determine if specific resources may be beneficial. In addition, mentors should target these areas when working directly with directors at their Centers.

• Further investigate Center Level differences in Center outcomes and apply professional develop and mentoring support where needed to maintain positive results over time.

• Further investigate the impact of Universal Preschool and multi-age classrooms and create appropriate PD and mentoring responses to that information Evaluation Procedures: • Continue to increase and improve the tracking of process evaluation outputs.

• This includes increasing data entry compliance with teacher and Center identification numbers and tracking teacher full-time or part-time status.

• Continue to focus on administering the CFSRP-identified assessments at the set time of year, regardless of impact on match status.

• Further investigate Center Level differences in child and Center outcomes and apply professional development and mentoring support where needed to maintain positive results over time.

Camp Fire School Readiness Program | 2018-2019 Program Evaluation Report

33

APPENDICES

 

APPENDIX A: LIST OF PARTICIPATING CHILD DEVELOPMENT CENTERS List of Child Development Centers

Camp Fire School Readiness Program | 2018-2019 Program Evaluation Report

34

APPENDIX B: METHODS26 The CFSRP evaluation consists of both a process and outcome evaluation. The process evaluation component was added for the 2018-2019 school year in order to get a clearer picture of how the CFSRP was being implemented in practice and determine to what extent the program was operating as designed in the theory of change. This included a specific focus on professional development participation and stipend allocation, mentor activities, and staff experiences. Mentoring activity was not tracked until January 2019, so only data for half of the 2018-2019 school year is available for this report. In order to gather more information about program implementation and to gain additional insight into potential areas for program updates, the CFSRP and CNM administered surveys to teachers, directors, and mentors in October and November of 2019. The surveys solicited information about retention, professional development, Center culture, and mentoring experience. Response rates were 14% (5 out of 35), 53% (10 out of 19), and 100% (5 out of 5), respectively. Because teacher response rates were so low, results from that survey are not included here. Because there were only five mentors, the CFSRP used their responses for internal decision-making and their responses are not report here. The CFSRP plans to administer the teacher survey again in 2019-2020 to gain more insight into teacher experiences. Please contact the CFSRP to request copies of the teacher, director, and mentor surveys.

In order to better describe the CFSRP student, teacher, and mentor population, the CFSRP also collected additional data related to student enrollment and retention, teacher retention and experience, and mentor workload and experience. This information has been added to the CFSRP Characteristics section of the report.

The outcome evaluation continued from previous years and included a focus on child outcomes (developmental, academic, and social-emotional) and Center outcomes (classroom environment and management, Center leadership and management). The CFSRP outcome evaluation also includes a comparative analysis of outcomes for CFSRP children and demographically similar groups non-CFSRP children enrolled in Fort Worth Independent School District (FWISD). This part of the evaluation uses assessments already in place in to compare the performance of the CFSRP children and the comparison groups when they enter pre-kindergarten and at the end of their kindergarten through third grade years.

Evaluation Questions To what extent did the CFSRP: 1. Improve the percentage of children demonstrating age-appropriate developmental, early literacy, and social-emotional skills during the 2018-2019 program year?

2. Impact children’s growth in developmental, early literacy, and social-emotional skills during the 2018-2019 program year?

3. Enhance the quality of teaching, classroom management, and centers’ family engagement practices during the 2018-2019 program year?

4. Impact CFSRP children’s school readiness as they enter pre-kindergarten and kindergarten?

5. Impact CFSRP children’s academic achievement as they continue through school?

6. Implement professional development, stipend allocation, and mentorship activities as intended?

26

Assessment tools and data analysis procedures are described in Appendix C and D. For a discussion of limitations and considerations, please see Appendix E.

Camp Fire School Readiness Program | 2018-2019 Program Evaluation Report

35

Terms • BOY: The abbreviation BOY refers to the beginning-of-the-year assessment scores which are usually obtained between October and November.

• MOY: The abbreviation MOY refers to the middle-of-the-year assessment scores which are usually obtained between January and February.

• EOY: The abbreviation EOY refers to the end-of-the-year assessment scores which are usually obtained between April and June.

• Assessment Fidelity: Measures the extent to which an assessment was implemented as intended. The CFSRP and CNM created inclusion criteria, whether or not a child’s assessment scores would be included in the evaluation, that included 1) which assessment should be given for a specific child based on their age, and 2) if the child had both BOY and EOY scores for the selected age-appropriate assessment.27

27

Assessment fidelity is outlined in Appendix G.

Camp Fire School Readiness Program | 2018-2019 Program Evaluation Report

36

APPENDIX C: CHILD DEVELOPMENT AND CENTER QUALITY: ASSESSMENT TOOLS AND DATA ANALYSIS The CFSRP contracts with CNM to provide program evaluation consulting services and CNM pact outcomes services. Program data collection activities occurred at the beginning of the school year (October 2018 to November 2018) and at the end of the school year (April 2019 to June 2019). CNM created secure, web-based online data entry spreadsheets for each CFSRP child development center. Directors at CFSRP-supported child development centers entered student and teacher enrollment information, classroom information, and assessment data. The Center on Research and Evaluation (CORE) administered the Classroom Assessment Scoring System (CLASS™) and submitted the final data to the CNM evaluation team to analyze. Other assessment data were provided by the CFSRP Director.

Prior to data analysis, CFSRP staff reviewed and cleaned final data. The table below presents each assessment and its associated assessment areas.

Assessment

Area Assessed

Data Analysis

Ages and Stages Questionnaire, Version 3 (ASQ-3)

Cognitive and Physical Development

The evaluation team calculated the percentage of children meeting the cut-off for developmental skills in five domains at the beginning and end of the year. The results were disaggregated by age group.

A standardized, screening tool designed to identify infants and young children who are and are not displaying typical age-appropriate development. CFSRP recommends that children ages three years and five months or younger receive the ASQ-3 assessment.

Language and Literacy Development

Circle Phonological Awareness, Language and Literacy Screener plus Math (CPALLS+)

Language and Literacy Development

A standardized, criterion-referenced assessment designed to measure children’s literacy and language skills. CPALLS+ recommends that children ages three years and six months or older receive the CPALLS+ assessment.

Cognitive Development (Math)

Devereux Early Childhood Assessment (DECA)

Social-Emotional Development

A strengths-based, standardized assessment and planning system that supports educators in promoting children’s social and emotional development, thus promoting resilience.

The percentage of children demonstrating improvement in developmental skills from the beginning to the end of the year was also calculated.

The evaluation team calculated the percentage of children meeting the cut-off for language and literacy skills at the beginning and end of the year. Separate analyses were conducted for three-year-old children (MOYEOY comparisons) and four-and five-yearold children (BOY-EOY). The results were disaggregated by age group.

The percentage of four- and five-year old children demonstrating improvement in developmental skills from the beginning to the end of the year was also calculated.

The evaluation team calculated the percentage of children who scored in the Typical or Strength category in socialemotional/resilience at the beginning and the end of the year. The percentage of children demonstrating improvement in their scores was also calculated.

Camp Fire School Readiness Program | 2018-2019 Program Evaluation Report

37

Assessment

Area Assessed

Data Analysis

Classroom Assessment Scoring System (CLASS™)

Classroom Management and Quality

The evaluation team included only teachers with matched pre- and post-assessment scores in the analysis. The average CLASS™ pre- and post-assessment scores were compared.

Quality in Teaching Practices

Based on the design of the BPOT assessments, the evaluation team calculated the total observations and created a weighted system that categorized scores as ‘needs support’, ‘emerging’, and ‘consistently meets’. The results were disaggregated by domain.

A standardized, observation-based assessment designed to assess classroom management and quality on a 7-point scale. The Infant CLASS™ measures the quality of responsive caregiving in infant classrooms. The Toddler CLASS™ measures the quality of emotional and behavioral support and engaged support for learning in toddler classrooms. The Pre-K CLASS™ measures the quality of emotional support, classroom organization, and instructional support. For Pre-K CLASS™, the quality threshold is set at 5 for the Emotional Support and Classroom Organization domains, and at 3.25 for the Instructional Support domain.

Best Practices Observation Tool (BPOT) A research-based observational checklist that measures the presence or absence of researchbased teaching practices that align with CFSRP professional development curriculum. This tool is intended for professional development purposes. Teachers in infant classrooms are rated on 105 best-practice teaching strategies, and teachers in toddler classrooms are rated on 110 best-practice teaching strategies. The BPOT for three-year-old classrooms and fouryear-old classrooms includes 110 and 120 bestpractice teaching strategies, respectively

Program Administration Scale (PAS) A 25-item research-based instrument that measures the quality of leadership and management practices of early childhood programs. PAS measures quality on a 7-point scale (1 = inadequate, 3 = minimal, 5 = good, 7 = excellent).

This assessment is used internally to assist mentors with identifying target areas for teacher development.

Center Leadership and Management Quality

The evaluation team used each center’s individual score to calculate an overall average for each of the 10 domains.

Camp Fire School Readiness Program | 2018-2019 Program Evaluation Report

38

APPENDIX D: SCHOOL READINESS AND ACADEMIC SUCCESS: ASSESSMENT TOOLS AND DATA ANALYSIS Camp Fire partners with Fort Worth ISD to gain access to data from student assessments being used in the school district. Through this partnership, the CFSRP evaluation team has been able to assess the impact of the program for seven years with statistical analyses of the children’s assessment scores not only as they enter school but also through their kindergarten, 1st, 2nd and 3rd grade years. Camp Fire contracted with AWare Research Solutions to provide evaluation and data analysis services.

Each year, using a modified propensity matching process, the evaluation team compares the assessment scores of children who attended a CFSRP-supported center in one of the prior seven years and children in -demographically similar comparison groups who did not attend one of the centers. This technique allows the evaluation team to create a random, non-biased sample of children who are similar to the sample of CFSRP children and, in turn, make valid comparisons between the two groups. Any statistically significant differences identified in the results provide evidence that the differences between CFSRP children’s scores and the comparison group’s scores can be attributed, in part, to the CFSRP program rather than to random chance.

The groups were matched on the following characteristics: • • • • • •

School location Grade level Ethnicity Free/Reduced Lunch Status Gifted Status Gender

In order to get the best possible match, the comparison groups are necessarily larger than the CFSRP groups.

Students with Special Education designation are not included in the analyses. (see Appendix I for the demographic descriptions of the CFSRP and comparison groups). Depending on the grade level for each set of comparison groups, the analyses were conducted with the FWISD assessment data described in the table on the next page.

Because very few CFSRP children are assessed in Spanish only the English versions of each assessment were included in the analyses. For the same reason, children with LEP status and Special Education status were not included in the analyses.

Camp Fire School Readiness Program | 2018-2019 Program Evaluation Report

39

APPENDIX D (Continued) FWISD Assessments Used in the Evaluation Grade Level

Assessment Description

PRE-KINDERGARTEN READINESS

Circle Progress Monitoring Tool (CIRCLE)28 CIRCLE is similar to the CPALLS+ assessment used in the CFSRP three and four-year old classrooms. It is a criterion-referenced assessment based on the Texas Education Agency (TEA) pre-kindergarten guidelines for literacy, math and social skills. Teachers use CIRCLE at the beginning of the school year to help identify children who meet or do not meet developmental benchmarks so they can plan individualized instruction.

KINDERGARTEN READINESS

Texas Kindergarten Entry Assessment (TX-KEA)29 TX-KEA is a screening tool designed to assess kindergarten children’s skills in six areas of school readiness. As with the CIRCLE assessment, the primary purpose of the TX-KEA is to identify children who may need additional support and to help teachers plan individualized instruction.

Texas school districts and charter schools are required to administer a Kindergarten assessment instrument for all children enrolled in Kindergarten. The TX-KEA is on the 2017-2021 Commissioner’s Approved List of Kindergarten Assessment Instruments for meeting this requirement. The 2018-19 school year is the second year FWISD administered the TX-KEA. The tool assesses several skills not assessed in prior years. Appendix K provides description of the measures used in the analyses for this evaluation.

KINDERGARTEN, 1ST AND 2ND GRADE ACADEMIC ACHIEVEMENT

Iowa Test of Basic Skills (ITBS) .30 The ITBS is a nationally normreferenced test that assesses Reading and Math academic performance in relation to a normed sample of same-aged children. The 2018-19 school year is the third year FWISD has administered the ITBS.

3rd GRADE READING AND MATH ACHIEVEMENT

State of Texas Assessment of Academic Readiness (STAAR)31 STAAR is the state of Texas criterion-referenced accountability assessment that measures whether children have met and/or not met state-mandated curriculum standards, defined in the Texas Essential Knowledge and Skills (TEKS).

28

CLI Engage (2017). CIRCLE Progress Monitoring System. https://cliengage.org/public/tools/assessment/circle-progress-monitoring/ .

29

Texas Kindergarten Entry Assessment (TX-KEA).https://www.texaskea.org/.

30

Houghton Mifflin Harcourt (2017). Iowa Test of Basic Skills (ITBS). https://www.hmhco.com/programs/iowa-assessments/overview.

31

Texas Education Agency (2018). The State of Texas Assessments of Academic Readiness (STAAR). https://tea.texas.gov/student.assessment/ staar/.

Camp Fire School Readiness Program | 2018-2019 Program Evaluation Report

40

APPENDIX E: CHILD DEVELOPMENT AND CENTER QUALITY: LIMITATIONS AND CONSIDERATIONS As with any evaluation, there are several considerations and limitations to reflect on when interpreting the findings presented in this report. First, the Ages and Stages Questionnaire (ASQ-3) is a screener for developmental delays that is currently used by CFSRP as a diagnostic and assessment tool. As such, it is not specifically designed to be an evaluation tool. The CFSRP and evaluation team recognizes this limitation of ASQ-3 and continues to use it due to the limited availability of assessments that can be used for evaluative purposes.

 

Camp Fire School Readiness Program | 2018-2019 Program Evaluation Report

41

APPENDIX F: FREQUENTLY ASKED QUESTIONS 1. What does it mean when the report indicates that CFSRP children were compared with a demographically similar group?

The FWISD comparison group was chosen using a modified propensity score matching process. Propensity matching is a systematic technique used identify a control group of demographically similar children for statistical comparison. This technique allowed the evaluation team to control for difference in children’s demographic backgrounds (e.g., race/ethnicity, gender, socio-economic status), which could potentially skew the results. In using this technique, the evaluation team was able to make valid comparisons of the CFSRP children’s academic outcomes with the outcomes of their Fort Worth ISD peers.

2. What are the strengths and limitations of propensity matching?

Although this process provides valid comparisons between the CFSRP student group and the FWISD comparison groups, there are some limitations to the comparisons. First, and likely due to mobility, the CFSRP groups get smaller as the children progress from kindergarten through the first, second, and third grade. Necessarily, the comparison groups also get smaller. With smaller sample sizes, it is more difficult to make inferences. Therefore, interpret results from the higher grade-level comparisons with caution.

3. What evaluation design is used for the Camp Fire School Readiness Program?

The evaluation team uses a cross-sectional design rather than a longitudinal design to evaluate the Camp Fire School Readiness Program. A cross-sectional design involves analyzing data from a sample at a specific time point (e.g., year-by-year snapshot) and a longitudinal design involves analyzing trends in data gathered from the same sample of children over time. For the evaluation of children’s school readiness and academic success in FWISD, a cross-sectional design is used because the CFSRP groups get smaller each year, the demographic make-up of the group changes, and a different comparison group is necessary.

4. What is statistical significance and why is it used in the evaluation?

Statistical significance is a calculation that compares two or more results measures to determine if a difference or relationship truly exists. It is used in the evaluation when comparing the FWISD outcomes of the CFSRP children and the control group in order to ensure that any identified differences between the groups are reliable. The statistically significant differences identified with these analyses provide evidence that the differences between CFSRP children’s scores and the comparison group scores are likely the result of the CFSRP program rather than random chance.

Camp Fire School Readiness Program | 2018-2019 Program Evaluation Report

42

APPENDIX G: CHILD DEVELOPMENT AND CENTER QUALITY: ASSESSMENT FIDELITY The term assessment fidelity refers to the extent to which assessments were administered to children of the appropriate age based on CFSRP criteria. Assessment fidelity is a measure of quality and accuracy, and is critical to obtaining desired outcomes. Each assessment, specifically the ASQ-3 and CPALLS+, should be administered to children based on their age. Also, if a child is administered one assessment at BOY, they should receive the same assessment at EOY.

An example of how these procedures were applied would that children were removed from the sample if they were given the ASQ-3 even though they were older than three years and five months at the time of the assessment.

For example, if a child’s birthdate was January 6, 2015 and they were assessed on September 18, 2018, then that child was three years and eight months old and should have been assessed using the CPALLS+ according to CFSRP assessment rules.

Number of Children Assessed and Appropriately Assessment

# unique assessments

# children receiving appropriate assessment based on age

% children receiving appropriate assessment based on age

% matched from BOY to EOY

Child

3 year, 5 months or younger by September 1 of program year

705

609

86%

46%

CPALLS+

Child

3 years, 6 months or older

by September 1 of program year

305

180

59%

52%

DECA

Child

NA

705

705

100%

43%

CLASS™

Classroom

NA

79

NA

NA

86%

ASQ-3

Level of Measurement

Age

Note: Percentages are rounded up to the nearest whole number.

Camp Fire School Readiness Program | 2018-2019 Program Evaluation Report

43

APPENDIX H: Director Survey Results Director Ratings of Professional Development and Stipend, 2018-2019

Director Ratings of Director Mentor, 2018-2019

NOTE: 1=Not at All, 5=To a Great Extent

Camp Fire School Readiness Program | 2018-2019 Program Evaluation Report

44

Director Ratings of Comfort with Mentor Interactions, 2018-2019

NOTE: 1=Uncomfortable, 5=Very Comfortable

 

Camp Fire School Readiness Program | 2018-2019 Program Evaluation Report

45

APPENDIX I: CFSRP and Comparison Group Demographics (FWISD, 2019-2020)

Assessment and Grade Level Demographic Description

Ethnicity

African American/ Black

Hispanic

Group

Spring 2019 KinderITBS

Spring 2019 1st Grade ITBS

Spring 2019 2nd Grade ITBS

Spring 2019 3rd Grade STAAR

Fall 2019 Pre -K CIRCLE

Fall 2019 Kinder KEA

CFSRP

38.1%

43.2%

40.9%

39.2%

63.6%

51.5%

COMP

40.0%

43.0%

36.4%

40.7%

62.1%

49.7%

CFSRP

29.5%

31.1%

32.3%

20.3%

25.5%

29.3%

COMP

33.5%

29.8%

33.4%

25.2%

23.2%

29.7%

CFSRP

21.9%

18.9%

17.2%

30.4%

6.4%

18.2%

COMP

20.5%

20.6%

21.3%

33.1%

7.2%

19.5%

CFSRP

10.0%

6.8%

8.7%

10.1%

4.5%

1.0%

COMP

6.0%

6.6%

8.9%

0.0%

7.5%

1.1%

CFSRP

70.5%

71.6%

60.2%

63.3%

90.0%

75.8%

COMP

72.5%

72.3%

61.0%

60.8%

89.9%

75.1%

CFSRP

29.5%

28.4%

39.8%

36.7%

10.0%

24.2%

COMP

27.5%

27.7%

39.0%

39.2%

10.1%

24.9%

CFSRP

50.5%

50.0%

38.7%

34.2%

49.1%

51.5%

COMP

49.1%

49.4%

39.2%

41.5%

48.5%

50.6%

CFSRP

49.5%

50.0%

61.3%

65.8%

50.9%

48.5%

COMP

50.9%

50.6%

60.8%

58.5%

51.5%

49.4%

CFSRP

6.0%

17.5%

25.8%

25.3%

NA

NA

COMP

4.2%

16.2%

25.7%

21.2%

NA

NA

White

Socio-Economic Status

Other

Economic Disadvantage

Not Economic Disadvantage

Gender

Male

Gifted

Female

Gifted

Camp Fire School Readiness Program | 2018-2019 Program Evaluation Report

46

APPENDIX J: YEAR-TO-YEAR COMPARISONS OF PRE-KINDERGARTEN AND KINDERGARTEN BEGINNING OF YEAR ASSSESSMENT RESULTS

FWISD Pre-Kindergarten CIRCLE Assessment Ratings: CFSRP Students and Comparison Group (Fall 2015 – Fall 2019)32 Fall 2015

Fall 2016

Fall 2017

Fall 2018

Fall 2019

CFSRP (n=103)

Comp. (n=520)

CFSRP (n=73)

Comp. (n=262)

CFSRP (n=88)

Comp. (n=414)

CFSRP (n=115)

Comp. (n=452)

CFSRP (n=92)

Comp. (n=456)

Letter Naming

62%

40%

47%

34%

39%

14%

42%

26%

46%

32%

Vocabulary

65%

52%

58%

39%

44%

32%

48%

49%

54%

48%

60%

60%

51%

49%

78%

66%

64%

63%

61%

59%

Syllabication

19%

18%

22%

8%

16%

12%

15%

10%

Rhyming II

19%

19%

20%

9%

12%

8%

14%

5%

Listening

54%

46%

37%

28%

32%

31%

30%

24%

21%

20%

99%

95%

95%

96%

99%

94%

Phon.

Awareness

Social Emotional

FWISD Kindergarten KEA Assessment Ratings CFSRP Students and Comparison Groups (Fall 2018 & Fall 2019) KEA ASSESSMENT ITEM Language Literacy

Executive Function

Social Emotional

32

Fall 2018

Fall 2019

CFSRP

Comp.

CFSRP

Comp.

Vocabulary

78.8%

74.6%

65.6%

60.2%

Listening Comp.

73.2%

71.9%

72.6%

63.0%

Letter Names

83.3%

73.0%

81.4%

67.1%

Letter Sounds

82.5%

68.3%

71.6%

58.5%

Blending

67.3%

66.1%

33.7%

32.5%

Spelling

96.3%

91.3%

79.3%

63.6%

Working Memory

66.3%

68.5%

56.5%

61.6%

Inhibition

55.9%

62.0%

63.7%

60.0%

Attention

72.0%

66.2%

74.4%

63.6%

Social Emotional

82.0%

84.1%

82.4%

77.9%

Emotional Management

78.7%

74.2%

68.5%

77.4%

Green cells highlight average ratings that are 5 or more higher for one group than the other. Patterned cells indicate statistically significant differences).

Camp Fire School Readiness Program | 2018-2019 Program Evaluation Report

47

APPENDIX K: DESCRIPTION OF TEXAS KINDERGARTEN ENTRY ASSESSMENT MEASURES33 LANGUAGE DOMAIN Vocabulary is a foundational language skill that supports learning in all content domains. Knowing a student’s vocabulary abilities helps teachers adjust their own vocabulary usage during instruction to levels that are most beneficial for individual children.

Listening comprehension is a foundational language skill that supports learning in all content areas. TX-KEA assesses the student’s ability to understand verbal information and follow directions. Knowing your student’s receptive language abilities will allow you to adjust the complexity of your own language during instruction.

LITERACY DOMAIN The Letter Names subtest assesses children’s knowledge of the names associated with various letters of the alphabet. Letter names is one component of letter knowledge which is an excellent predictor of reading achievement The Letter Sounds subtest assesses children’s knowledge of letter sounds. Letter knowledge at kindergarten entry is a strong predictor of literacy achievement The Blending Sounds subtest assesses children’ phonological awareness, or sensitivity to the sound structure of oral language. Phonological awareness is necessary for learning to read and write and is predictive of literacy achievement.

The Spelling subtest assesses children’s early spelling abilities, which is the ability to use sound-symbol relationships to write words. TX-KEA assesses spelling because it is highly related to later literacy achievement.

Attempting to spell words requires children to apply multiple literacy skills simultaneously, such as alphabet knowledge and phonological awareness.

EXECUTIVE FUNCTIONING The Executive Functioning domain addresses the cognitive skills used by children to plan, problem solve, and follow classroom rules.

Inhibition. Students are asked to respond accurately to a specific stimulus (e.g., butterfly), and withhold, or inhibit, a response to a different stimulus (e.g., bee). Scores reflect the student’s ability to respond accurately while inhibiting a response Working Memory. Students are assessed on their ability to hold in memory 1 – 3 pieces of information in an increasingly complex setting. In this subtest, children recall where cars are parked in a garage. The number of cars and the number of parking spaces increases as the student progresses.

Attention. Students are assessed on their ability to focus their attention, stay on task, as well as quickly and accurately focus on relevant features of the task. They are provided 2 minutes to make as many correct matches as possible between the target object, a flower, and 5 answer choices, other flowers.

SOCIAL-EMOTIONAL COMPETENCE AND EMOTION MANAGMENT The Social and Emotional Competence subtest focuses on children’s social and emotional skills within a classroom setting. The subtest evaluates children’s pro-social skills, approaches to learning, and emotion understanding.

The Emotion Management subtest focuses on children’s ability to manage their emotions and respond appropriately to an emotional experience. They are evaluated on whether they can adapt to the demands of a classroom and school environment.

33

This description is taken from the Texas Kindergarten Entry Assessment User Guide at: https://cliengage.org/user-guides/User_Guide_TXKEA_8.13.2018.pdf.

Camp Fire School Readiness Program | 2018-2019 Program Evaluation Report

48

APPENDIX L: 2018-2019 CFSRP Target Outcomes

Center Quality

Classroom Quality

Child Development

School Readiness and Academic Success

Improved quality in center leadership and management practices • Directors will demonstrate at least a .5 increase in their average Staff Orientation (PAS) score or achieve/ maintain a 5 or higher by the end of the year.

• Directors will demonstrate at least a .5 increase in their Staff Development (PAS) score or achieve/maintain a 5 or higher by the end of the year.

• Directors will demonstrate at least 1 point increase in the Program Evaluation (PAS) score or achieve/maintain a 5 or higher by the end of the year.

• Directors will demonstrate at least a .5 increase in their average Family Support and Involvement (PAS) score or achieve/ maintain a 5 or higher by the end of the year.

Improved use of best practice teaching strategies • 85% of teachers will demonstrate improved best practice ratings on the Best Practice Observation Tool (BPOT) by the end of the program year.

Infants will demonstrate age-appropriate developmental and social-emotional skills • 70% of infants will be developmentally on-target in problem solving skills (ASQ) by the end of the year.

• 65% of infants will be developmentally on-target in communication skills (ASQ) by the end of the year.

• 75% of infants will be developmentally on-target in gross motor skills (ASQ) by the end of the year.

• 70% of infants will be developmentally on-target in fine motor skills (ASQ) by the end of the year.

• 70% of infants will be developmentally on target in personal-social skills (ASQ) by the end of the year.

• % of infants demonstrating social-emotional resilience will increase by the end of the year as demonstrated by an increase in the % of infants in the “Typical” and “Strength” categories (combined) on DECA from BOY to EOY. • Establish baseline for % of infants who increase T-score (raw score) on DECA from BOY to EOY. Toddlers will demonstrate age-appropriate developmental and social-emotional skills • 80% of toddlers will be developmentally on-target in problem solving skills (ASQ) by the end of the year.

• 80% of toddlers will be developmentally on-target in communication skills (ASQ) by the end of the year.

• 85% of toddlers will be developmentally on-target in gross motor skills (ASQ) by the end of the year.

• 70% of toddlers will be developmentally on-target in fine motor skills (ASQ) by the end of the year.

• 80% of toddlers will be developmentally on-target in personal-social skills (ASQ) by the end of the year.

• % of toddlers demonstrating social-emotional resilience will increase by the end of the year as demonstrated by an increase in the % of infants in the “Typical” and “Strength” categories (combined) on DECA from BOY to EOY. • Establish baseline for % of toddler who increase T-score (raw score) on DECA from BOY to EOY.

Children will enter school prepared and demonstrate academic success in FWISD. Pre-Kindergarten and Kindergarten Readiness

Improved quality in classroom environment and management • Infant classrooms will demonstrate an increase in average ratings in responsive caregiving (CLASS) scores by the end of the year.

• Toddler classrooms will demonstrate an increase in average ratings in engaged support and emotional-behavioral support (CLASS) scores by the end of the year.

• Preschool classrooms will demonstrate an increase in average ratings in classroom environment, emotional-behavioral support, and instructional support (CLASS) scores by the end of the year.

Preschool-age children will demonstrate age-appropriate literacy and social-emotional skills • 90% of preschool-age children will be developmentally ontarget in listening skills (CPALLS+) by the end of the year.

• 80% of preschool-age children will be developmentally ontarget in rhyming I skills (CPALLS+) by the end of the year.

• 60% of preschool-age children will be developmentally ontarget in rhyming II skills by the end of the year.

• 95% of preschool-age children will be developmentally ontarget in early math by the end of the year.

• % of preschool-age children demonstrating socialemotional resilience will increase by the end of the year as demonstrated by an increase in the % of infants in the “Typical” and “Strength” categories (combined) on DECA from BOY to EOY. • Establish baseline for % of preschool-age children who increase T-score (raw score) on DECA from BOY to EOY.

• CFSRP children will enter school better prepared in language and literacy compared to their Fort Worth ISD peers.

• CFSRP children will enter school better prepared in social-emotional skills compared to their Fort Worth ISD peers.

Academic Success in Reading • CFSRP children in first grade will perform better on standardized reading assessments compared to their Fort Worth ISD peers.

• CFSRP children in second grade will perform better on standardized reading assessments compared to their Fort Worth ISD peers.

• CFSRP children in third grade will perform better on standardized reading assessments compared to their Fort Worth ISD peers.

Academic Success in Math • CFSRP children in first grade will perform better on standardized math assessments compared to their Fort Worth ISD peers.

• CFSRP children in second grade will perform better on standardized math assessments compared to their Fort Worth ISD peers.

• CFSRP children in third grade will perform better on standardized math assessments compared to their Fort Worth ISD peers.

Camp Fire School Readiness Program | 2018-2019 Program Evaluation Report

49

SCHOOL Readiness CAMP FIRE FIRST TEXAS

2018-2019

Participant Demographics

1%

Asian

14%

Hispanic

Caucasian

21%

1,177 Children 74% were retained for the full program year

8% Multiracial

African American

56%

24 %

36 %

40 %

Infants

Toddlers

Preschool & Pre-K

49 % Male

201

51 %

Directors & Teachers

25

Level 2 Basic

Female

60% Have only a high school diploma or GED

15% Have less than

25%

1 year experience

Have 11 or more years experience

26%

45

Some College

Level 3 Excelling

Experience in the field

12%

17

79%

Program teacher retention rate

Have an Associate’s, Bachelor’s or Master’s degree

36%

Have 7-10 years experience

12%

Level 4 Sustaining

12%

Have 1-3 years experience

Have 4-6 years experience

87

Classrooms

have 80% of3+ teachers years in the CFSRP

* 2% Not Reported

Camp Fire School Readiness Program | 2018-2019 Program Evaluation Report

50