3 minute read
3.3.2 Student Characteristics
6.2.5 Summary of Pooled DiD Results
Early Grade Reading
Table 15 and Table 16 below display a summary of raw and standardized EGRA results from our three specifications.
Table 15. Summary of EGRA Results - Raw Scores
Subtask Applicable Grades
Letter Sounds Onset Sounds
KG-G2 KG-G2 Non-Word Reading KG-G3 Familiar Word Reading KG-G3 Passage Fluency G1-G3 Reading Comprehension G1-G3
Simple DiD [1]
20.2 5% 1.7 9.1 13.7 11%
Regression w/ Baseline Scores Only [2]
20.3 6% 1.6 8.6 12.1 10%
Regression w/ Baseline Scores & Characteristics [3] 21.5 7% 1.5 9.1 12.6 10%
Subtask Applicable Grades Simple DiD [1] Regression w/ Baseline Scores Only [2]
Regression w/ Baseline Scores & Characteristics [3]
Statistical Significance of [3]
Letter Sounds KG-G2 3.13 3.14
Onset Sounds KG-G2 0.23 0.25
Non-Word Reading
KG-G3 1.42 1.32 Familiar Word Reading KG-G3 1.34 1.26 Passage Fluency G1-G3 1.58 1.40 Reading Comprehension G1-G3 1.24 1.16
Average Effect Size 1.49 1.42
3.34 0.29 1.34 1.34 1.46 1.19
1.49
** * ** ** ** **
1.49
Bridge PSL Public school students achieved large differential gains in all six EGRA subtasks, representing a large amount of learning. The results are stable across all three specifications. Given specification 3 is most comprehensive in its inclusion of student baseline characteristics and likely provides the most precise estimate of the Bridge PSL effect, we highlight its results in more depth here.
On average, students at Bridge PSL public schools pronounced an additional 21.5 letter sounds, read an additional 9.1 familiar words per minute, and read an additional 12.6 story words per minute beyond their peers at traditional public schools, when controlling for baseline ability and other student characteristics. They also learned to answer 10% more reading comprehension questions correctly.
42 The statistical significance column shows two asterisks (**) for 99% confidence level and one asterisk (*) for 95% confidence level. The average effect size under this column is designed to count insignificant coefficients as 0, although this adjustment is not needed here because all coefficients are significant. Among EGRA subtasks, effect sizes were very large: 1.49 SDs on average.43 While this is driven in part by the large number of zero scores on baseline assessments, 44 it still represents a substantial movement of students from having zero measurable literacy skills to progressing steadily on the path towards literacy.
Student-level characteristics do not contribute directly to endline EGRA assessment scores. The inclusion of baseline test scores likely absorbs much of the information these additional observables provide. However, we do see that female students learned less on Non-word Reading holding all else constant. Additional years of early childhood education are associated with less progress on the Letter Sound subtask. Older students in the study had lower growth on Passage Fluency and Reading Comprehension.
Early Grade Math
Table 17 and Table 18 below display a summary of raw and standardized EGMA results from our three specifications.
Table 17. Summary of EGMA Results - Raw Scores
Subtask
Applicable Grades
One to One Correspondence KG Number Identification KG-G1 Quantity Discrimination KG-G2 Addition 1 KG-G3
Addition 2 G2-G3
Subtraction 1 Subtraction 2 Word Problems G1-G3 G2-G3 G1-G3
Simple DiD [1]
13.5 4.7 10% 3.6 10% 1.6 7% 4%
Regression w/ Baseline Scores Only [2]
4.5 5.0 11% 3.4 9% 1.6 7% 5%
Regression w/ Baseline Scores & Characteristics [3] 9.4 4.5 11% 3.7 8% 2.1 7% 4%
43 An alternative to clustering at the school-grade level is to collapse data at the school level. The resulting summary effect size for EGRA would be 0.96 standard deviations considering only coefficients significant at the 95% confidence level or better, or 1.28 standard deviations considering only coefficients significant at the 90% confidence level or better. See Appendix A7.3 Standardized Difference-in-Differences, Data Collapsed at School Level, Table 43 for these regression results. 44 A large number of zero scores narrows the distribution of baseline scores, lowering the standard deviation. Standardized scores (using z-score methodology) are calculated as (Individual Score - Average Score)/Standard Deviation. Consequently, a small standard deviation results in large standardized scores, and hence a larger effect size.
