DRAFT KENYA PRIMR 2014 ENDLINE READING FLUENCY PROGRESS BRIEF – ENGLISH INSTRUCTION This brief summarizes change in oral reading fluency (ORF) associated with Kenya PRIMR after two years of implementation.The findings presented are representative of Grade 2 students attending schools that benefited from full treatment under the program across the implementation cohorts.The program’s data collection structure did not assess students at the same point during the school year, as is recommended by USAID Education Strategy methodology. As such, change in reading outcomes is not measured based on a comparison of cohorts but rather by contrasting the full treatment group with a control group composed of students that did not receive any intervention.The data used for this brief are from the endline reading assessment in English.
Program Summary Implemented by RTI International, the Primary Math and Reading (PRIMR) program aimed at improving learning outcomes in reading and mathematics in 547 schools in five counties, as well as low cost private schools in Nairobi. It strengthened the Ministry of Education’s delivery systems to enhance learning outcomes in Kiswahili, English and mathematics at the national and county level, using government personnel and systems. While a three-year program, PRIMR’s classroom implementation occurred primarily over two years (2012 and 2013). Total Estimated Cost
Geographic Coverage
$8.1 Million
Nairobi, Kiambu, Nakuru, Murang’a and Kisumu counties
Duration
Three years August 2011 to August 2014
Program Highlights • 82,800 student beneficiaries reached in School Years 2012 through 2014 • 16,400 more students reading at least 45 correct words per minute • 5,100 more students able to read at least one word Intervention Grades
Intervention Languages
1 and 2
English and Kiswahili
Evaluation Design RTI conducted an impact evaluation using the early grade reading assessment (EGRA). Throughout the evaluation, over 25,000 students from about 300 schools were tested. The main evaluation indicator was oral reading fluency (ORF) in correct words per minute (CWPM). Assessment Groups
Assessment Phases
Assessment Grades
Assessment Languages
Full treatment (cohorts 1 and 2) and control
Baseline (January 2012), Midline (October 2012), Endline (October 2013)
1 and 2
English and Kiswahili
This Reading Fluency Progress Brief is made possible by the support of the American People through the United States Agency for International Development (USAID). It was produced for review by USAID and was prepared by Management Systems International, a Tetra Tech company, for the E3 Analytics and Evaluation program.
DRAFT
Reading Ability Levels
In order to maintain consistency across program briefs, USAID’s Education Office in the Bureau for Economic Growth, Education and Environment’s (E3/ED) set four generic performance categories based on ORF: 0 CWPM (non-readers), 1 to 10 CWPM, 11 to 44 CWPM, and 45 or more CWPM. These generic categories should not be used as a replacement for country-specific benchmarks. E3/ED encourages programs to conduct performance standard setting (or benchmarking) workshops with the host government to determine the country-appropriate performance categories.
ORAL READING FLUENCY RESULTS BY PERFORMANCE CATEGORY
43%
44%
44% 59%
63%
67%
45+ 11-44
35%
39%
1-10
43%
Non-Reader
27%
25%
23%
7%
4% 5% 7% Treatment
13% Control
15% Control
Overall
5%
2%
9%
11%
Treatment
Control
Male
5% 5% Treatment Female
Out of 82,800 students reached (41,400 male, 41,400 female), at the endline, there were:
5,100
(-2,600 male, -2,500 female)
300
11,600
16,400
(-800 male, 1,100 female)
(-3,200 male, -8,400 female)
(6,600 male, 9,800 female)
fewer
more
fewer
more
0 CWPM readers
1-10 CWPM readers
11-44 CWPM readers
45+ CWPM readers
KENYA PRIMR 2014 ENDLINE READING FLUENCY PROGRESS BRIEF – ENGLISH INSTRUCTION
2
DRAFT
Reading Improvements
ORF score percentiles can be used to understand how ORF changed for students of different ability levels. For example, as shown in the table below, 75 percent of males in the control group scored below 62 CWPM. In the treatment group, this same percentage scored below 83 CWPM. Therefore, the change for males at the 75th percentile was 21 CWPM.
ORAL READING FLUENCY GAIN SCORES
Lower performers
Higher performers
Percentile Rank 95 90 85 80 75 70 65 60 55 50 45 40 35 30 25 20 15 10 5
Male Oral Reading Fluency Control Treatment Change
Female Oral Reading Fluency Control Treatment Change
105
121
+16
87
117
+30
92
104
+12
76
105
+29
84
95
+11
70
95
+25
70
88
+18
66
90
+24
62
83
+21
62
85
+23
57
76
+19
57
81
+24
51
70
+19
51
76
+25
48
64
+16
47
72
+25
42
60
+18
44
67
+23
39
55
+16
41
64
+23
34
49
+15
38
58
+20
33
43
+10
36
53
+17
28
40
+12
31
47
+16
23
34
+11
26
42
+16
18
29
+11
21
37
+16
7
23
+16
17
31
+14
1
12
+11
13
24
+11
0
5
+5
0
10
+10
0
0
0
0
0
0
Out of 82,800 students reached (41,400 male, 41,400 female), at the endline:
77,000
75,600
71,300
(34,500 male, 36,800 female)
26,800
(37,900 male, 39,100 female)
(37,300 male, 38,300 female)
had gains of at least
had gains of at least
had gains of at least
had gains of at least
1 CWPM
5 CWPM
10 CWPM
20 CWPM
KENYA PRIMR 2014 ENDLINE READING FLUENCY PROGRESS BRIEF – ENGLISH INSTRUCTION
(4,200 male, 22,600 female)
3
DRAFT
Reading Statistics
The table below indicates that the percentage of non-readers between control and treatment groups in Grade 2 decreased overall, for males and females by 6, 7 and 7 percentage points, respectively. The average ORF increased overall, for males and females by 16, 13 and 18 CWPM, respectively.
ORAL READING FLUENCY STATISTICS Control
Treatment
13%
15%
12%
% Non-readers
7%
8%
5%
43
43
43
Average ORF (CWPM)
59
56
61
520
263
257
Sample Size
1,592
795
797
Overall
Male
Female
Overall
Male
Female
Assessments and Sampling The EGRA tool included ORF and reading comprehension components. Because PRIMR began before the USAID Education Strategy was finalized, the data collection structure differed from the final guidelines that data be collected at comparable time points at baseline, midline and endline. For the baseline, the EGRA was conducted near the beginning of the school year. The midline and endline assessments were conducted near the end of the school year. Cohorts 1 and 2 began receiving the non-ICT (Information and Communications Technologies) aspects of PRIMR in 2012 and 2013, respectively. A randomized controlled trial of the impact of three ICT interventions was implemented in Kisumu county only, and its results are not part of this brief. Baseline and endline versions were statistically equated. Sampling weights were used to produce estimates for the population of beneficiaries.
ASSESSMENT AND SAMPLING CHARACTERISTICS Critical Characteristics
Yes
No
Were the assessments conducted at comparable time points? Were the assessment instruments (tests) equated? Were schools randomly selected for assessment? Were sampling weights available for use in the analysis? Was inter-rater reliability monitored during the data collection? This is one in a series of standard-format progress briefs exploring changes in early grade oral reading fluency associated with USAID-supported reading programs. Briefs focus on oral reading fluency because it is a powerful predictor of student reading comprehension. Briefs prioritize early grades because early grade reading is key to further academic success. KENYA PRIMR 2014 ENDLINE READING FLUENCY PROGRESS BRIEF – ENGLISH INSTRUCTION
4