Beyond Beneficiaries: The Use of Information Systems for Cost-Effective Evaluation
Suzanne Duryea Research Department Inter-American Development Bank December 10, 2003
Information Systems â€˘ Information systems such as SISBEN, CAS, SIPO were developed to better target beneficiaries (predominantly the poor and at-risk populations). â€˘ A quantitative score is used as a proxy for poverty and/or the lack of income generating resources within the family. Families scoring below a threshold are eligible to participate in certain programs.
Information Systems for Social Programs â€˘ Can facilitate certain types of evaluations â€˘ With some complementary additions: the databases can form a stronger foundation for evaluation
What do we mean by “impact evaluation?” • “What would have happened to the beneficiaries in the absence of the program?” • “What was the effect of the program?”
Some options 1) Randomized Designs • Examples: a) Geographic regions are randomly selected for later phase-in of a program (Oportunidades, PRAF) b) Individuals are randomly selected to form a control group (such as school vouchers Colombia) • Randomized evaluations are viewed by the experts as the best option because they solve many of the problems encountered in evaluation. (Duflo and Kremer, 2003) • However, randomization can be politically difficult and is not always possible or appropriate.
Some options 2) Ex-Post Evaluation â€˘ Aim is to compare outcomes of beneficiaries and nonbeneficiaries who were similar BEFORE program participation â€˘ Only observe individuals at one point in time -- after enough time has passed for program to have an effect. â€˘ Critical assumption: After controlling for observed characteristics, beneficiaries and non-beneficiaries do not differ in unobserved characteristics
Some options 3) Before and After Comparisons -- “Double Difference” • Monitor the beneficiaries and non-beneficiaries over the length of time necessary for the program to have effects • I.e., Take baselines for both groups then compare the change in the beneficiaries’ behavior to the change in the non-beneficiaries’ behavior. All time invariant unobservable differences are removed. • Critical assumption: No time varying unobserved differences between beneficiaries and non-beneficiaries.
Creation of Counterfactual Group • The most critical part of evaluation design: who can be used as a natural comparison • Information Systems Can Plan an Important Role • There are families who will not quality for programs because they are slightly above the eligible score (threshold). These families look very similar to families with similar scores who have qualified. Discontinuity Design: (Campbell, 1969, Buddlemeyer and Skoufias 2003) • Assign program and then compare outcomes across the families who are close to the threshold.
Important Caveats • This is a local average treatment effect: not necessarily comparing the effect on the poorest families • If there are heterogeneous effects at different levels of scores this is only measuring the effect at the cutoff score • Threshold must exist: enforcement of rules necessary.
Example of an ex-post evaluation: Super茅monos S. Duryea and A. Morrison (2003) Relied heavily on information systems from Instituto Mixto de Ayuda Social (IMAS), Costa Rica: Sistema de Informaci贸n sobre la Poblaci贸n Objetivo (SIPO) Sistema de Atenci贸n a Beneficiarios (SAB)
Superémonos • Food coupon (U.S. $30 per month during school) • Targets poor households with school age kids (ages 6 to 18) at risk for poor school attainment using SIPO (proxy means test) • Conditional transfer: families agree that all children will regularly attend school
Sistema de Información sobre la Población Objetivo (SIPO) • Targeting mechanism • SIPO score depends on – – – – –
Occupation of household head Material used in house construction Household income Education of household head Net household wealth
• Over 250,000 households
SIPO and SAB • Very efficient • We provided IMAS with a list of characteristics (requested 3 regions, ages 10-16, started program in 2001) and they provided a cross-referenced list of beneficiaries within minutes • Were in the field with our survey within two months • Very fast in contrast to adding questions to a national household survey
First however: we were missing a critical component for evaluation â€˘ No information was available in SAB regarding potential counterfactual group (those just above the threshhold) We formed a counterfactual group by conducting surveys in the same neighborhoods and getting families with similar probabilities of participating in Superemonos (propensity matching approach) Cost of data collection for evaluation of SuperĂŠmonos 1788 households under $30,000 Mexico Progresa 24,407 households $450,000 Argentina Trabajar 2,800 households $350,000 Source: Blomquist 2003
Summary Information on Survey Samples
number of total observations average age of child percentage female percentage in San Jose percentage in Alajuela percentage in Cartago percentage of mothers with incomplete primary education percentage of households lacking working electricity
Not from list
746 12.90 50.27 60.19 6.03 33.78
1032 12.95 48.55 56.88 4.94 38.18
Survey design and implementation • • • •
Tailor-made survey Pilot tested Power tested Sample size: 746 beneficiaries of Superémonos 1,042 non-beneficiaries
• Data collected in 3 urban centers: San Jose, Alajuela and Cartago • Collected information on labor force participation, school attendance, school performance, and a host of household characteristics • Approx. cost: $15 per survey (no addresses)
Duryea Morrison Strategy: Ex-Post Evaluation Information System Used to Identify Beneficiaries Only We found that beneficiaries ages 12-15 were 5 percentage points more likely to attend school than non-beneficiaries. In this program we have no reason to think there are serious selection problems. Academic record not considered for eligibility. Ex-post methodology may be appropriate.
Alternative Strategy: What we might have done Double Difference: Information Systems Used to Identify Treatment and Control Groups • Create comparison group from those who fall just above the threshold on SIPO for Superemonos participation • How: Match 3 non-beneficiaries to each beneficiary (based on SIPO score within same geographic unit) • Follow beneficiaries and non-beneficiaries over time (double difference estimation)
Hypothetical Example: Effect of Conditional Transfer Program on Hours of School Activities Double Difference Measures Positive Effect of Program Ex-Post Measure Misses Positive Effect of Program Hours Spent on School Activities
Double Difference = C - B = 27 - 18 = 9 amount of change by beneficiaries is larger
A = ExPost Difference
20 15 10
Beneficiaries 5 0
Months of Execution of Project
Policy Conclusions and Recommendations 1. Information systems are critical, both for targeting as well as for evaluation. 2. Information systems can facilitate cost-effective ex-post evaluations. 3. Possible to modify information systems to provide more rigorous evaluations at reasonably low cost. Monitoring non-beneficiaries in addition to beneficiaries is the most important modification. Part of evaluation strategy for Chile Solidario. 4. Encourage stricter enforcement of the implementation of scoring thresholds.