7.5 Performance Triangle for Two Programs

Measuring the Performance of Targeting Methods | 491

Figure 7.5 Performance Triangle for Two Programs

Coverage

Generosity

Program A

Source: Original compilation for this publication.

Program B

Benefit incidence

the impacts on welfare caused by the higher generosity and much more progressive incidence in program B.

A closer look at the redistributive impacts through the DCI confirms that despite being small, program B has more progressive incidence and higher generosity than program A, and program B seems to have a higher transfer for those in the bottom tail of the welfare distribution compared with those in the upper tail. The fact that program A’ DCI is less than 1 implies that the change in social welfare (marginal benefit) achieved by transferring a standardized budget (say, $1) through the program is not efficient for program A, as less than $1 would go to changing the welfare of the poor.

Despite the challenges of having a proper counterfactual in the absence of the programs, table 7.16 assumes that all transfers were consumed by the household and each counterfactual is estimated independently of the other program. Program B’s higher generosity among the poor leads to a higher impact on FGT(0) combined with the higher program coverage of the poor. However, as shown by the DCI, program B has more redistributive power and more progressivity, which translate into a higher impact on FGT(1). This impact is corroborated by the almost double cost-benefit ratio of program B compared with that of program A.

In conclusion, by looking only at exclusion errors, program A—a universal child allowance—would have better targeting performance than program B—a guaranteed minimum income program. When other indicators are considered, the small, narrowly targeted guaranteed minimum income would show greater redistributive impact and targeting performance than the child allowance and a larger impact on the poverty gap, FGT(1). The smaller impact on the poverty headcount, FGT(0), occurs because the

492 | Revisiting Targeting in Social Assistance

guaranteed minimum income reaches those far from the poverty line. The higher coverage of the child allowance manages to reach those close to the poverty line who easily cross the poverty line with a small transfer. The larger impact of the child allowance on the reduction in the poverty headcount comes at the cost of covering 5.5 times more people and spending twice as much as the guaranteed minimum income program. The findings show the poverty- and distribution-related considerations for a policy maker. However, policy can have other objectives as well. In the country from which this example is drawn, the child allowance also has an objective of increasing the birth rate among women of childbearing age, and the analysis explained here is inadequate to shed light on that objective or the weighting between the poverty and fertility impacts of the programs. Further, the analysis is not conclusive about whether a guaranteed minimum income or a child allowance program is better. That depends on the weights given to different factors, and indeed many countries have both programs as they serve different policy niches.

6. The most common basis for measuring targeting outcomes is household surveys, but they suffer from some limitations.

The explosion of the availability of household survey data in the past 20 or 30 years has been a huge boon to understanding the performance of social programs. In many countries, consumption and expenditure surveys, Living Standards Measurement Study surveys, multiple indicator cluster surveys, Demographic and Health Surveys, or similar have become a part of the statistical infrastructure with some regularity and credibility and are now a part of the expected toolkit of policy analysis. The random sampling frames and representativeness of household surveys are important to be able to make statements about distribution.

However, the design of questionnaires and often inadequate or poorly representative sample sizes or the type or absence of welfare collected by household surveys may limit their ability to cast full light on the performance of targeted social assistance programs,12 due to nonsampling and sampling errors.13 The questionnaires of many such surveys contain questions that are relevant for only a subset of social programs, and the questions are not always adapted to the specificities of different programs. Household surveys can suffer from poor question wording, definitional differences between the nature of the indicator and the way the question is asked, misunderstandings on the parts of both the interviewer and the interviewed, lack of knowledge on the program received (for example, the respondent is not aware of the benefit amount or frequency of a particular program received by another household member), inability of the interviewed to keep up with program name changes,14 and deliberate

Measuring the Performance of Targeting Methods | 493

misreporting of welfare or program participation, which can also be associated with the respondents’ misunderstanding of the objectives of household surveys and the fear that these may be government audits/spot checks of program beneficiaries.15 Moreover, very often, information is not disaggregated by individual programs, the transfer amount is not collected, or public and private transfers are mixed in the same questions or collected for a different assistant unit.16 Survey estimations can also be imprecise due to sampling errors if the population of interest is not adequately represented in the sampling frame, which causes loss in the statistical precision of any indicator derived from the household survey. The problem may be especially marked in countries where social assistance programming has been small or consisted of start/stop programs so that statistical institutes would not have been able or expected to capture social assistance programming in their surveys.17 Moreover, sample sizes or designs may be a problem especially in capturing information about small social protection programs.18 Therefore, it is important to estimate the standard errors19 of each indicator produced when comparing the performance of countries or programs to show differences through calculating confidence intervals, to determine the precision of the indicators and compare different programs. Household surveys may not measure the same concept of welfare as programs use— focusing on income rather than consumption or vice versa or focusing on the same concept but with different degrees of thoroughness on issues such as own-produced food or treatment of owner-occupied housing.20 Household Income and Expenditure Surveys, Living Standards Measurement Study surveys, and the like will have detailed income or consumption modules on which to base distributive analysis, but Demographic and Health Surveys have only limited asset information on which to build an index of wealth.

Finally, household data are available only periodically; thus, they capture the welfare of households at a different time than that when eligibility assessments for any programs they benefit from would have been done. Therefore, data from household surveys may cast some light on some aspects of the overall success of the larger and more stable programs, but they are not the same as tests of actual eligibility decision-making processes. For those, audits, simulations, process evaluations, and/or decision-contemporary special-purpose surveys are needed. For example, suppose there is a direct cash transfer program for those living below the poverty line. The determination of eligibility for the program is done through a means test, and the program rules are clear that the selected beneficiary will receive 36 months of transfers. Suppose that a recipient household uses the transfers to make small investments (for example, in poultry) and after a year such investments start bringing extra income to the household. Two years later, when the household survey data collection