5.8 Illustrative Lessons from the Simulations Literature

318 | Revisiting Targeting in Social Assistance

BOX 5.8

Illustrative Lessons from the Simulations Literature

The comparative simulations literature reinforces the usefulness of country-specific work, showing some of the methodological choices and how the range of results can vary between countries. This box refers to two well-known papers: Brown, Ravallion, and van de Walle (2016) and Acosta, Leite, and Rigolini (2011).

Brown, Ravallion, and van de Walle (2016) focus on nine African countries—looking mostly at “hard case” countries with relatively high poverty, small social protection programs, and low social protection administrative capacity at the time (Burkina Faso, Ethiopia, Ghana, Malawi, Mali, Niger, Nigeria, Tanzania, and Uganda). The paper simulates the performance of various proxy means testing (PMT) methods and compares those scenarios with a basic income scheme or transfers using various demographic criteria. The simulations are budget neutral, with a budget sufficient to eliminate the poverty gap. For each scenario, the budget is equally divided by the total number of individuals who resided in designated eligible households and distributed according to their size. Two relative poverty lines (bottom 20 and bottom 40 percent of the population) are used, but the main results for poverty impact are emphasized for the bottom 20 percent of the population. The following are among the key findings:

• Although the contours of the simulation were set with a budget to close the poverty gap if perfectly targeted, none of the targeting methods comes close to doing so. This emphasizes the difficulty of the “targeting problem.” None of the methods reduces the poverty headcount by more than 5 percentage points (from 20 to 15 percent) nor the poverty gap by half. • On average across all the countries and formulations, PMT methods would have about twice the impact on the headcount as categorical methods. In the best performing versions of each model, from a baseline of 20 percent poverty, transfers targeted with PMT reduce the headcount to 15.5 percent. The best demographic scenarios reduce the headcount to about 17 percent, and a uniform transfer across all households reduces it to about 17 percent as well (Brown, Ravallion, and van de Walle 2016, table 11). • The differences are somewhat more marked when considering the poverty gap, moving the poverty gap from an initial 0.05 to 0.03

continued next page

Choosing among Targeting Methods | 319

BOX 5.8 (continued)

for the better of the PMTs, in contrast to about 0.04 for both the demographically targeted and uniform transfers (table 5.3 in the main text). • As the authors said, the PMT helps filter out the nonpoor and thus makes higher transfers available to the poor who would receive transfers, but at the cost of missing some of the poor. • One set of findings pertains to the details of how the PMT is designed—which variables are included and which methods of inference are used (such as probit or ordinary least squares, quantile regressions or poverty-weighted least squares), a topic taken up in chapter 6. The findings indicate that methods matter, although they do not overcome the essential issue of statistical error in PMTs. • The study is unusual in exploring more sets of demographic criteria than is typically done. Those related to children reduced poverty and the poverty gap somewhat more than those focusing on the elderly, widows, or disabled.

In actual policy, none of the countries covered in the study uses only PMT for targeting their main programs. In none of the countries are the programs fully national, and all of them use geographic targeting to a degree. They all use communities for selection and/or validation, although community-based targeting is all but impossible to model in simulations such as these. Many of the programs use demographic targeting to a degree to ensure that households with children and/or elderly members are prioritized as well. Where PMT is used, it is in combination with all these other methods (for Burkina Faso, Malawi, Niger, and Tanzania, see Beegle, Coudouel, and Monsalve Montiel (2018); for Ethiopia, see World Bank (2015a and 2015b) and Beegle, Coudouel, and Monsalve Montiel (2018); for Ghana, see Agbenyo, Galaa, and Abiiro (2017); for Mali, see Heath, Hidrobo, and Roy (2020) and World Bank (2018a); for Nigeria, see NASSCO (2019); and for Uganda, see Hickey and Bukenya (2021)).

Acosta, Leite, and Rigolini (2011) compare simulations of means testing and demographic targeting for 13 Latin American countries across a large range of incomes, generally with high inequality, substantial social protection programs, and administrative capacity ranging from medium to high. The simulations compare social assistance programs that are fully categorically targeted (benefits given to all children or

continued next page

320 | Revisiting Targeting in Social Assistance

BOX 5.8 (continued)

elderly individuals) and programs of equal budget that are confined to the poor within those demographic groups, using means testing as the method to identify the poor. The scenarios are set to distribute 0.5 percent of gross domestic product and consider an absolute poverty line of $2.50/per person/day, a threshold that counted about 15 percent of the Latin American and Caribbean population as poor in 2010. The authors add to the simulation potential errors in the means testing (an exclusion error of 30 percent of the poor), since no program is perfectly targeted, and some extra administrative cost that would reduce the budget allocated for transfers in poverty targeted programs.

• Categorical targeting can produce strikingly different results depending on the age group selected and the patterns of poverty.

On average, categorical transfers to children are 1.6 times more effective in reducing poverty than categorical transfers of equal budget to the elderly, and poverty-targeted transfers are twice as effective. The reasons are straightforward: poverty rates among the elderly are, on average, lower than for children, and poorer families have more children but not more elderly people. The simulations also suggest that the common belief that cash transfers to the elderly can substantially reduce poverty by trickling down to all family members has limited validity: with fewer elderly than children living in poor households, for the trickle-down effect to be effective, money should be transferred across family members living in different households, which is a much less likely event. • There is also considerable variation across countries in how the methods compare. In Nicaragua, a perfectly targeted program would reduce poverty rates twice as much a categorical one, while in Colombia (the other extreme) this ratio jumps to 7.1. These differences are not explained by income levels alone: effectiveness in

Nicaragua and Argentina, two countries with very different income levels, is very similar. Rather, differences in impact depend on a more complex combination of factors, such as how widespread are pockets of poverty with people far off the poverty line.

Both studies reflect on the extent to which the gains from householdspecific targeting are substantial enough to overcome some of the challenges. Brown, Ravallion, and van de Walle (2016) show that even within the set of PMT or demographic methods, there are significant differences in the results that can be expected depending on how the criteria are set. Acosta, Leite, and Rigolini (2011) show how different the results from the same methods can be across countries. Both of these findings imply the importance of doing country-specific analysis.

Choosing among Targeting Methods | 321

experience in programs selected to be similar along one or more dimensions—design of the program; country context in terms of poverty, inequality, and maybe demographics; and/or implementation capacity and approach—can be helpful.

A small vein of experimental research seeks to compare householdspecific methods using comparative treatment arm designs with PMT, HEA, and/or CBT and self-targeting, which is particularly valuable since CBT and self-targeting are the methods that are the most difficult to simulate. Work by Alatas et al. (2012, 2016) in Indonesia; Premand and Schnizter (2018) and Schnitzer (2019) in Niger; Pop in Ghana (2015); Stoeffler, Mills, and del Ninno (2016) for Cameroon; Sabates-Wheeler, Hurrell, and Devereux (2014) for Kenya; and Escot (2018) and Kameli et al. (2018) for Mali are the classics. Dervisevic et al.’s (2020) study of the Lao People’s Democratic Republic is not exactly a comparative treatment arm design, but it sheds similar light on the issue. Some of the findings are as follows: • In Indonesia, PMT is found to be slightly more accurate than CBT as defined against poverty ($2/day). Self-targeting in the application process led to richer households selecting out while those poorer households who also selected out were more likely to be excluded by PMT (meaning self-targeting did not increase exclusion errors beyond those already driven by PMT). • In Northern Cameroon, PMT is found to be more accurate than CBT as defined against per capita consumption and alternative measures (household food insecurity, multidimensional poverty, and community perceptions) and thresholds. • In Niger, PMT can more effectively identify households suffering from persistent poverty, but HEA is better for identifying those suffering transient food insecurity. PMT and CBT performed similarly on food security, asset ownership, income per capita, and malnutrition. • In Ghana, in a very small study, PMT was slightly more accurate than

CBT. • In Lao PDR, village heads performed about the same as a PMT in selecting poor women to participate in a public works program. • In Kenya’s Hunger Safety Net Program23 phase 1 assessment, PMT was predicted to perform better than CBT at identifying the poorest and food insecure households. CBT performed better than indicator targeting for a social pension (individuals ages 55 years and older) and a dependency ratio (households with a dependency ratio above a certain threshold). • In the region of Gao in Mali, HEA24 did not seem to distinguish the type of vulnerability,25 although it was meant to select only food insecure households. The authors also found that the results improved when