Another relevant limitation of randomized experiments concerns the di¢ culty in generalising the …ndings to other settings, that are di¤erent from the original one. In short, they can have a high internal level of validity, but not an external one. Hence, analysts that use this approach have to replicate a speci…c experiment before reaching general conclusions. Non-randomized studies usually use data coming from large geographical areas, with diversi…ed populations and their …ndings are applicable to a wider range of di¤erent contexts. But often these methods are less e¢ cient in terms of internal validity. Randomized experiments are designed carefully, and their implementation is planned closely, but extension to a large scale can yield di¤erent results. Pilot studies46 can be used to evaluate if a policy produces relevant impacts on a small scale, and then eventually to procede with the implementation on a wide scale. When a randomized experiment is implemented, the initial random assignment of the treatment must be maintained during the entire period of analysis. The limits here are due to attrition and contamination. The …rst problem (attrition bias) is not predictable and can overestimate or underestimate the impact of intervention. Contamination may happen if, for example, a new micro…nance institution starts working with a comparison group, giving …nancial services to the people that do not receive any kind of treatment. Finally, randomized approaches impose their logic on the selection of program participants. This implies that not the entire population will receive the treatment, and the choice of who obtains the loan cannot be made based on fairness assumptions (such as "those who need it most"). Several nongovernment organizations may be unwilling to accept this statistical method. (Armendàriz, Morduch, 2010). 6. Experimental versus Non-experimental: a comparison Recently, a growing amount of literature has tried to use randomized experiments to validate non-experimental methods. Lalonde (1986) suggests that many statistical procedures used in impact evaluations did not lead to precise estimates and often these di¤er substantially from experimental results. Glazerman, Levy, 46
Pilot study represents the phase before the program is scaled up. This is an occasion for researchers to assess the e¤ectiveness of the program and a chance to improve the experimental design. (Du‡o et al., 2007)
34