CONCEPTUAL ANALYSIS AND EVALUATION FRAMEWORK
179
Analyzing the interaction between contextual factors and I-SEA in explaining outcomes: A critical success factor for SEA effectiveness is the ability to adjust the scope and methodology of an SEA to contextual factors (e.g., Hilding-Rydevik and Bjarnad贸ttir 2007). The interaction between the pilot and its context therefore merits attention in evaluating the contribution of an SEA pilot to observable outcomes. The evaluator should distinguish between factors under control of the SEA team and external factors. Formal as well as informal institutions in the country,28 windows of opportunity for policy reform, and political economy conditions affecting the implementation viability of reforms are examples of external factors that could define (favorably or unfavorably) I-SEA outcomes. It is difficult to identify ex ante which contextual factors are most important in explaining I-SEA outcomes. As a rule of thumb, the evaluator should intend, early in the evaluation process, to get a broad overview of the historical, political, economic, social, cultural, and institutional factors that may be crucial to the policy intervention at hand. The evaluator should then try to narrow the focus to those contextual factors that seem to be most important in explaining the influence or lack of influence of the SEA pilot. In evaluating the interaction between the pilot and its context, the evaluator should also analyze the role of the factors potentially under control of the SEA team. Among those that merit consideration are the ability to access and involve key stakeholders and decision makers in the I-SEA process, communication of I-SEA findings and results, and the ability to take advantage of windows of opportunity for influencing decision making and effecting institutional change. 9.3 Evaluation Process Evaluation team: The evaluation of pilots will be carried out by specialists independent of the World Bank.29 Evaluators are encouraged to team up with local expertise or seek assistance from local specialists to undertake the evaluation of the SEA pilots. Evaluation steps: The evaluation of each pilot will involve the following steps: Preparatory work: Thorough preparations will be key for successful fieldwork. Preparatory activities suggested include (i) document review, (ii) development of a plan for the fieldwork including an interview guide, and (iii) draft context analysis. Fieldwork: Each pilot evaluation will include at least one trip for carrying out fieldwork activities. Report writing: A draft evaluation report may be written during the field trip. This may allow for a validation of some of the findings during the field trip. The final report should incorporate comments received on the draft report. Documentation of findings: Each evaluation team should establish an electronic database including documents, interview protocols, and other sources of information