POLICY FORMULATION Supplementary reference material Eula Marie Mangaoang
19 June 2014
CONTENTS Introduction ---------------------------------------------------------------------------------------------------- 2 Understanding the nature of public policies --------------------------------------------------------------- 3 What is policy formulation? --------------------------------------------------------------------------------- 3 Key actors in policy formulation ---------------------------------------------------------------------------- 4 Towards an evidence-based approach to policymaking ------------------------------------------------- 5 Steps in Policy Formulation --------------------------------------------------------------------------------- 7 Setting criteria for assessing policy options ------------------------------------------------------------ 7 Identifying alternative policies --------------------------------------------------------------------------- 9 Assessing alternative policies ----------------------------------------------------------------------------- 9 Choosing course of action -------------------------------------------------------------------------------- 10 Results-based monitoring -------------------------------------------------------------------------------- 10 Evaluating after implementation ------------------------------------------------------------------------ 13 References ----------------------------------------------------------------------------------------------------- 15
Page | 1
POLICY FORMULATION Introduction “A problem well put is half solved.� –John Dewey Traditional discourse on public policy points to policy formulation as merely coming up with the best solution to the problem (Koliba, Meek, & Zia, 2011). Alternatively, we can view policy formulation as a framework for policy implementation and evaluation, which therefore makes it an integral part of the public policy process. The concept of policy formulation suggests different perspectives. Some view policy formulation as finding the optimal solution to a particular problem, while others see it as a political activity (Hayes, 2014). That being said, in the process of policy formulation, political support is important, but so is decision making founded on evidence. More importantly, there should be an active link between evidence, data, policy and the leadership to come up with a results-driven policy. This module on policy formulation aims to provide the training participants insights on the nature of public policies, and from there come up with an evidence-based approach to policy formulation. At the end of the module, participants should be able to develop the following policy skills: 1) 2) 3) 4) 5)
Understand the nature of public policies Define policy formulation Identify key actors in policy formulation Understand the importance of evidence-based policy formulation Formulate policies based on the following steps: a. Set criteria for assessing policy options b. Identify alternative policies c. Assess alternative policies d. Choose course of action e. Monitor implementation and outcomes f. Evaluate after implementation
The above-mentioned policy formulation and analysis steps are slightly modified from those outlined in handout used in the Workshop on Policy Formulation and Analysis for Staff of Regional Civil Society Organizations (PIANGO/ADB RETA 6319, 2007).
Page | 2
Understanding the nature of public policies Policymakers struggle to come up with policies that effectively address societal needs and deliver public goods and services. The process involved is often discontinuous, with periods of marginal adjustment in-between. Oftentimes, they are elaborations of policies that have already been established (Kirlin, 1984). Milakovich & Gordon (2008) described public policy processes as lacking centralized direction, with focus on the interaction of governments and different interest groups. They are loosely coordinated, and are therefore highly competitive, fragmented and specialized, and largely incremental. Ultimately, they are subject to many different political and economic forces. Meanwhile, public problems exist in a way that (PIANGO/ADB RETA 6319, 2007): 1) 2) 3) 4) 5) 6) 7)
They are often ill-defined They have political as well as purely technical aspects They often lack a good cause-effect knowledge base They may be solved but in so doing new problems emerge Their solution often involves trade-offs between cost and effectiveness It may be hard to measure adequacy of results It may be hard to measure fairness of results
Choices and decisions are built upon understandings, which are in turn prognoses—expectations of what is currently happening and what will happen after the implementation of a particular policy (Kirlin, 1984). It is therefore important that you are able understand the content, nature and the forms of interaction involved in the public policy process. More importantly, you need to understand the policy issue from which the process arises. Basically, policy issues involve the following components (NCPAG-CPED, 2011): 1. A social problem, which, according to Nakamura & Smallwood (1980), can either be a crisis situation, a concern detrimental to the interest of specific groups, or general public concerns and issues; 2. Players—individuals, groups, organizations—affected by the problem and will try to correct the problem (e.g. policymakers and non-government individuals/groups); and 3. A public policy to try to correct the problem.
What is policy formulation? Policy formulation comes in many definitions, often either in terms of technical or political aspects. Hayes (2014) states an idea of policy formulation that will prove useful as we move on with this module. He defines policy formulation as: ―…the development of effective and acceptable courses of action for addressing what has been placed on the policy agenda‖ (http://profwork.org/pp/formulate/define.html) We will try to dissect this definition through the key words, ―effective‖ and ―acceptable‖. Page | 3
What constitutes effective policy formulation? In analyzing policies, goals are taken as the givens, and policymakers must find ways by which they are best achieved. You as one of the policy actors must ensure that the policies are valid, efficient and implementable across times, places and given circumstances. In doing so, you will be coming up with not only one, but many different policy alternatives, and identify which of them best solves the problem. An effective formulation requires good analytical skills. Meanwhile, policy formulation is made acceptable when it is authorized by the decision makers, usually through majority-building in a bargaining process. If the policy is likely to be disapproved by the Congress or the Senate, to suggest it may be impractical. Based on these two aspects, we can view policy formulation as having two phases: 1) Analytical phase – in this phase, policymakers must develop policy options that are conceived and clearly articulated based on sound analysis. Policies that are unworkable or ineffective in practice are likely to be rejected. 2) Political phase – in this phase, policymakers decide on the best policy option, which should be legitimated through a political process, e.g. legislation or regulation.
Key actors in policy formulation Societal problems are often interrelated with each other; consequently, they tend to affect not just one, but many different sectors. For example, in a recent effort of scientists to put climate change into the spotlight, national governments, there emerged a growing awareness among governments and interest groups that climate change affects them in many ways. Some of them merged around the problem, and vowed to bring climate change to the public agenda. Koliba, Meek & Zia (2011) asserted that literature on governance saw the formation of iron triangles, issue networks and even intergovernmental networks in the course of policy formulation. Key players can be, on one hand, the policymakers in the executive, legislative or judicial branches of the government. On the other hand, there are non-government organizations and special interest groups that try to influence public policies (Nakamura & Smallwood, 1984; NCPAG-CPED, 2011). They may form ―governance networks‖ which, in the stage of policy formulation, are mobilized to define the problem, look at alternatives, and/or plan for policy implementation (Koliba, Meek, & Zia, 2011). This mechanism works to the extent that policy planners and technical analysts are keen and informed enough to provide sound analysis, and that policymakers are reasonable and responsive to the needs of the stakeholders. The role of the policy analyst and the decision maker should complement each other. If there is something wrong with the policy, the analysis by policy planners might have been faulty, or the policymakers may have either exercised bad judgment, chosen the less effective policy options, incorrectly defined the problem, or ―played politics‖ with public policy (Hayes, 2014). Page | 4
Towards an evidence-based approach to policymaking ―The good news is that evidence can matter. The bad news is that it often does not.” (Sutcliffe & Court, 2005) In contrast to traditional scientific research, which addresses broad, theoretical and complex questions, analyzing policy options is said to be more practical, flexible and situational. It is thus considered to be more of a craft or art than science. That being said, policy formulation and analysis should be as rigorous and as evidence-based as possible (PIANGO/ADB RETA 6319, 2007). The emergence of the evidence-based policy (EBP) approach in the UK in late 1990s led to an increasing demand for developing countries to design policies that are founded on a combination of ―experience, judgment and expertise with the best available external evidence from systematic research‖ (Sutcliffe & Court, 2005). The approach is basically a shift from the traditional opinion-based policy formulation, employing rigorous, systematic approaches to better inform policy decisions that will eventually produce better outcomes. Evidence has two main purposes: to help define the problem, as well as identify potential solutions. In some cases, one has to look at evidence outside the target area (e.g. Philippines) where the problem exists (Office of the First Minister and Deputy First Minister, 2006). Unfortunately, this does not seem to be the case in the typical policymaking scenario. Head (2008) pointed out that policy decisions stem from political judgments and debates, leaving out facts that have been obtained from technical analysis. As a result, societal problems in a number of countries have deepened because policymakers have ignored the evidence supporting the existence of these problems (Sutcliffe & Court, 2005). Meanwhile, we should also not forget that policy formulation, and policymaking as a whole, is ―neither objective nor neutral‖ (Sutcliffe & Court, 2005: iv). Policies should be formulated based on three important forms of ―evidence‖ or knowledge (Head, 2008): 1) Political know-how 2) Rigorous scientific and technical analysis; and 3) Practical and professional field experience. At the stage of policy formulation, policymakers should gain a more detailed and comprehensive understanding of a particular problem to be able to make informed choices on which policy should be implemented. This involves identifying the possible outcomes of a policy alternative, and the expected costs and impacts. In using the EBP approach, quantity and credibility of the evidence is important. In using evidence for policy formulation, we should keep in mind the following considerations (Sutcliffe & Court, 2005): 1) Policy should be informed by a wide breadth of evidence, not just hard research. Key issues include the quality, credibility, relevance and the cost of the policy; 2) Evidence is needed, and in different ways, at a number of different points of the policy cycle; and Page | 5
3) Time constraints will affect the mechanisms available to mobilize evidence – urgent issues require different approaches than processes to develop strategic policy directions. To ensure that evidence used for policy formulation is relevant and of good quality, policymakers must consider tapping expert knowledge to help evaluate what evidence is useful. Below is a checklist for assessing evidence (Table 1) (Office of the First Minister and Deputy First Minister, 2006). Table 1. Key principles in assessing evidence.1 1) Does the study address the key policy issues Is it relevant? and questions? 2) Is it appropriate to use evidence collected in a different context? i.e. How far can results of local or national studies inform a regional policy? Is the social, cultural and economic context for an overseas study similar to that in the Philippines? 3) Was the study undertaken recently - have things changed since it was done? (This does not mean that research evidence can be ignored just because it is old - in some policy areas, research can remain relevant for a long time.) 4) Does the study clearly identify implications for policy and/or practice? 1) Are the research methods used appropriate to Is it good-quality? the key questions being asked? 2) Does the study consider the issues from a range of perspectives e.g. involving service users/other stakeholders? 3) Has the study been conducted properly - is there information on how the methods were implemented e.g. response rates for surveys? 4) Does the individual or organization which undertook the study have previous experience of research on the issue and/or the methods used? 5) Has the study been undertaken, commissioned or funded by individuals or organizations with views or vested interests which may favor particular conclusions?
1
Lifted from “Practical Guide to Policymaking in Northern Ireland�, and is slightly modified to fit the local context.
Page | 6
Steps in Policy Formulation2 Here are some start-up questions that will help you in defining the problem and formulating policies: 1) Quickly identify the central decision criterion of the problem (What is the most important factor in selecting a course of action to address the issue?) 2) Identify what types of public sector actions can be taken (Taxing, spending, sanctions, incentives, moral suasion, education?) 3) Avoid the "one best way" approach (Have many tools in the tool box, not just one) 4) Learn how to deal with uncertainty (Admit it, estimate its possible effects) 5) Say it with numbers (Charts, graphs, tables, maps, etc.) 6) Make the analysis simple and transparent (Provide details in a technical appendix) 7) Check and re-check the facts (Use multiple sources of facts, triangulation) 8) Learn to anticipate the objections of opponents (Improves the ultimate product) 9) Give analysis, not decisions (Distinguish between analysis and advocacy) 10) Push the boundaries of the envelope (Expand the problem definition; introduce novel solutions) 11) Policy analysis is never 100% complete, rational, and correct (How much time, money, and personnel is available to do the job?)
Setting criteria for assessing policy options When a set of policy problems are defined, policy options should accomplish a particular goal or set of goals to be able to address the problems. These goals should be translated into concrete, specific objectives, which outline the desired ends, time tables, target groups, and resource limits. Objectives should be SMART: specific, measurable, attainable, realistic and time-bound. American businessman Peter F. Drucker once said, ―If you can’t measure it, you can’t manage it.‖ Criteria, as measurable aspects of policy objectives, are used to compare how close or far different policy options will go to attain a goal and solve the problem. This will guide in the analysis phase of problem formulation, particularly in comparing policy alternatives. Criteria come in many different types; Kraft & Furlong (2004) provide a matrix of criteria useful for evaluating policy options (Table 2). Criteria measurements should be definite, straightforward, impartial and simple to measure. They should also produce uniform results, regardless of who does the measuring or how many times the measurements are done. It is important that before policy options can be developed, the problem must be defined and the criteria for assessing options should also be specified.
2
Taken from “Policy Formulation and Analysis”, Workshop for Staff of Regional Civil Society Organizations, Nadi, Fiji, 18-21 December 2007.
Page | 7
Table 2. Kraft & Furlong’s (2004) selected criteria for evaluating public policy proposals. Criterion Definition Limits to Use Effectiveness Likelihood of achieving policy Estimates involve uncertain goals and objectives or projection of future events. demonstrated achievement of them. Efficiency Achievement of program goals or Measuring all costs and benefits is benefits in relationship to the costs. not always possible. Policy decision Least cost for a given benefit or the making reflects political choices as largest benefit for a given cost. much as efficiency. Equity Fairness or justice in the distribution Difficulty in finding techniques to of the policy’s costs, benefits and measure equity; disagreement over risks across population subgroups. whether equity means a fair process or equal outcomes. Liberty/Freedom Extent to which public policy Assessment of impacts on freedom extends or restricts privacy and are often clouded by ideological individual rights and choices. beliefs about the role of government. Political feasibility
The extent to which elected officials accept and support a policy proposal.
Social acceptability
The extent to which the public will accept and support a policy proposal.
Administrative feasibility
The likelihood that a department or agency can implement the policy well.
Difficult to determine. Depends on perceptions of the issues and changing economic and political conditions. Difficult to determine even when public support can be measured. Depends on saliency of the issues and level of public awareness. Involves projection of available resources and agency behavior that can be difficult to estimate.
Technical feasibility
The availability and reliability of technology needed for policy implementation.
Often difficult to anticipate technological change that would alter feasibility.
Where Most Likely Used Virtually all policy proposals where concern exists over how well government programs work. Regulatory policies, e.g. workplace safety and environmental protection; consideration of market-based approaches. Civil rights, disability rights, tax equity, access to health services and higher education. Proposed national identification cards, restrictions on Internet use, property rights, abortion rights, regulatory actions that constrain choices of corporations and individuals. Any controversial policy, such as gun control or changes in environmental regulations. Any controversial policy, such as crime control or abortion rights.
Expansion of agency duties, use of new policy approaches or new technologies, policies with complicated institutional structures. Science and technology policy, environmental and energy policies, telecommunications, defense policies.
Page | 8
Identifying alternative policies A large number of alternatives can be generated at first, but usually these are broken down into a manageable size, around four to seven. You should treat alternatives as if they are the status quo, and at the same time radically different. More importantly, consider what may be possible under different circumstances. Sources of alternative policies include: 1) 2) 3) 4) 5)
The status quo or no action alternative; Experiences of others with similar problems, from reported research findings, etc.; Re-define the problem from others' points of view, including opponents of any change; Consider the ideal, then apply political, economic, and other constraints; and Quick surveys, literature review, case studies, develop typologies. You may also use analogies, expert opinion, brainstorming, feasible manipulation, modify existing solutions, etc.
Meanwhile, there are a number of pitfalls to avoid in developing policy options: 1) 2) 3) 4) 5) 6) 7)
Too much reliance on past experience Failure to capture ideas and insights (listen, write them down, record them) Too early closure on problem definition Setting a policy preference too soon before all the alternatives are known Criticizing new ideas as they are offered Some alternatives are ruled out too early on Failure to re-consider discarded alternatives if conditions change
Assessing alternative policies In assessing policy options, goals should be taken as givens, and policy actors should find ways by which they are best achieved. Usually, these policy options span across different circumstances; in doing so, different decisions are evaluated before they are reached (Kirlin, 1984). The following analytical tools may be used in this step3: 1) Forecasting – using statistical formulas, theoretical models, extrapolations, etc. 2) Cost-benefit analysis – an attempt is made to value in monetary terms all the factors involved, may they be commercials, social or environmental, regardless of to whom the costs and benefits accrue; evaluated in terms of the public’s willingness to pay, either for them (benefits), or to avoid them (costs);
3
For detailed discussions on each of the analytical tools, please refer to this link: http://piango.org/CapacityBuilding/ADBRETA/PIANGOADBRETA6319_RegCSOWshop_4-PolicyFormandAnalysis.pdf
Page | 9
3) Risk analysis – some decision-makers are less averse, and may want to minimize any possible losses rather than to pursue the (riskier) maximum possible gains; 4) Environmental impact assessment – predicts the likely repercussions of a proposed development upon the social and physical environment.
Choosing course of action The policy analyst is often faced with trying to present multiple policy alternatives which have been assessed in terms of multiple decision criteria. There are various methods which can be used to display this information in a way that facilitates decision making. Methods include paired comparisons, satisficing, grading method, lexicographic ordering, weighted decision criteria, and the Groller scorecard, among others. Among the simplest methods would be the SWOT analysis method and the decision matrix. Advice from technical specialists may be needed in weighing the policy options. For instance, economists can assist with economic analysis by conducting relevant research (Office of the First Minister and Deputy First Minister, 2006).
Results-based monitoring Monitoring is a prerequisite for evaluation. It establishes factual premises about public policies, and determines how power or authority is exercised or controlled in the process of implementation. Basically, it answers the questions, ―What happened? How? Why?‖ (Sapru, 2004). We often associate monitoring with evaluation. While they are interdependent with each other, these two terms are different. Other concepts related to monitoring and evaluation should also be distinguished from one another. Table 3 provides a comparison of different M & E concepts.
Table 3. Comparison of different M & E concepts (Metz, 2005). Criteria Monitoring and Evaluation Concepts, Main Features Level of Policies are set at aggregate, high Programs and projects (P&P) are set aggregation: hierarchical levels (macro, national, at lower hierarchical levels and are Policies vs. sectors) and aim at wide-spread more narrowly defined in terms of programs & (country-wide) impacts. Subject of objectives set, issues and/or target projects policy monitoring are the policy groups addressed and/or geographic objectives, strategies for and means areas covered. Monitoring at P & P of implementation and the policy level is organized by the P & P impacts. A policy monitoring management. Ideally, P & Ps form system is organized at central level, part of a strategy for policy with linkages to but based on implementation, and the results of P program and project monitoring. & P monitoring feed into the policy monitoring system. Page | 10
Criteria Monitoring of impacts vs. implementation
Monitoring vs. evaluation
Monitoring and Evaluation Concepts, Main Features Impact monitoring traces and Implementation monitoring of assesses the outcomes of policies/programs/projects policy/program/project covers financial and other inputs used interventions in regard of overall (accounting/input monitoring), objectives set and possible activities performed (activity unplanned side effects. monitoring), outputs achieved (output monitoring), and compares actual achievements with plan. Monitoring is a continuous Evaluation: Specific reviews of exercise, internally organized by policies/programs/projects; a singular the organization in charge of or sequential exercise, often policy/program/project externally initiated and organized by management, to generate real-time higher level decision makers, data and information on supervising bodies and financing policy/program/project agencies, in order to provide evidence performance regarding on performance (efficiency, implementation and/or impacts as relevance, effectiveness, basis for management control, for sustainability) as basis for decisions being able to trace possible on extension, expansion, diversions between reality and plan modification and/or replication of in time and hence to make interventions. necessary adjustments as early as possible.
In the course of policy implementation, it is important to monitor how policies are going in terms of the financial and other inputs (input monitoring), activities performed (activity monitoring), and outputs attained (output monitoring). Meanwhile, you also need to know whether the policy has achieved its objectives and to what degree have these been achieved (Metz, 2005). A resultsbased monitoring system tracks both implementation and results of the policy, as featured in the diagram on the next page (Figure 1).
Page | 11
Figure 1. Results-based monitoring system featuring two key types of monitoring (World Bank, 2000)
The common practice for policy managers is to first assign tasks and identify activities right away. But doing so does not necessarily mean that policy goals are reached. It is thus important for you to first have a set of policy outcomes, and a plan to ensure that the stakeholders/focal organizations will be able to attain these outcomes. A results plan must precede results-based monitoring. Now that you have a set of outcomes, you have to translate them into activities, in the form of programs and projects. The expected outputs, i.e., products and services provided by a particular activity, should also be laid out. Now, how will you know that these have been effective in reaching the intended outcomes? A performance monitoring system framework will help policymakers monitor the performance of policy implementation in attaining a particular outcome. For each policy outcome, you must be able to come up with: 1) Indicators – variable for measuring change, e.g. poverty rate, level of infant malnutrition, percentage of women who vote in the elections, Gini coefficient Page | 12
2) Baseline data – current data on a particular indicator, e.g. in 2006, 42% of eligible voters voted in national elections 3) Target – expected changes to a particular indicator, e.g., from the 2006 data of 42% voter turnout, the policy is expected to increase voter turnout to 70% 4) Data collection strategies 5) Data analysis strategies 6) Reporting plan In building the monitoring system, you should bear in mind that: 1) 2) 3) 4)
There are results information needs at the project, program, and policy levels Results information needs to move both horizontally and vertically in the organization Demand for results information at each level needs to be identified Responsibility at each level needs to be clear for: a. What data are collected (source) b. When data are collected (frequency) c. How data are collected (methodology) d. Who collects the data e. Who analyzes the data f. For whom the data are collected g. Who reports the data
Monitoring and managing results will also involve analytical work and systematic learning on technical skills, expert knowledge and practical experience. The evidence gathered in building a monitoring system should be objective, thorough and relevant. It should also be effectively communicated into the ongoing policy process (Sutcliffe & Court, 2005). More importantly, every monitoring plan should have a sense of ownership, management, maintenance, and credibility (World Bank, 2000).
Evaluating after implementation Post-implementation evaluation becomes an integral part of the policy formulation process, when it answers questions such as the following: 1) 2) 3) 4) 5) 6)
Was the problem correctly identified, or was the correct problem identified? Were any important aspects overlooked? Were any important data left out of the analysis? Did this influence the analysis? Were recommendations properly implemented? Is the policy having the desired effect? Are there any needs for modification, change, or re-design? What should be done differently next time?
Evaluation determines whether policies have achieved their intended effects. It applies a range of social science research methods to public programs, which can guide in the planning and execution of policy evaluation. Page | 13
Policies can be evaluated using the following approaches: 1) Before-and-After Evaluation: a policy is evaluated for the changes it has produced since its implementation; the situation is controlled to exclude other possible influences on the outcome. 2) With-and-Without Evaluation: a policy is evaluated for producing changes in the target population, compared to another population without the policy. 3) After-Only Evaluation: the extent to which the policy goals were achieved, compared to the state of affairs before the policy was implemented; but the situation is not controlled to exclude other possible influences on the outcome. 4) Time-Series Evaluation: the changes produced by the policy, tracked over a long time period.
Page | 14
References Hayes, W. (2014, June 1). Defining Policy Formulation. Retrieved June 17, 2014, from The Public Policy Cycle Web Site: http://profwork.org/pp/formulate/define.html Head, B. W. (2008). Three Lenses of Evidence-Based Policy. The Australian Journal of Public Administration, 67(1), 1-11. Kirlin, J. J. (1984). Policy Formulation. In G. R. Gilbert (Ed.), Making and Managing Policy: Formulation, Analysis and Evaluation. New York: Marcel Dekker, Inc. Koliba, C., Meek, J. W., & Zia, A. (2011). Governance Networks in Public Administration and Public Policy. New York: CRC Press. Kraft, M., & Furlong, S. (2004). Public Policy: Politics, Analysis and Alternatives. CQ Press. Metz, M. (2005, November). Monitoring Policy Impacts (MPI): The Role of MPI in Policy Formulation and Implementation. Retrieved June 19, 2014, from Food and Agriculture Organization of the United Nations: http://www.fao.org/docs/up/easypol/382/mpi_pol_form_impl_056en.pdf Milakovich, M. E., & Gordon, G. J. (2008). Public Administration in America. Boston, MA: Wadsworth. Nakamura, R. T., & Smallwood, F. (1980). Policy Formation (Environment I). In R. T. Nakamura, & F. Smallwood, The Politics of Policy Implementation. New York: St Martin's Press. NCPAG-CPED. (2011). Public Policy Development and Analysis Process. Academic Course on Public Policy Development and Advocacy (pp. 8-41). Diliman, Quezon City: Center for Policy and Executive Development, UP NCPAG. Office of the First Minister and Deputy First Minister. (2006). Practical Guide to Policy Making in Northern Ireland. Retrieved June 19, 2014, from Office of the First Minister and Deputy First Minister: http://www.ofmdfmni.gov.uk/policylink-a-practical-guide-topolicy-making PIANGO/ADB RETA 6319. (2007). Policy Formulation and Analysis. Workshop for Staff of Regional Civil Society Organizations. Nadi, Fiji: PIANGO/ADB RETA. Sapru, R. K. (2004). Public Policy: Formulation, Implementation & Evaluation. New Delhi: Sterling Publishers. Sutcliffe, S., & Court, J. (2005). Evidence-Based Policymaking: What is it? How does it work? What relevance for developing countries? UK: Overseas Development Institute. World Bank. (2000). Designing and Building a Results-Based Monitoring and Evaluation System: A Tool for Public Sector Management. Retrieved June 19, 2014, from www.ideas-int.org/documents/Document.cfm?docID=87
Page | 15