OPEN GOVERNANCE SCORECARD METHODOLOGY March 2015
INTRODUCTION A growing international movement of governments, international institutions, civil society organizations and advocates has made ‘open government’ a priority in recent years. Formerly disparate discussion groups and communities of practice seeking to advance transparency, access to information, social accountability and participation in policy have seized on the Open Government Partnership and other global debates to strategically align their areas of concern, engage national and international stakeholders and define a transformation agenda to improve government. Against this backdrop, more recent discussions have sought to more clearly define the agenda, and the specific legal and institutional changes needed to foster effective open governments. The Open Government Standards initiative, coordinated by Access Info Europe and convening over twenty national, regional and international organizations, has developed a set of standards for what the initiative considers the most important legal and institutional conditions for openness along three dimensions: transparency, participation and accountability. The work builds on a growing movement of practitioners and experts promoting the right to know, participation and accountability. Broad and diverse sets of principles and recommendations have emerged from these debates. As a result, there is increasing clarity over what it takes to foster open. This is important, because the legal and institutional changes needed are often extraordinary. Transparency International (TI) and its broad network of partner organizations and chapters have actively participated in these discussions, aware of the remaining challenges to ensure transformations lead to improvements in people’s lives. TI has developed a framework to engage existing debates and link them with a broader concern with governance. Governance is paramount for our partners because it captures a broader set of conditions than those associated with government. The concept moves beyond the traditional notion and functions of government, and considers the relationships between leaders, public institutions and citizens, their interaction and decision-making processes. Our effort to develop a framework for understanding open governance is partly driven by the dialogue described above. We want to learn from, and further the discussions, principles and standards stemming from the varied communities of practice and the open government movement. Our goal is to provide our partners and chapters with a useful tool for assessing their own government’s commitments to fostering open governance, effective participation in policy, substantial accountability and the realization of the right to know. In what follows, this note presents TI’s Open Governance Scorecard, including its methodology basic definitions, the type of indicators and) a brief outline of how the scorecard was completed and tested, including the sources used and the methods for validation.
1
TI’S OPEN GOVERNANCE SCORECARD PURPOSE The general goal of the scorecard is to provide TI national chapters and other organizations with a quick reference guide to the conditions required for open governance and a tool to assess whether basic legal and institutional conditions are met in each country. The specific objectives of the scorecard are: i. ii. iii. iv. v.
To identify gaps in the laws hindering accountability, i.e. transparency, participation and oversight. To explore diverse methods for assessing whether legal provisions and openness standards are met in practice. To identify what factors prevent identified good practice, and propose adjustments to policy, rules and government practices. To provide specific information to allow partners and TI chapters to formulate advocacy asks, and support advocacy strategies broadly. To provide a tool to track progress in promoting open governance in each country in the medium and long term.
The Open Governance Scorecard is intended as a tool for our partners and chapters, and it does not represent an attempt to develop extensive aggregate measurements or compare conditions prevailing in different countries. It appropriates and builds on some of the indicators that have been developed for comparison purposes, by Transparency International and other organizations, but it adapts the questions supporting each indicator to serve as a stand-alone test of whether a particular condition is met. To construct the assessment tool, TI carried out consultation processes with partners and chapters invested in assessing openness. Between January and September 2014, we worked with a group of specialists and advocates to adjust and extend the standards, to identify indicators and assessment tools already in use, and to formulate indicators when they had not already been drafted by partner communities of practice.
FRAMEWORK Transparency International’s Open Governance Scorecard assesses open governance around three pillars: transparency, participation and oversight. These three pillars contribute to accountability, responsiveness and efficiency of governments. Jointly, they can transform the relationship between citizens, politicians and public officials. The conditions enabling open governance through transparency, participation and oversight lead to specific accountability outcomes.
2
In order to achieve open governance, transparency, participation and oversight need to be implemented in a country; they need to be transformed from concepts into actions. The roadmap to achieve open governance consists of three key elements: 1. transparency and participation must be recognized as human rights; 2. the institutional architecture, policies and practices must exist to fulfil these rights and achieve perform control and oversight; 3. these policies and practices must be supported by the necessary tools and available infrastructure. Together, all three pillars and the key governance elements form the Open Governance framework on which the scorecard is designed. Figure 1 – Open Governance Framework
Elements to achieve open governance
Transparency
Participation
Control / Oversight
Acknowledgment of the right to know
Acknowledgment of the right to participate
Institutions & policies to fulfil the right to know
Institutions & policies to fulfil the right to participate
Institutions & policies to perform control / oversight
Tools & infrastructure to facilitate the fulfilment of the right to know
Tools & infrastructure to facilitate the fulfilment of the right to participate
Tools & infrastructure to facilitate control / oversight
Pillars of accountability
Open Governance
Source: own elaboration.
THE SCORECARD The Open Governance Scorecard is a dashboard that assesses whether basic conditions are met that foster open governance. The Open Governance Scorecard is based on a group of open governance standards1 that adapt many of the principles already developed by the open government movement. To facilitate assessment of what conditions need to change to advance open governance, and to gage progress once minimal conditions have been met, we developed a set of indicators based on the state of the art discussions and instruments for assessing and promoting accountability.
1
The Open Governance Standards are available in a separate document. 3
Because the scorecard assesses whether conditions are met, it does not use survey information, composite quantitative indicators for a particular result, or aggregation methods for comparing within each system. The indicators are questions that can be answered by an advocacy organization or researcher based on a set of specific evidence, (e.g. reviewing existing regulations or interviewing government authorities). An indicator can take three values following standard scorecard methodology:   
Basic condition / target is met; Basic condition / target is partially met; Basic condition / target is not met.
Where the question refers to a specific legal provision or practice which has no plausible intermediate answer the condition will either be met or not met. Where an indicator is only partially met, the scorecard asks for further information to discover why. Where a researcher considers condition to be met, they are to indicate the source and include a citation. Where the condition is only partially met or not met, researchers are to provide commentary.
STANDARDS AND INDICATORS The scorecard developed indicators to test a set of commonly agreed to standards. The standards are widely discussed, broadly accepted good practices anchoring the rights to know and participate, effective control and oversight, and open governance. The indicators are necessary conditions for achieving the desired good practice and warranting open governance, in three pillars: transparency, participation and oversight. A set of standards and in-law indicators were developed and piloted between March and May 2014. Where in-law indicators permitted furthering our inquiry to assess whether the conditions desired to meet a standard were met in-practice, we incorporated new specific questions. These questions are subindicators to assess whether conditions to meet a legal standard are met in-practice. Not all in-law indicators permit further inquiry, as some create a condition that is fully met by incorporating provisions in the laws or regulatory framework, so we do not have sub-indicators for inpractice assessment for all in-law indicators. We have proposed indicators on the basis of relevance (identifying which in-practice conditions are most relevant for meeting the standard selected), and depth (identifying for which standards provisions made in law are patently insufficient for warranting good practice). The result is a set of indicators includes 129 in-law indicators, and 93 in-practice indicators that break down into 330 questions. In total, the scorecard is composed of 459 questions.
4
Figure 3 – Dimensions, standards and indicators
Transparency
Participation
Control & Oversight
18 Standards
9 Standards
8 Standards
59 in-law indicators
32 in-law indicators
38 in-law indicators
44 indicators with in-practice considerations
20 indicators with in-practice considerations
29 indicators with in-practice considerations
144 sub-indicators for inpractice assessment
82 sub-indicators for inpractice assessment
104 sub-indicators for inpractice assessment
Source: own elaboration.
In developing the indicators, we used and make extensive reference to various instruments including the right to information legislation rating developed by Access Info Europe and the Canadian Center for Law and Democracy; the Global Integrity Report; the World Bank’s Public Accountability Mechanisms Initiative; and the Organization for Economic Cooperation and Development (OECD) Indicators for measuring openness in government (developed by Involve). All the indicators developed collaboratively for the scorecard attempt to be comprehensive—that is, they inquire about criteria that must be met for a condition to be satisfied; they inquire about specific proof, especially where the information requested must be incorporated in an aggregate, oversight or other government report; and they inquire about problems and challenges faced, especially when the condition tested requires the concurrence of citizens and government authorities. This last set of questions is not translated into an indicator testing a specific condition, but rather is expected to produce rich, diverse information that can warrant extensive qualitative analysis. The scorecard measures five types of conditions: Legal provisions, system arrangements, institutional mechanisms, disclosure practices and, in a small subset of cases, policy outputs. While in-law indicators test the legal provisions, the in-practice indicators and follow-up questions (subindicators) test all other conditions.
Legal provisions are specific rules anchoring transparency, participation and oversight systems, institutions and practices established in diverse legal instruments. System arrangements are processes through which institutions and branches of government interact between themselves and with citizens to facilitate oversight. Institutional mechanisms are processes through which a government branch / agency discloses information, facilitates participation or complies with oversight provisions. Disclosure practices are specific actions carried out by branches of government and agencies that afford citizens, groups and private entities access to public information.
5
Policy outputs are intermediate results stemming from the conditions established above. In a few cases, our indicators assess whether these specific outputs exist, for example when participation is taken into account to adjust strategies (thus policies not only are in place, but they reflect consultation processes); when resources are allocated to meet citizen demand or when information not previously disclosed is opened up through establishing a legal precedent.
DATA SOURCES AND VALIDATION The assessment considers four types of sources for the indicators: 1. direct reference to an official document where it exists (direct answers provided by government or agency authorities in a questionnaire or in depth interview, carried out by country researchers; 2. assessments of experts and organizations consulted through an in-depth interview, and 3. additional information provided by consulted expert organizations and practitioners. Direct Reference We asked researchers to use three four types of document as sources: 1. Laws and agency rules to check for specific legal provisions. 2. Report documents that should be public and available, conditions we will test directly, including: the access to information consolidated report, and other ATI specific documents and reports; the budget process documents; internal oversight reports and audits (comptroller general or equivalent agency); and the Supreme Audit Institution’s year-end report. 3. Proactively disclosed information available online that is not included in an overarching report, and will likely be agency specific, containing policy guidelines and directives, including: organizational and administrative information of policy programs for the agencies assessed; organizational and administrative information of the branches of government; directives and guidelines for open data, for ICT policy, for social accountability mechanisms, procurement and codes of conduct. 4. Specific documents that are not proactively disclosed but we will request to warrant and substantiate specific questions, including guidelines for receiving and processing complaints; the results of consultation, participation and social accountability processes; guidelines for financial and interest disclosure forms, as well as their audits, and others. Direct answers from public officials through questionnaire and interview For a large set of questions (where we are asking about institutional mechanisms, system arrangements and policy outputs), we formulated questions for each indicator and directed them to government officials and key staff in the legislative and judiciary. Where possible, researchers followed two tracks for consultation: 1) they sent a questionnaire requesting specific answers and information from specific
6
government officials, and 2) they requested an interview to formulate these and follow-up, questions depending on the answer. We sent questions to 19 government agencies: 1) Information Commission 2) Comptroller General or head of the Internal Control Agency 3) Supreme Audit Institution 4) Legislative Branch 5) Judiciary (Supreme Court only) 6) Ministry of Health 7) Ministry of Education 8) Ministry of Agriculture 9) Ministry in charge of Social Services 10) Ministry in charge of National Security 11) Ministry tasked with social inclusion and development
12) Ministry in charge of Economic Development 13) Agency in charge of Housing 14) Agency in charge of Human Rights defense and realization 15) Ministry in charge of Environmental Protection 16) Agency in charge of anti-trust and business regulation 17) Head of police agency 18) Ministry of Finance 19) Agency in charge of the ICT and Open Data policies
Assessments by experts and organizations In addition to the in-depth interviews with public officials in the agencies referred above, country researchers formulated similar questions to a group of experts, partners and civic organizations with expertise in the areas under scrutiny. To facilitate this exchange, researchers identified experts and organizations around five areas on inquiry: 1. Transparency: Organizations and experts adept at requesting information, and perusing the access to information system. 2. Participation: Organizations that participate in policy consultation processes at the national level. 3. Monitoring: Organizations that carry out monitoring and social accountability, and who participate directly in the provision of public services at the local level. 4. Legislative expertise: Organizations that work with Parliament / the Legislative branch, including committees. 5. Oversight: Experts on accountability and integrity systems in each country, the work of the internal and external control mechanisms. Additional information provided by experts and organizations participating in the assessment This includes specific evidence of problems faced by partner organizations when engaging authorities, to request information or to participate in the policy process; process information (what challenges have partner organizations faced, how did they circumvent them); and public information obtained by partner organizations through direct engagement with authorities. Here are some examples of additional information:  
the litigation paper trail of organizations’ requests for specific information that should be made public, but is not; redacted public versions of requested information that is reserved or confidential;
7
legislative and judicial branch background information used when deciding cases or approving specific legislation; evidence of participation in budget processes at the local or national level, and results; evidence of insufficient time or feedback loops when engaging in policy consultation processes; evidence and process information of direct participation in monitoring public services.
Asking for specific information on challenges faced by organizations already involved in promoting accountability and accessing information is the only way to include process information in the short time we have, and to explore the most relevant problems faced by organizations and citizens when opening government. Finally, all researchers were asked to request and consider secondary sources. A note on context and power analysis One of the most discussed aspects of the scorecard during our meetings with the advisory group was whether it would be able to pick up the context and power analysis necessary to really assess what prevents open governance, what the main obstacles are, and how they can be overcome. Participants from all countries suggested strong qualitative analysis was important, and two countries emphasized the need to complement the indicator set with sound political economy, context and power analyses. It was widely agreed that the in-practice assessment should complement indicator scoring with an extensive narrative that focuses on the context, power arrangements and institutional conditions of open governance. From the perspective of methods, we agreed researchers would focus on obtaining useful information for feeding this narrative when carrying out the assessment. We agreed to do this through the interviews and, where possible, focus groups with experts and organizations yielding three different types of information: 1) Specific information about processes our partners are engaged in, or were engaged in, suggestive of structural and systemic problems. 2) General, open questions to inquire about the challenges faced by organizations engaging authorities, which can be complemented with an analysis of context and influence to understand why problems persist. 3) Feedback and discussion from our partners, which will help researchers identify what problems are most relevant, and identify what conditions our partners perceive to be causing these problems. With these bases, country researchers are expected to provide a political economy analysis in the scorecard narrative tailored around the most glaring gaps between the standard and the legal and institutional conditions needed to meet them. This analysis will be developed in country reports and their presentation and discussion with government authorities.
Validation meeting and presentation After the completion of the each question in each indicator, researchers conduct a validation meeting with relevant stakeholders including government officials, academics, practitioners and civil society organizations to discuss the results of the assessment. In the meeting, researchers present the main draft findings and selected indictors, comments are sought and discussed. Comments, observations and corrections are incorporated to the scorecard and answers are modified as necessary. After the validation meeting the scorecard is finalized. The scorecard is presented in full with all data collected, including values, sources and researcher comments for each indicator. In addition, a short report is drafted which includes: country context, overall findings, key findings for each of the pillars, and conclusions and recommendations. 8