Ocean Nexus Special Report - Contributing Policy Analysis to Ocean Nexus A Primer on Our Approach

2023

OCEAN GOVERNANCE, LAW, AND POLICY

Contributing Policy Analysis to Ocean Nexus: A Primer on Our Approach

AUTHORS

OVERVIEW

Grant H. Blume, PhD Deputy Director, Policy Analysis, Nippon Foundation Ocean Nexus

By positioning policy analysis as a critical link that connects scholarly research to the political and administrative decision-making processes that create public policy, this manuscript provides an overview of the policy analysis process to the Ocean Nexus scholars, researchers, and analysts who work collaboratively on the pressing problems facing the world’s oceans.

Associate Teaching Professor, Evans School of Public Policy and Governance, University of Washington

Acknowledgements The author gratefully acknowledges support for this work from the Nippon Foundation Ocean Nexus Center at the University of Washington EarthLab. All opinions expressed in this manuscript are those solely of the author.

Table of contents Table of contents

2

Introduction

3

What is policy analysis?

4

The Four Phases of the Policy Analysis Process

5

Determine Scope

7

Set Parameters

12

Analyze

17

Communicate

22

Concluding Thoughts

23

Critique #1

23

Critique #2

24

Critique #3

24

References

26

Ocean Nexus Special Report Contributing Policy Analysis to Ocean Nexus: A Primer on Our Approach 2

Introduction Policy research is an integral part of most interdisciplinary domains of scholarly inquiry, yet

policy analysis is often missing from such literature. On the one hand this omission is not surprising since policy analysis is usually created for internal decision-making processes.

On the other hand, an implication of this missing policy analysis is that researchers, scholars, and policy analysts may lack a shared understanding of policy analysis's purpose and process. Such a lack of shared understanding, in turn, prevents policy analysis from informing decisions in the public domain.

By positioning policy analysis as a critical link that connects scholarly research to the political and administrative decision-making processes that create public policy, this manuscript provides an overview of the policy analysis process to the Ocean Nexus scholars, researchers, and analysts who work collaboratively on the pressing problems facing the world’s oceans. Current collaborations between these scholars and the Ocean Nexus policy analysis team are designed to generate actionable analysis that can impact ocean governance and the people who depend on the oceans. This collaboration will translate and extend Ocean Nexus research further into the domain of public policy through the policy analysis framework outlined in this manuscript. Moreover, our intentional focus on equity as a core principle means we aspire to introduce equity-oriented analysis and findings into the policy debates and conversations that shape, and are shaped by, research on the world's oceans.

The manuscript begins with a brief overview of key terms in policy analysis with a distinction between policy research and policy analysis. I then describe the four phases of the policy analysis process and explain how each phase represents an iterative process in which an analyst1 makes design choices and draws from research to implement analysis that is evidence-based and grounded in empirical findings. After discussing some of the challenges that arise when implementing policy analysis, I conclude with a response to some of the most common criticism levied against policy analysis as described in this manuscript.

1

Policy analysis is nearly always a team effort and will be implemented among Ocean Nexus projects in that spirit, but for the sake of parsimony in this manuscript I will refer to the "analyst" in the singular form.

Ocean Nexus Special Report Contributing Policy Analysis to Ocean Nexus: A Primer on Our Approach 3

What is policy analysis?

Dunn (2008) defines policy analysis as an applied social science discipline that uses multiple research methods to create, critically evaluate, and communicate policy-relevant knowledge. Aside from this technical definition, policy analysis may also be simply defined as a way for public actors to make informed decisions about solving problems. Such a parsimonious definition likewise reveals the inherent goal of policy analysis: to infuse evidence and analysis into public decision-making.2 Our focus on “problems” in policy analysis is theoretically grounded in a belief about liberal3 democracy. That is, we assume that government does not simply implement policy for policy's sake but that public policy is an instrument to solve problems and improve people's lives. In a liberal democracy, government seeks to minimize intrusion into the lives of citizens, use resources efficiently to solve problems, and draw from evidence and best practices to maximize the odds that public policy is effective. These goals drive the rationale behind using policy analysis to inform decision-making. While theory behind why some policy problems gain the attention of decision-makers while other policy problems languish for decades is beyond the scope of this manuscript, we can nonetheless acknowledge that once a policymaker decides to address a problem because of their own values and beliefs or because public sentiment compels such attention, the policy analyst steps in to deliver information to the policymaker on which informed policy decisions can be made.

In describing what policy analysis is, a helpful distinction can be made between policy research and policy analysis. Policy research is question-oriented and developed for a broad audience whereas policy analysis is problem-oriented, client-based, and generally narrow in its scope. Policy research draws from primary data and aspires to be inherently objective in its contribution to a body of scientific knowledge, whereas policy analysis draws from secondary data and research and is inherently subjective because the analysis is based on a decision-maker's values. These characteristics are explored further throughout this manuscript but the distinction between policy research and policy analysis

In the field of policy studies, the term “policy analysis” usually describes ex ante information designed to inform public decision-making whereas “program evaluation” describes the methods used to answer the ex post causal question: to what extent did a policy deliver its intended effects?

3 “Liberal” here is used to describe a political philosophy or ideology grounded in self-determination, personal responsibility, and a government organized to limit institutional power, in the spirit of such philosophers as John Locke, Jean-Jacques Rousseau, and John Stuart Mill.

2

Ocean Nexus Special Report Contributing Policy Analysis to Ocean Nexus: A Primer on Our Approach 4

is noted here to facilitate an understanding of the differences between the two and the ways in which policy research informs policy analysis.

A point also worth making explicit is that policy analysis reflects values. One decisionmaker may be interested in the equity implications of policy options while another decision-maker may care deeply about the extent to which policy options potentially infringe on individual liberties.4 This explicit acknowledgement of how values shape the criteria we use in policy analysis and, in turn, the political processes through which public policy is analyzed, chosen, and implemented, is a result of policy analysis's client-oriented focus. That is, if policy analysis aspires to deliver useful information to a decision-maker, the analyst must implement that decision-maker's values as criteria in the analysis.

The importance of assumptions is a final characteristic of policy analysis to make explicit before examining the four phases that comprise the policy analysis process. Assumptions must always be made explicit so a reader, or the decision-maker for whom the policy analysis is created, understands the assumptions on which the analysis is based. Policy analysis comprises choices and predictions about an uncertain future. Making the analysis’s assumptions explicit allows a reader or decision-maker to potentially disagree with the analyst's underlying assumption but regard their technical analysis as logically sound and empirically valid. Without making assumptions explicit, the analyst potentially opens the door to a reader questioning the overarching legitimacy of the analysis. We will return to the importance of assumptions in policy analysis in subsequent sections of this manuscript.

The Four Phases of the Policy Analysis Process Policy analysis is often presented as a linear set of steps (e.g., Dunn, 2008; Weimer & Vining, 2017) but policy analysts will often describe policy analysis as a fast-paced iterative process that requires flexibility, adaptation, and creativity (Bardach & Patashnik, 2019). For this reason, the following policy analysis “process” – that is, the steps undertaken to generate evidence-based information and analysis for public decision-making – are presented as four broad “phases” that represent inherently iterative sub-processes. These four phases are best defined as:

4

I do not offer these two examples to suggest one of these values is naturally opposed to the other but to simply illustrate how diversity of political thought manifests in policy analysis.

Ocean Nexus Special Report Contributing Policy Analysis to Ocean Nexus: A Primer on Our Approach 5

1. Determine scope 2. Set parameters 3. Analyze 4. Communicate For each of these phases I describe the iterative process involved, use an applied example to demonstrate the iterative implementation of the process, and describe the critical places where research fits into that phase of the process. For an applied example I will use the topic of salmon in the Pacific Northwest. To not bias the policy analysis phases reviewed in this section the topic being introduced here (“salmon”) is left intentionally vague at this point; we will periodically return to this example to demonstrate how policy analysis steps are applied.

A final note about the semantics of policy analysis: a comprehensive policy analysis process generally contains each of the four phases outlined here but policy analysis need not contain all these iterative phases to constitute a “policy analysis” outcome. For instance, empirical work that carefully defines a policy problem, maps the problem’s symptoms and underlying causes, and acknowledges the jurisdictional bounds of political power and public decision-making (relative to the problem as it is defined) can represent a productive first step in translating policy research to policy analysis. Likewise, systematically evaluating a set of context-specific policy options or alternatives may serve as the necessary conceptual bridge to connect a broad policy problem to a specific geopolitical jurisdiction in which the problem occurs. The essence of policy analysis ultimately comes down to intent: Is the intent to generate empirical analysis that supports public actors in making informed decisions about solving problems? If so, such analysis falls within the broad domain of policy analysis.

Ocean Nexus Special Report Contributing Policy Analysis to Ocean Nexus: A Primer on Our Approach 6

Determine Scope Figure 1: Determine Scope

We begin the policy analysis process by determining the project’s analytical scope. If we assume that the purpose of policy analysis is to generate information for public decision-making we must answer two interrelated questions up front: (1) Who is the decision-maker? (2) What is their capacity to realistically choose and implement the policy options we are evaluating in our analysis? We think carefully about this because we want to analyze a set of policy options on which our client (either real or hypothetical) can realistically act.5 The concurrent task in this first phase of determining scope – define the problem – is often the most challenging step in policy analysis. When we define a problem in policy analysis, we aspire to define the problem as simply as possible or at least disentangle as many problems as possible into a set of clearly defined problems. As we will discuss throughout this section, a clearly defined parsimonious definition of a policy problem is one of the most important components in effective, tractable policy analysis.

Defining a policy problem requires an iterative process by which we distill the problem's definition into its simplest terms. The problem's definition will be inherently normative because we assume problems are injurious and should be solved. Oftentimes we can begin to define a problem as “Too few…” or “Too many…” to reveal the baseline conundrum on which our policy analysis will focus. That is, “too few” of something suggests a need for more whereas "too many" suggests that government has a role in reducing this unit of the problem.

5

If we are a policy analyst who works for a public decision-maker then this first step is moot because the decision-maker is our boss; in the case of Ocean Nexus, we can be creative with the client we choose based on the problem we are analyzing and any access we may have to actual policymakers who can make decisions to affect the problem at hand.

Ocean Nexus Special Report Contributing Policy Analysis to Ocean Nexus: A Primer on Our Approach 7

The rationale behind striving for a simple definition of the problem is twofold. First, working to define a policy problem in its simplest terms leads to a policy problem that is easily understood by both a decision-maker and the problem's relevant stakeholders, which in turn optimizes the problem's tractability in the political sphere (Bardach & Patashnik, 2019; Peters, 2005). Second, a clearly defined problem facilitates the next step in the policy analysis process – setting parameters – by (1) revealing the mechanism through which policy options may address the problem's causes and (2) defining criteria in a way that makes such criteria relevant and measurable (more in this in the next section). Put succinctly, a clear definition of the problem reveals what we will count to gauge the effectiveness of the policy options considered in the analysis.

Defining a policy problem is an exercise in which a problem's symptoms and causes are “mapped out” by the analyst. Mapping symptoms and causes is a conceptual tool through which an analyst disentangles multiple problems that co-exist in a policy space. A policy problem's symptoms are generally the observable “units” of the problem that we see around us (the “too few” or “too many” of something), and often what appear in media reports as attention around a problem builds. Causes, on the other hand, are the source of the problem but are likely harder to observe and potentially contested among experts, stakeholders, and decision-makers. Ideally the analyst identifies a problem's symptoms and causes clearly because a policy option that targets a problem's symptoms will potentially have very different results, especially in the long-term, compared to a policy option that targets a problem's underlying causes. And whereas we aspire to be as simple and parsimonious as possible in defining a policy problem, such guidelines can be relaxed when defining the underlying cause(s) of a problem. The critical purpose in cataloging the

causes of a problem is to identify those that fall within the jurisdiction of what the analyst's client or decision-maker can realistically target with public policy interventions.

We can turn to our applied salmon problem to demonstrate how these ideas are implemented in practice. In the past three decades, wild salmon have declined precipitously in the Pacific Northwest (Levin & Schiewe, 2001; Miller et al., 2014; Spromberg & Scholz, 2011; Walters et al., 2019). Declines in salmon have captured the attention of a broad group of stakeholders and decision-makers at multiple levels of government (i.e., regional, state, federal, and international). Although the interrelated problems are numerous, we can define two policy problems related to wild salmon for the purpose of our example:

Ocean Nexus Special Report Contributing Policy Analysis to Ocean Nexus: A Primer on Our Approach 8

Policy Problem 1A: Too few wild salmon reach their spawning ground.

Policy Problem 1B: Too few members of the Umatilla tribe can access wild salmon.

Given the broad scope of these problems, we can further refine these problems in a manner that bounds their geographic scope and directs us toward potential decision-makers who have an interest in the problem and the capacity to act: Policy Problem 2A: Too few wild salmon reach their spawning ground in Oregon.

Policy Problem 1B: Too few members of the Umatilla tribe can access wild salmon in Oregon.

Policy problems 2A and 2B now suggest that the governor of Oregon could be a potential decision-maker with the jurisdiction to address the problem. Federal decision-makers (e.g., a U.S. senator or representative) could also consider policies to address this problem.

Policy problems 2A and 2B are related but distinct from each other, in large part based on the problem's focus. In policy problem 2A, the “unit” of which there are “too few” is salmon; in policy problem 2B, the focus is the Umatilla tribe's access to salmon, or more simply, members of the Umatilla tribe. Both problems are important, worthy of a policymaker's attention, and may share similar underlying causes (e.g., stream degradation, dams, overfishing) and symptoms (fewer fish). Yet the nuanced difference between these problems will compound throughout the policy analysis process to ultimately yield two distinct bodies of information to inform potential decision-making processes.

The problems are also distinct from each other based on their orientation. Problem 1A is a biological/ecological problem in which the focus is salmon populations and their ability to reproduce; problem 1B is a problem focused on a group of people, who in this case have been historically oppressed and exploited by the U.S. government (Lewis, 1995; Wilson, 1998). This focus on a particular group of people who have historically had unequal access to public goods and resources (often as the result of intentional barriers created by government) means that policy problem 2B embeds an equity orientation in the definition of the policy problem compared to how policy problem 2A defines the problem. This equity-oriented definition of the problem is an important consideration for Ocean Nexus scholars who may aspire to build policy analysis with equity as its centerpiece.

The applied example related to salmon helps to illustrate two more nuances of policy problems worth noting. First, tractable policy problems for policy analysis often avoid Ocean Nexus Special Report Contributing Policy Analysis to Ocean Nexus: A Primer on Our Approach 9

embedding the solution to the policy problem in the definition of the problem. For instance, problems related to salmon in the Pacific Northwest could be defined as “Too many wild salmon are getting eaten by sea lions” or “Too few wild salmon are able to pass through dams.” These problem definitions are potentially too narrow for policy analysis because they reveal, or at least are highly suggestive of, one particular policy solution (e.g., eliminate the sea lions eating salmon and build fish ladders or remove dams, respectively).

Second, the definition of the problem should also focus on the problem and not the fact that “too few policies” or “too much regulation” lead to the problem. One could easily argue that “too few policies exist to protect salmon in Oregon.” Such a problem definition intuitively leads to the conclusion that the policy answer to the problem is “more policies” but does not reveal the kind of nuanced analysis that policy analysis is intended to deliver to a decision-maker.

In the case of policy problems with embedded solutions or those that imply a lack of policy is the problem, one way to test the parsimony and tractability of a policy problem is to simply ask “So what?” If asking “So what?” reveals another layer to the problem, then that subsequent layer of the problem may deserve the attention of the analyst. If we ask “So what?” to the problem that too few wild salmon are able to pass through hydroelectric dams, then we may logically come to the policy problems defined as 2A and 2B. Asking “so what” is also an easy heuristic to employ if the problem's definition initially appears to be “too few policies…” or “too little regulation…” because asking “So what?” or put another way, “Why do we care that there are too few policies to...?” helps reveal the essence of the problem. It is this essence of the problem around which we aspire to build our policy analysis.

Research and empirical evidence play a critical role in multiple aspects of defining a policy problem, mapping a problem's causes and symptoms, and identifying a viable decision-maker for the problem. Most importantly, extant research is critical to building evidence to demonstrate the problem is indeed a problem that warrants the attention of decision-makers. Policy analysts must draw from research to demonstrate the magnitude of the problem and validate the severity of the problem's symptoms (i.e., the data that represent observable aspects of the problem). Research also demonstrates linkages between a problem and its causes, thereby supporting how an analyst maps a policy problem to the causes which policy options will target. Finally, extant research can inform the process through which analysts identify decision-makers that have the authority or jurisdiction to address a policy problem. Returning to our salmon example, there are vast bodies of literature that explore the decline of wild salmon over time, the ecological, Ocean Nexus Special Report Contributing Policy Analysis to Ocean Nexus: A Primer on Our Approach 10

industrial, and commercial causes attributed to such declines, and the geopolitical dimensions of salmon. Depending on how the problem is defined, some or all of these domains of research would provide essential information to inform and bolster the analyst's efforts to appropriately determine this project's scope.

One of the greatest challenges in defining a problem is making the choice to analyze one problem at the expense of not analyzing others. This can be particularly agonizing when an interconnected web of policy problems requires urgent attention; by choosing one problem the analyst may feel like the other problems are being ignored. Dunn (2008) can help us make sense of why we need to disentangle policy problems to identify the well-defined problems that are tractable within a policy analysis framework. Dunn conceptualizes the distinction between well-structured and ill-structured problems and describes these problems across five characteristics (Table 1).

Table 1: Characteristics of Well-Structured and Ill-Structured Problems Well-Structured

Ill-Structured

Decision-Makers

One or Few

Many

Policy Options

Limited

Unlimited

Values

Consensus

Conflict

Outcomes

Degrees of

Unknown

Certainty Probabilities

Calculable

Incalculable

Note. Adapted from Dunn (2008).

What appears in Table 1 will become increasingly intuitive as we continue to explore the four phases of the policy analysis process in this manuscript; the well-defined problems are those with a clear decision-maker, a feasible set of policy options, a general consensus on the values (i.e., criteria) around which the analysis should be implemented, an ability to project outcomes within a range of certainty, and an ability (to varying extents) to predict the probabilities of such outcomes. The point worth noting from Table 1 is that while society's attention and concern is often drawn to ill-structured problems (e.g., climate change, poverty, institutional racism) these problems span numerous jurisdictions and levels of government, not to mention a limitless number of possible policy solutions. The design and delivery of targeted public policy, on the other hand, depends on identifying and analyzing well-structured problems. In short, faced with the challenge of defining and analyzing a single policy problem, the analyst should feel confident knowing that analyzing Ocean Nexus Special Report Contributing Policy Analysis to Ocean Nexus: A Primer on Our Approach 11

one problem will likely provide information of greater value to a decision-maker than the delivery of scattershot analysis that tries to encapsulate a web of related problems.

Set Parameters Figure 2: Set parameters

Policy options are the interventions, initiatives, laws, programs, and regulations that a decision-maker may consider to address the problem they face. The term is intentionally vague here to capture a broad range of options. Each policy problem will lend itself to a set of policy options appropriate to the policy problem at hand but the choice of which policy options are included in the analysis is ultimately up to the analyst working on behalf of the decision-maker.

There is no set formula for how to identify viable policy options for analysis but a few practices are common among analysts. First, analysts will often look for “promising practices” from other jurisdictions where analogous cases suggest a policy option that was effective in that context might be effective in the context being analyzed. Promising practices may have been implemented or simply proposed by other decision-makers, but in either circumstance such insight may be transferable to the policy problem being analyzed by the analyst in search of viable policy options.

Analysts may try to balance a mix of direct and indirect policy options to create a set of diverse approaches a decision-maker may use to address a policy problem (Salamon, 2001). Direct options are considered those directly implemented and enforced by public actors, government, or governance networks. Such direct options include the direct provision of goods and services, regulations enforced by government, and government-sponsored information campaigns designed to change the behavior of individuals or firms. Indirect policy options, on the other hand, leverage markets to address policy problems. Indirect policy options are numerous and include such common interventions as market-based fees and permits, contracting with private or nonprofit firms to provide a good or service, and the use of vouchers to facilitate individuals’ consumption of a market-based product or service.

Ocean Nexus Special Report Contributing Policy Analysis to Ocean Nexus: A Primer on Our Approach 12

The third practice analysts will often employ to define a set of policy options is to embrace specificity in defining each policy option. Given that the goal of policy analysis is to deliver detailed, tractable information to a decision-maker to address a policy problem, the tractability of an analysis will always correlate with the specificity of the options being analyzed. In other words, an analyst analyzing “tax rebates” as a theoretical intervention produces a vague analysis compared to an analyst who analyzes a $1,000 tax rebate based on three eligibility factors with a forecasted 75% utilization rate; the former could be considered a type of policy research whereas the latter falls squarely within the domain of policy analysis.

Finally, the analyst will always present the decision-maker with a policy option that represents that status quo. Capturing the status quo as a distinct “policy option” is important for two reasons. First, the status quo should capture all the policy initiatives (if any) targeting the policy problem that are underway or forthcoming within the decision-maker’s jurisdiction. This assessment of current and near-term policy leads to the second important reason for including the status quo; an analyst should provide this information to the decision-maker so as to reveal the implications of a choice to do nothing. That is, what will happen if none of the analyzed policy options are chosen by the decision-maker? In some cases where robust policy initiatives are already underway, the problem may improve substantially without further intervention; in other cases, a problem may be rapidly worsening and the choice of no action spells disaster. In either case, the utility of understanding the status quo is an important factor in the analysis.

Turning to the second component of the phase in which parameters are established, criteria in policy analysis represent the measures against which policy options will be systematically evaluated. Like with policy options, the criteria included in the analysis represent choices that an analyst must make with their decision-maker client in mind. The fluidity and contextual nature of policy analysis means there is no single standard set of criteria required for analysis, but the following criteria generally appear in some form across most policy analyses. These criteria are briefly described here with the caveat that a policy analyst will always need to carefully define each criterion in the analysis with an explicit description for how each criterion is measured consistently across policy options given the context of the policy problem.

Ocean Nexus Special Report Contributing Policy Analysis to Ocean Nexus: A Primer on Our Approach 13

Effectiveness The nature of policy problems means of utmost importance is the question: to what extent will this policy option address the problem at hand? Yet embedded in this criterion is the often-perplexing question: What will we count to determine if the problem is getting better? In other words, how do we measure improvement? The choice of what to count is likewise a choice of what is not counted (Stone, 2010), which means an analyst must choose a measure of effectiveness carefully and with a clear rationale.

Equity Conventional policy analysis tends to conceptualize equity in distributional terms but the implementation of equity as a criterion can (and should) extend across multiple dimensions. In a distributional sense equity attempts to measure the extent to which a policy option allocates a policy's benefits or burdens intentionally, and in doing so may acknowledge historical or contemporary oppression, marginalization, or exploitation of a group or class of people. Equity can also be analyzed with a focus on decision-making processes, known as procedural equity; the time horizon for when a policy’s costs and benefits are realized, often described as intergenerational equity; and the extent to which a policy potentially achieves representation for otherwise marginalized groups affected by the problem, which could be described as recognitional equity. Place-specific social norms, institutions that mediate how policy options play out in practice, and the spatial- temporal scale of the policy problem all converge to make equity context dependent (Pascual et al., 2014). Acknowledging the context in which equity is analyzed also reveals the initial conditions (i.e., potential inequity) that is present at the policy’s starting point (McDermott et al., 2013).

Cost The scarcity of public resources means a public decision-maker will always need to gauge the cost of a policy option against both the policy option's effectiveness, the relative cost of the other options being analyzed, and the extant public resources available or earmarked to address the problem. For these multiple reasons a cost criterion must be carefully defined and applied consistently against each policy option.

Ocean Nexus Special Report Contributing Policy Analysis to Ocean Nexus: A Primer on Our Approach 14

Manageability Since policy options are never implemented in a vacuum a policy analyst must decide how to systematically measure one or more aspects of manageability across options. For instance, “implementation feasibility” as a criterion may seek to gauge the administrative complexity involved with implementing each policy. A dimension of manageability could also capture the extent to which a policy option utilizes existing administrative infrastructure or requires a new administrative or bureaucratic system for its management (Salamon, 2001, labels this a policy option's level of “automaticity”). As with other criteria, manageability will be specified as a criterion based on a decision-maker's priorities.

In practice a criterion can be defined and implemented in multiple ways. For instance, equity may be implemented as a criterion across multiple groups of people and dimensions (e.g., intergenerational equity, procedural equity). Costs may be measured separately as two criteria to gauge a policy’s startup costs distinct from its annual costs, potentially revealing important differences that would otherwise be obscured by a broad cost criterion. Likewise, if a set of policy options have varying degrees of implementation timelines that affect the timeframe to generate observable results, effectiveness may be measured in a manner that captures short-term effectiveness (a policy option’s observable results after 12 months) and long-term effectiveness (a policy option’s observable results after 10 years). The choices for how criteria are defined are ultimately a decision made by the analyst, based on the analyst’s goal to deliver the most useful information possible to their decision-maker client.

Choosing and defining criteria involves the difficult type of choice-making similar to the task of defining a policy problem; like with choosing one policy problem at the expense of not choosing others, the analyst must decide on a set of criteria which inevitably means a decision on which criteria to not include. In choosing criteria an analyst uses their judgement to include what type of information the client most likely wants to see but also must also consider what information the decision-maker needs to see. In other words, important trade-offs among policy options can only emerge if the analyst has chosen diverse, informative criteria through which to analyze the policy options.

The added challenge with criteria is that the analyst not only selects the criteria but must clearly define how each criterion will be systematically measured across each policy option. Choices about how to define effectiveness and equity, for instance, have a Ocean Nexus Special Report Contributing Policy Analysis to Ocean Nexus: A Primer on Our Approach 15

consequential impact on how the chosen policy options and their trade-offs are considered. Likewise, the analyst has a responsibility to implement a cost criterion in a manner that generates the most informative cost estimates relative to the information the decision-maker needs to best steward scarce public resources. Without careful attention to how criteria are applied across each policy option, the analyst risks creating a policy analysis product with the appearance of criteria that are chosen to simply endorse one preordained policy option over others.

Returning to our hypothetical salmon example, we can use our two different problems to demonstrate the nuanced way that criteria may vary based on how we define a problem. If we define the problem as “Too few wild salmon reach their spawning ground in Oregon” then we would count the number of wild salmon that reach their spawning grounds as our central measure of a policy's effectiveness. Alternatively, if we define the problem as “Too few members of the Umatilla tribe can access wild salmon” then we would judge effectiveness by generating estimates for the increased number of tribal members who can access salmon based on each policy option.

This example reveals an interesting pattern that can occur in policy analysis. Consider again the policy problem that “Too few wild salmon reach their spawning ground in Oregon” and assume the decision-maker cares about the fact that a symptom of this problem is that too few members of the Umatilla tribe have access to wild salmon. Knowing this about their client, the analyst may choose to define equity here as the extent to which a policy option's benefits are accessible to the Umatilla people, which may be more formally operationalized as an equity criterion that measures the increased number of tribal members that would be able to access wild salmon based on a policy option. Note this definition of equity as a criterion for policy problem 1A mirrors the definition of

effectiveness for policy problem 1B. This demonstrates how equity, in particular, can be conceptualized in multiple ways within the policy analysis framework presented here.

Identifying policy options and criteria often constitute an iterative process because the analyst is attempting to define each criterion in a manner that is systematically applicable across each policy option. In other words, if a definition of effectiveness is only feasibly applied across three of four policy options in an analysis, the analyst is faced with the choice of redefining effectiveness in a manner that applies to all four options, or modifying the nonconforming policy option in a way that is more amenable to the effectiveness criterion's specifications. The goal with such iteration is to develop analysis that allows for the greatest amount of comparability between options as possible. This type of lateral or Ocean Nexus Special Report Contributing Policy Analysis to Ocean Nexus: A Primer on Our Approach 16

direct comparison facilitates the analysis of trade-offs, which is a central task of the next phase of the policy analysis process.

As with the earlier step of defining a policy problem and identifying an appropriate decision-maker, research and empirical evidence factor centrally in setting the parameters of policy analysis. Research that explores the effectiveness of policy initiatives and interventions, for example, serves as a critical source for identifying policy option candidates for analysis. Likewise, empirical case studies can be a rich source of evidence cataloging what has worked, or not worked, in specific contexts relevant to a policy problem at hand.

Research similarly informs the way in which criteria are chosen and analyzed. In some cases, an analyst may turn to a body of literature to adopt a conventional measure of effectiveness that is widely used in scholarship on the policy problem. Research on inequities or disparities can also help an analyst construct an equity criterion that is grounded in empirical evidence. Cost-benefit and cost-effectiveness research provide an intuitive foundation to construct cost-related criteria.

Analyze Figure 3: Analyze

The preceding two phases – determining scope and setting parameters – demonstrate that a substantial amount of critical thought and effort occur prior to implementing a systematic evaluation of policy options against a slate of well-defined criteria. The “analyze” phase of the policy analysis process comprises two important tasks whereby the analyst is implementing the actual analysis and continuously assessing the key trade-offs to emerge from the analysis.

Implementing the analysis marks the beginning of a new phase defined by a substantial amount of empirical investigation. Because each policy will be evaluated against each criterion the analyst is faced with a set of data points equal to the number of policy options, m, multiplied by the number of criteria, n. In other words, the analyst must Ocean Nexus Special Report Contributing Policy Analysis to Ocean Nexus: A Primer on Our Approach 17

generate m x n data points based on the best and most reliable research and evidence available.

A longstanding practice in policy analysis is to assemble the m x n data points in a policy analysis matrix (Figure 4). The policy analysis matrix serves as a powerful tool for the analyst to inventory and organize their empirical analysis as it unfolds. The matrix may not appear in a final report or presentation but the organization of data points in the matrix format represents an important undertaking along the policy analysis process.

Figure 4: Prototypical policy analysis matrix Criterion #1

Criterion #2

Criterion #3

Criterion #4

Policy Option #1 Policy Option #2 Policy Option #3 Status Quo

The m x n cells in the matrix, which become the empirical foundation of the analysis, represent the analyst’s best estimates and predictions for each policy option relative to each criterion. For instance, consider the analyst's assessment of Policy Option #1’s effectiveness; holding all else equal, in a hypothetical world in which Policy Option #1 is implemented, the analyst predicts the estimated change in the “units” of the policy problem attributed to this policy option. Similarly for a cost criterion, the analyst calculates and records in the appropriate cell the estimated cost of Policy Option #1 (based on how the cost criterion is defined). Whenever possible an analyst attempts to provide quantified estimates for these predictions to maximize the utility of the information delivered to the decision-maker. Such criteria as effectiveness, cost, equity, and timelines are quantified as precisely as possible. When point estimates are not feasible an analyst may generate predicted outcomes using a categorial scale (e.g., low, medium, high) or a numeric scale if such intervals intuitively align with the criterion (e.g., a quantified assessment of implementation feasibility that corresponds to a scale ranging from 1 to 5 which each interval's value clearly defined).

Ocean Nexus Special Report Contributing Policy Analysis to Ocean Nexus: A Primer on Our Approach 18

The words “predict” and “estimate” are critical to note in describing criteria because policy analysis is always conducted with a focus on the future. That is, the analyst must predict a hypothetical outcome for each policy option against each carefully defined criterion. These estimates and predictions, however, will never be made with complete certainty because public policy is always deliberated, chosen, and implemented in an uncertain, ever-changing political world (Manksi, 2013). Yet even in such an unpredictable political world, policy analysts can account for this uncertainty in two ways. First, uncertainty can be reflected in a policy option's estimates and predictions by using a range instead of a point estimate. As with wider confidence intervals reflecting a greater range of likely values for a statistical estimate, a predicted policy outcome reported as a range in policy analysis can reflect the degree to which the analyst is uncertain in making such an estimate. A stand-alone uncertainty criterion is the second way in which an analyst may account for the inherent uncertainty in an analysis. In this manner a criterion seeks to assess a policy option's overall uncertainty. Such a criterion can provide valuable information if some policy options are novel and untested and hence have little research bolstering the estimates and predictions generated by the analyst.

Assessing trade-offs is the second task undertaken in the “Analyze” phase of policy analysis. The essence of analyzing trade-offs is summed up in the observation that in the content of policy analysis and assessing trade-offs, “If you are on one side of an issue, you are on the wrong side” (Winter, 2013, p. xiv). Winter's (2013) quip captures the essence of tradeoffs – and policy analysis more generally – in that policy analysis will never reveal a “good” or “bad” policy or point the decision-maker to the “right” choice. Rather, policy analysis reveals the key tradeoffs that oftentimes represent the inherent complexity of solving a policy problem.

Conventional tradeoffs in policy analysis include assumptions that effectiveness and cost are positively correlated (i.e., a more effective policy will carry a higher cost; Wheelen, 2011) and a negative correlation between efficiency and equity (i.e., achieving greater equity corresponds to decreased efficiency; Okun, 1975). In reality, these conventions about effectiveness-and-cost and equity-and-efficiency tradeoffs are dogmatic and not grounded in empirical evidence (Le Grand, 1990; Stone, 2011). Undertaking policy analysis to reveal key trade-offs is precisely why an analyst must empirically determine the extent to which, if at all, a tradeoff between effectiveness and cost emerges in the analysis.

Perhaps with this in mind Wildavsky (1980), an early pioneer of policy analysis, observed that policy analysts “are strongly committed to thought experiments” of all kinds (p. 16).

Ocean Nexus Special Report Contributing Policy Analysis to Ocean Nexus: A Primer on Our Approach 19

Research and empirical evidence play a central role in systematically evaluating policy options against clearly defined criteria. Every cell in an m x n policy analysis matrix will likely have at least one citation or reference to support the estimate or prediction generated by the analyst. The analyst's task in this phase is to produce a parsimonious prediction – a policy option's estimated effectiveness or level of equity, for instance – knowing that each figure in a matrix cell may represent many hours of consuming research, talking with experts, and ultimately settling on a conclusion that is empirically sound.

Returning to our applied wild salmon example, imagine an analyst is implementing analysis for the problem “Too few wild salmon reach their spawning ground in Oregon” with criteria including: ●

Short-term effectiveness, defined as the statewide increase in wild salmon five years after the policy's full implementation.

●

Long-term effectiveness, defined as the statewide increase in wild salmon 25 years after the policy's full implementation.

●

Distributional equity, defined as the number of additional members of the Umatilla tribe that would have access to wild salmon.

●

Intergenerational equity, assessed on an ordinal scale to measure a policy's benefits and burdens relative to present and future generations.

●

Cost, measured as the total cost in dollars of the policy to Oregon state government over ten years.

●

Implementation feasibility, assessed on an ordinal scale to measure relative complexity of the policy option's implementation.

The decision-maker is Oregon’s governor, and the options6 the analyst has chosen to include in the analysis are: ●

A program to implement habitat restoration

●

A ban on recreational and commercial salmon fishing

●

An end to hatchery-stocked salmon

●

The status quo

6

For simplicity’s sake these policy options are borrowed from Hall and White (2008) and Lackey (2017). Note that dam removal, a widely studied policy option to address declining salmon populations (Harnish et al., 2014; Stanley & Doyle, 2003), is not included here since a governor in the United States would be unable to implement such a policy option; dam removal is a policy option for a federal client since such a policy option would require extensive review, approval, and implementation by the federal government.

Ocean Nexus Special Report Contributing Policy Analysis to Ocean Nexus: A Primer on Our Approach 20

Armed with these parameters the analyst begins to estimate the effectiveness of each option. The analyst would draw from research, interviews with experts, and other sources to make these predictions for the short-term and long-term effectiveness of each policy option. Upon generating effectiveness estimates for each option, the analyst would turn to generating estimates for the remaining criteria.

In the case of distributional equity, the analyst would assess the extent to which each policy option intentionally redistributed or allocated a policy's benefits to the Umatilla tribe.

The elimination of hatchery-stocked salmon, for instance, has been theorized as a potential means to free up habitat and food supply for wild salmon (Lackey, 2017) but hatchery salmon are also a critical source of sustenance for Indigenous people (Northwest Indian Fisheries Commission, 2016). Hence the analyst would need to think carefully about the equity implications of a policy option to end hatchery-stocked salmon since the intervention could generate a modest increase in wild salmon (captured in the effectiveness criterion) but lead to an overall negative impact for the Umatilla people if the community loses a source of sustenance in the short term. The analyst would continue their analysis by carrying out this type of thorough evaluation of each policy against each criterion.

Communicate Figure 5: Communicate

The final phase of the policy analysis process entails an iterative process of writing the results of the analysis, presenting results to the decision-maker client and key stakeholders, and listening openly to others to understand how the analysis can be bolstered, adapted, or extended in a manner that ultimately meets the needs of the decision-maker. In drafting a report, the analyst will recall that policy analysis is nearly always delivered to a decision-maker who has a sense of urgency to solve policy problems

Ocean Nexus Special Report Contributing Policy Analysis to Ocean Nexus: A Primer on Our Approach 21

and limited time to read lengthy reports. This dynamic means policy analysis reports should be concise and always err on the side of brevity, with technical or detailed information organized in appendices for a reader interested in such background.

Policy analysis should not stop with the delivery of a report to a client. Rather, policy analysis should be communicated – with the analyst open to listening and feedback – so that the information being delivered to a decision-maker is as relevant as possible to the context of the policy problem at hand. Bardach and Patashnik (2019) provide useful advice in this domain by suggesting a policy analyst should share their findings with potential critics of their work. Getting this kind of critique can bolster the analysis and afford the analyst an opportunity to proactively anticipate and respond to criticism that may be levied against the analysis.

Another important consideration in this communication phase of policy analysis is sharing the analysis with stakeholders affected by the problem. Unfortunately, this type of interaction – between policy analysts and stakeholders who have an interest in the analysis – is not a widespread practice (Walters, Aydelotte, & Miller 2014). Such interactions and communication are likely critical, however, for the decision-maker who seeks an inclusive process and especially in circumstances where a policy problem is intentionally defined with an equity focus.

Research, in this phase, plays an obvious role in the composition of the written analysis.

The analyst until now has kept extensive notes and written work to document the numerous places where research has factored centrally in the prior three phases of the policy analysis process. In this final stage the analyst’s responsibility is to carefully document the research on which the analysis is based, not just to lend legitimacy to the analysis but also to provide readers with the means to further investigate the scholarly articles and research reports of interest that may have informed the policy analysis at hand.

In the final application of the example of wild salmon in Oregon, the analyst with a drafted policy analysis report would have numerous stakeholders and audiences to engage from which they could learn and refine their analysis. Assuming the analyst does not have a background in ecology or biology, an intuitive group of stakeholders for feedback would be university researchers or scientists from the state’s Department of Fish and Wildlife. The analyst may want to engage a trade association or advocacy groups, such as the Northwest Sportfishing Industry Association or the Save Our Wild Salmon Coalition, respectively. Finally, the analyst would want to present findings of the analysis and listen Ocean Nexus Special Report Contributing Policy Analysis to Ocean Nexus: A Primer on Our Approach 22

carefully to the perspective of the Umatilla people in Oregon to understand the extent to which the analysis adequately reflects the perspective and interests of the community.

Concluding Thoughts This manuscript ends with a look at a few of the common critiques of policy analysis. A brief response to each critique is offered with the caveat that each of these critiques could easily entail a more nuanced discussion than what space allows for here.

Critique #1: Policy analysis excludes the voice of the public and disenfranchised communities.

This critique can indeed be valid for a policy analyst who does not engage external stakeholders or draw from a broad range of resources from diverse communities and researchers. Policy analysts risk developing insular analysis when they draw from a narrow range of sources. Moreover, the analyst’s choice to privilege certain types of research and evidence over others (e.g., industry-sponsored research, or research generated by a particular school of ideological think tanks) can likewise exclude the public or disenfranchised communities that are unrepresented in such bodies of work.

Walters, Aydelotte, & Miller (2014) note that a common critique of policy analysis along these lines is that “policy analysis undermines democratic institutions by replacing public participation and debate with esoteric expert analysis” (p. 458). Our work with Ocean Nexus scholars will aspire to avoid this pitfall by casting a wide net in collecting research and evidence that supports a broad range of perspectives. Carefully and intentionally implementing equity – either as the lens through which a problem is defined, as a carefully-constructed criterion, or both – will also decrease the odds that our policy analysis excludes the perspective and voice of those who have a rightful stake in the problem being analyzed.

Critique #2: Policy analysis is technocratic and inaccessible.

Policy analysis, especially highly technical analysis such as cost-benefit analyses, are specialized undertakings that may not be accessible to a broad audience. An analyst may also risk designing highly technical analysis for other analysts and decision-makers at the expense of more accessible material created for a decision-maker but also available to a broad audience. Yet even for the analysis that is complex and specialized in its analytic approach, the analyst always has an opportunity and obligation to present the analysis in a nontechnical manner to maximize the accessibility of their work.

Ocean Nexus Special Report Contributing Policy Analysis to Ocean Nexus: A Primer on Our Approach 23

This is particularly salient when it comes to presenting and discussing the key trade-offs to emerge from the analysis. Such tradeoffs may be the product of highly technical specifications, but the essence of the tradeoffs can be communicated in a manner that resonates with nontechnical audiences. Head (2015) notes one promising approach to overcoming the perception of policy analysis's inaccessibility is “knowledge-brokering” which “goes beyond simply ‘telling’ others about” policy analysis and instead seeks to “add value for end users of knowledge through various types of dialogue and coproduction of insights in new contexts” (p. 479). The collaborative policy analysis carried out among Ocean Nexus researchers and the policy analysis team can foster such collaboration and co-production when an analysis simultaneously strives for both methodological rigor and accessibility.

Critique #3: Policy analysis ignores the reality that policy decisions are inherently political.

Choices about public policy are indeed political and rarely follow a rational, linear decision-making process (Stone, 2013). Policy analysis does not deny this reality. In fact, acknowledging the political environment in which public policy is created, policy analysis can be a valuable tool to deliver relevant information to decision-makers that facilitates compromises and potentially dulls divisive political rhetoric (Weimer & Vining, 2017).

A policy analyst generally approaches policy analysis with pragmatism, acknowledging that information provided to a decision-maker through policy analysis will inform a policy decision but never wholly determine a decision in the absence of the decision-maker's political considerations. Policy analysis “introduces ideas into politics,” notes Shulock (1999, p. 240). Similarly, policy analysis can exist in the political domain to explain a decision-maker's rationale for policy choices with an acknowledgement that “using evidence to support a political position should not be dismissed as politicization of science but rather as conducive to high‐quality democratic discourse” (Schlaufer, Stucki, & Sager, 2018, p. 648). This pragmatic approach to policy analysis aligns directly with the aspiration of Ocean Nexus to bridge the gap between decision-makers, policy makers, and the people most dependent on the oceans.

These critiques of policy analysis are worth noting because in light of the utility and merits discussed in this manuscript, the intent of this material is not to construe policy analysis as

the panacea for good governance. Rather, policy analysis can serve as a critical link

Ocean Nexus Special Report Contributing Policy Analysis to Ocean Nexus: A Primer on Our Approach 24

between scholarly research and the political decision-making processes that create public policy. Evidence-based public decision-making depends on a policy analysis process in which research and evidence is infused at every phase of the process.

The underlying rationale behind policy analysis is a belief that society is better off if policy analysis imbues evidence-based information into an inherently political process of decision-making. The careful implementation of the four phases outlined in this manuscript do not guarantee high-quality policy analysis per se; rather, the four phases provide a roadmap that, if followed, can increase the value of information delivered to public decision-makers. The roadmap for high-quality policy analysis is iterative and uncertain, yet such an endeavor is critically needed to connect research and evidence to the policy process through which public policy problems are solved.

Ocean Nexus Special Report Contributing Policy Analysis to Ocean Nexus: A Primer on Our Approach 25

References Bardach, E. S., & Patashnik, E. M. (2019). A Practical Guide for Policy Analysis: The

Eightfold Path to More Effective Problem Solving (6th ed.). CQ Press.

Dunn, W. (2008). Public Policy Analysis: An Introduction (3rd ed.). Pearson Prentice Hall.

Hall, T. E., & White, D. D. (2008). Representing recovery: science and local control in the framing of US Pacific Northwest salmon policy. Human Ecology Review, 32-45.

Harnish R. A., Sharma, R., McMichael, G. A., Langshaw, R. B., & Pearsons T. N. (2014).

Effect of hydroelectric dam operations on the freshwater productivity of a Columbia River fall Chinook salmon population. Canadian Journal of Fisheries and Aquatic

Sciences, 71(4), 602-615.

Head, B. W. (2016). Toward more “evidence‐informed” policy making? Public

Administration Review, 76(3), 472-484.

Lackey, R. T. (2017). Science and salmon recovery. In E. Weber, D. Lach, B. Steel (Eds.)

New Strategies for Wicked Problems: Science and Solutions in the 21st Century (pp. 69-94). Oregon State University Press.

Le Grand, J. (1990). Equity vs. efficiency: The elusive tradeoff. Ethics, 100(3). 554-568.

Lewis, D. R. (1995). Native Americans and the environment: a survey of twentieth-century issues. American Indian Quarterly, 19(3), 423-450.

Levin, P. S., & Schiewe, M. H. (2001). Preserving salmon biodiversity: The number of Pacific salmon has declined dramatically. But the loss of genetic diversity may be a bigger problem. American Scientist, 89(3), 220-227.

Manski, C. (2013). Public Policy in an Uncertain World: Analysis and Decisions. Harvard University Press.

McDermott, M., Mahanty, S., & Schreckenberg, K. (2013). Examining equity: A multidimensional framework for assessing equity in payments for ecosystem services. Environmental Science and Policy, 33, 416-427.

Miller, K. M., Teffer, A., Tucker, S., Li, S., Schulze, A. D., Trudel, M., ... & Hinch, S. G. (2014).

Infectious disease, shifting climates, and opportunistic predators: cumulative factors potentially impacting wild salmon declines. Evolutionary Applications, 7(7), 812-855.

Northwest Indian Fisheries Commission. (2016). Questions and answers on tribal salmon

fisheries. https://nwifc.org/about-us/fisheries-management/questions-andanswers-on-tribal-salmon-fisheries/ Okun, A. (1975). Efficiency and equity: The Big Tradeoff. Brookings Institution.

Pascual, U., Phelps, J., Garmendia, E., Brown, K., Corbera, E., Martin, A., ... & Muradian, R.

Ocean Nexus Special Report Contributing Policy Analysis to Ocean Nexus: A Primer on Our Approach 26

(2014). Social equity matters in payments for ecosystem services. Bioscience,

64(11), 1027-1036.

Peters, G. B. (2005). The problem of policy problems. Journal of Comparative Policy

Analysis, 7(4), 349-370.

Salamon, L. (2001). The Tools of Government: A Guide to the New Governance. Oxford University Press.

Schlaufer, C., Stucki, I., & Sager, F. (2018). The political use of evidence and its contribution to democratic discourse. Public Administration Review, 78(4), 645-649.

Shulock, N. (1999). The paradox of policy analysis: If it is not used, why do we produce so much of it? Journal of Policy Analysis and Management, 18(2), 226-244.

Spromberg, J. A., & Scholz, N. L. (2011). Estimating the future decline of wild coho salmon populations resulting from early spawner die‐offs in urbanizing watersheds of the Pacific Northwest, USA. Integrated Environmental Assessment and Management,

7(4), 648-656.

Stanley, E. H., & Doyle, M. W. (2003). Trading off: the ecological effects of dam removal.

Frontiers in Ecology and the Environment, 1(1), 15-22.

Stone, D. (2013). Policy Paradox: The Art of Political Decision Making. Norton.

Walters, L. C., Aydelotte, J., & Miller, J. (2014). Putting more public in policy analysis. In N. C. Roberts (Ed.), The Age of Direct Citizen Participation (pp. 458-472). Routledge.

Walters, C., English, K., Korman, J., & Hilborn, R. (2019). The managed decline of British Columbia's commercial salmon fishery. Marine Policy, 101, 25-32.

Weimer, D. L., & Vining, A. R. (2017). Policy Analysis: Concepts and Practice (6th ed.).

Routledge.

Wheelen, C. (2011). Introduction to Public Policy. W. W. Norton & Company.

Wildavsky, A. (1980). Speaking Truth to Power: The Art and Craft of Policy Analysis.

Springer.

Wilson, J. (1998). The earth shall weep: A history of Native America. Grove Press.

Winter, H. (2013). Trade-Offs: An introduction to Economic Thinking and Social Issues.

University of Chicago Press.

Ocean Nexus Special Report Contributing Policy Analysis to Ocean Nexus: A Primer on Our Approach 27