Regulators’ Experimentation Toolkit • 2: Regulatory experiments
Part D: Designing and running an experiment This section of the guide provides an overview of the practical steps you’ll need to consider regardless of the type of experiment you’d like to run. We also include references to other practical resources where you can get more in-depth information.
Before you begin: ingredients for a successful experiment As we’ve seen, there are a variety of different forms an experiment can take. Despite these distinct methodologies, all types of experiment require the same essential ingredients to be successful: • A problem worth experimenting with As we’ve covered in Part B: Is regulatory experimentation right for you?, experiments are useful when you have a clear question that you believe you can test through an experimental design. Doing the work up front to make sure you’re asking the right question is essential. • A hypothesis to be tested or learning objectives to be fulfilled The hypothesis is a statement that sets out the expected outcome that is being tested through the experiment. For example, a hypothesis in the context of financial regulation could be “if we enable more open banking, then it will create better competition and choice for consumers.” The hypothesis is important for laying out the scope of the experiment and defining its focus. It should also help you identify your metrics – what you are going to measure to understand if the outcomes are being achieved? Thinking about metrics will also be important if you opt to define learning objectives instead of a hypothesis. • A counterfactual or baseline for comparison To understand the impact of an experiment, ideally you need to be able to compare it to what would have happened if it hadn’t taken place. Your counterfactual is an estimation of this, and is generated either by your control or comparison group – or, in a pre-post experiment, where you compare only one group before and after the experiment, you need to collect baseline metrics before you begin. • Monitoring and evaluation While monitoring looks at ‘what’ changes have occurred since the beginning of the experiment, evaluation looks at ‘whether’ the hypothesis has been confirmed or the learning objectives have been reached. In other words, has the intervention met its objectives and why. You’ll need to develop a plan for how you will analyze both. • A plan to apply the learning in practice An experiment will only be useful if you’re able to act on the results, so before you even begin you’ll want to have a good idea of what action you’ll take from the learnings.
36
