Appendices
concrete changes proposed; the publication of a ToR or reports to follow up on MOPAN products; and the mention of MOPAN assessments in official documentation as well as in the minutes of the governing bodies104. Most of this work was done by collecting information from the websites of assessed organisations (“web scraping”). This means that the level of information obtained at this stage depends to a certain extent on the transparency of the organisations assessed. This step ends with the consolidation of “reputed uses”, which are confirmed by several lines of evidence. Interviews are then conducted in which our claims on reputed uses are presented and discussed. In these interviews we aim to collect convincing narratives about MOPAN use. “Convincing” means that stakeholders can say who used the MOPAN assessment, for what, how, and why, and with what other knowledge products. In the process, the initial claims become increasingly specific. Vague statements, e.g. “MOPAN was used in the context of the evaluation function” are rephrased as “A finding of the MOPAN assessment triggered a study by the evaluation function which led to a change in X”. At the end of this process, the remaining claimed uses are specific (i.e., embedded in specific examples, with specific actors, supported by a rationale for change, and lead to a specific, traceable change) and robust. When they are rejected, it is possible to conclude that that specific use did not occur. However, other uses that have not been investigated remain plausible. To draw an analogy with statistics, the approach used here is very good at ensuring the validity of a claim of use and at rejecting Type I errors (false positive). It is less good at ensuring the validity of a non-use claim and at rejecting Type II errors (false negative), because cases of “reputed non-uses” have been voluntarily left aside. The ability to reject Type I errors improves the causal claims. Rejecting fewer Type II errors means that there could have been greater use than what was concluded. This approach leads to extremely robust cases of use. The focus in this process is very much on use. Whether the changes brought by the assessment ultimately led to improvements in performance and whether this performance led to better delivery on the ground is outside the scope of the case studies.
Use by Members Members’ identify use differently than MOs. Here the objective was to identify types of use and patterns explaining the use or lack of use rather than to know everything about how every member country used assessments. The process is as follows: Exploratory interviews are carried out with at least one representative in each MOPAN member country and used to identify different types of use and to feed into the ToC, especially the CCs. A list of empirical tests was then developed to support the five contribution claims related to members’ use. Approximately 60 such tests were developed about use; the context allowing use, and conditions. 104 This list is based on Delahais, T., & Lacouette-Fougère, C. (2019). Try again. Fail again. Fail better. Analysis of the contribution of 65 evaluations to the modernisation of public action in France. Evaluation, 1(3), 1‑18. P. 10.
108
