Ian Goldman #AfricaEvidence Forum

African Evidence Forum 6-7 March 2018

Reflections on use of evidence by governments in Africa Dr Ian Goldman 6 March 2018

1

5

What is the problem (DPME research, 2011) Problem s not treated as an opportunity f or learning and im provem ent

54%

Senior management do not champion M&E and honesty about performance

45%

M &E is regarded as the job of the M &E unit, not all m anagers

44%

There is not a strong culture of M&E in the department

Problems not treated as opportunity to learn

40%

M &E is seen as policing and a way of controlling staf f

39%

The M &E unit has little inf luence in the departm ent

39%

Fear of adm itting m istakes or problem s

33%

The hierarchy m akes it dif ficult to openly and robustly discuss perf orm ance

29%

Little respect f or evidence-based decisionm aking in the departm ent

27%

Resistance f rom of f icials to transparent decision-m aking processes

13%

Problem s are concealed

13% 0%

10%

20%

30%

Source: Umlaw et al, African Evaluation Journal, 2015

40%

50%

60%

Just repeated for Twende countries but not finished analysis 2

Type of evidence most often used for policy decision-making (interviews with 54 senior managers) All Research Synthesis Scientific Research Formal Research More of

Substantive

Current

Informal Opinion None 0

5

10

15

20

25

Paine Cronin and Sadan, African Evaluation Journal, 2015, Adapted from: Hayes, W, 2002, The Policy Cycle 3

8

The main factors that influence policy (interviews with 54 senior managers) Evidence Pragmatics & Contingencies

Experience & Expertise Future Current

Judgement Resources Values Habits & Traditions Lobbyists & Pressure Groups 0

5

10

15

20

25

30

Paine Cronin and Sadan, African Evaluation Journal, 2015, Adapted from: Hayes, W, 2002, The Policy Cycle 4

9

Key elements of complex context • •

• • •

• • •

Suffering from colonial history – legacy lack of/poor education Dominant political economy - powerful groups in society appropriating resources so continuing burden of inequality which means growth much less effective. How to deal with that in evidence context?

Often weak institutions Senior managers very political – often change with new government Limited resources: • Need to be efficient and effective • Need to focus • Need to be innovative in models – often inherited colonial models of service delivery Dominance of north continues from donors – Paris agreement often token and donor systems undermine country systems Bureaucratic state not very responsive to citizens, and has consuming logic where can spend all time and energy How to address the development challenges in this context?

5

What is the problem?

Programme being evaluated in South Africa Early Childhood Development (ECD) Business Process Services Programme Grade R (reception year of schooling) Nutrition Programmes addressing under 5s Land Recapit. and Dev Prog (RECAP) Comprehensive Rural Dev Prog (CRDP) Export Marketing Investment Assistance Comprehensive Agricultural Support Programme Land Restitution Programme Government Coordination System (clusters/MinMECs/Implementation Forums) Micro Agric Financial Institution (MAFISA)

Outcome needed from evaluation results Close Major Minor No changes changes changes needed needed needed

X X X X X X X

X X X

X

6

14

Policy/programme cycle Intervention

Analysis of the problem and options

Options for addressing the problem

Theory of change

Understanding the root causes

Diagnosis stage missed

Design Diagnosing

What is known about the problem

to see what working/not

Outcome & impact

Value for money?

Policy/programme outcome and impact

Planning/ design

Docum ent, reflect, learn

Agenda What is the change – Not enough desired and evaluation undesired?

Policy/Programme Planning/design

Are planned outcomes being achieved?

Output

Review, refine and continue

Operational plan and resourcing

Implementing the plan

Monitoring the plan, environment and budget Policy/programme

Implementation and monitoring 7

4 main types of evidence (Vaka Yiko)

8

What can we see on supply Item Total no of national evaluations completed or underway at 31 Dec 2016 No. of evaluations started in 2016 Completed evaluation reports

Benin 15 (from 2010) 1 14

Uganda 23 (from 2008) 4 14

South Africa 56 (from 2012) 8 32

• SA/Benin/Uganda countries developed national evaluation systems, WACIE in West Africa

• Some work on systematic reviews in Health • Tested out models for synthesis and rapid response using synthesis – on topics and on evaluations (Smallholder) – SA – Environmental Affairs, DPME – Uganda - Makerere University

• Data – surveys (CSOs), panel surveys eg Uganda and SA, linking up with SDGs, 6 monthly Cabinet retreats in Uganda build on performance information • Capacity development: – Various countries training eg CLEAR DETPA, Harmattan School, unis, collab curriculum project for Twende Mbele

9

What can we see on demand • Work with Parliamentary committees – – – –

CLEAR/Twende Mbele – many countries AFIDEP – Kenya, Malawi DPME in SA APNODE

• Training senior decision-makers eg Vaka Yiko in Ghana (200+), SA (300+) • Shared experience – with Latin America eg Mexico, Colombia – other African governments, notably through Twende Mbele African M&E partnership – focus on improving demand (eg Ministers, Parliamentarians, top levels of public service)

10

Different ways evidence used • • • •

• • • •

Remember symbolic, process, conceptual, instrumental use Monitoring – do what planned and be held accountable – mechanical operation, track against plans – most countries, innovative examples eg Barazas and Cabinet 6 monthly retreats in Uganda Using existing evidence, ART roll out in SA, rapid response service in Makerere, Uganda; ECD Diagnostic Review in SA (200 existing studies) Learn from others/weaknesses – Uganda, Benin taking on MPAT, Uganda/SA/Benin with national evaluation systems (Twende Mbele) – danger of mimicry Research to see potential poverty reduction methods – eg National Minimum Wage in SA Develop evidence base for sector and use to inform policy (Human Settlements, SA) Delivery Units/Big Fast Results – Tanzania, Uganda, SA – bring stakeholders together in intensive labs, diagnose, develop plan and budget, then intensive M&E (expensive) Experiment with new – learn from experience and scale up – youth wage subsidy in SA, CSG in SA, Old Age Grant in Uganda, Kenya government experimenting with evaluations.

11

Barriers and facilitators n refers to number of studies Top 5 barriers to use of evidence

Top 5 facilitators of evidence use

Availability and access to research/ improved dissemination (n = 63)

Availability and access to research/ improved dissemination (n = 65)

Clarity/relevance/reliability of research findings (n = 54)

Collaboration (n = 49)

Timing/opportunity (n = 42)

Clarity/relevance/reliability of research findings (n = 46)

Policymaker research skills (n = 26)

Relationship with policymakers (n = 39)

Costs (n = 25)

Relationship with researchers/ info staff (n = 37)

Source: Oliver et al. BMC Health Services Research 2014, 14:2 http://www.biomedcentral.com/1472-6963/14/2 12

How do we make sure evidence makes a difference Social and political need – common identification of problem

Credible and contextually relevant evidence at the right time

Emotional response – needs a story which interacts with the reality of policy-makers

Relationship between producers, knowledge brokers, and policy-makers – and ideally co-production (problem of supply chain

13

Challenges • Complex context - departmental sensitivities/rivalries, need to learn about systems, need for consultation (co-production) – role of internal and external knowledge brokers • Short-term problem solving vs long-term agenda and institutionalisation • Cost and time of evaluations and systematic reviews – importance of process (implementation evaluations) • Too much focus on new evidence rather than existing • Who - researchers, evaluators, synthesis experts, other data sources and analysis • Skills - limited skills outside health sector in synthesis methods, impact evaluations, analytical skills to use data • Quality of data • Limited attention to wider ecosystem - capacity development, communication • Limited communication of results • Improvement plan process 14

Lessons (Vaka Yiko) Guideline 1. Use a broad definition of ‘robust evidence’ • New and existing evidence • Better use of data, especially admin data • Grey literature not just peer reviewed

Guideline 2. Link evidence needs to policy priorities • Recognise which battles to fight - political necessity eg Zuma on HE • Need credible champions who want to make a difference but people change so need broad breadth of supporters • Message that support political agenda – but how can evidence support how to implement • Tools for short-term problem solving as well as long term evidence agenda – eg in course for DGs consider 2 weeks, 6 months, 2 years

Wills, A., Tshangela, M., Shaxson, L., Datta, A., and Matomela, B. (2016) Guidelines and good practices for evidence-informed policymaking in a government department. Pretoria: Department of Environmental Affairs; and London: Overseas Development Institute.

15

Lessons (Vaka Yiko) Guideline 3. Link an evidence-informed approach with business planning, budgeting and reporting • SA linking through budget prioritisation process and legislation to link to annual planning, SEIAS for policy development • Uganda/Benin/SA seek to link to planning, budgeting (Twende Mbele activity)

Guideline 4.

Ensure evidence processes are inclusive and participatory • Twende Mbele testing out role of CSOs in national M&E processes

Guideline 5. Work towards co-design and co-production of evidence and policy: • Co-production even where use SPs • Focus on technical people, decisionmakers, politicians, knowledge brokers and affected stakeholders

Wills, A., Tshangela, M., Shaxson, L., Datta, A., and Matomela, B. (2016) Guidelines and good practices for evidence-informed policymaking in a government department. Pretoria: Department of Environmental Affairs; and London: Overseas Development Institute.

16

Plea to you • • • • •

• • • • •

Timescale - quick and not-too-dirty (eg REA) and longer term Understand political agenda – essential for relationship and common departure point Be careful of systems which depend on N expertise eg IEs: – Uganda example Think about systems – and how support the whole ecosystem Need for donors to support ongoing system development over long period (10 years) and adapt their evidence systems to support local systems More diverse evidence (vs ODI) More sharing across countries – eg Twende supporting development of NEP in Ghana, can lessen need for consultants More learning-by-doing capacity development and online support Building partnerships – policy-maker, implementor, research eg with ACE, challenge with contracting model And finally – it’s a complex and messy world – please dive in!

17

Thank You Dr Ian Goldman Acting DDG: Evaluation, Evidence and Knowledge Systems Department of Planning, M&E ian@dpme.gov.za @iangoldmansa

18