USAF Capacity Building- Monitoring and Evaluation

MONITORING & EVALUATION for UNIVERSAL SERVICE PROGRAMS/PROJECTS

Introduction • Personal – Former CEO USF Pakistan – First employee - built USF Pakistan and ran it successfully for nearly 5 years – Now a consultant in several countries of Asia and Africa

• This Webinar – NOT about M&E itself – Rather about M&E in USOF Projects & Programs!

– After an introduction to the concept of M&E, “setting the stage”: how to set up M&E in a Universal Service Organisation

Agenda • Part-1 – Session 1- Setting the stage – Session 2- Monitoring – Session 3- Evaluation

• Part-2 – – – – – –

Session 1- Organizational Capacity Assessment Session 2- Sourcing Options Session 3- In-House M&E Session 4- Outsourced M&E Session 5- Procurement – if outsourced Session 6- Worked Example for Bid Evaluation

Part 1, Session 1

SETTING THE STAGE

Project life

Goals

Objectives

Inputs

Outputs

Focus of Monitoring

Outcomes

Impacts

Focus of Evaluation

Definitions INDICATOR - a metric for measurement of inputs, processes, outputs, outcomes, & impacts for projects, programs, or strategies.

INPUTS - the main resources required to undertake activities & to produce outputs.

OUTPUTS - physical &/or tangible goods / services delivered by the project.

Describes scope of the project OUTCOME - what the project intends to accomplish by the end of it’s implementation.

IMPACT - result of the project, accrued in medium to long term.

Part1, Session 2

MONITORING

Monitoring is an ongoing process by which stakeholders obtain regular feedback on the progress being made towards achievement of their planned objectives.

PLAN

Planning Phase

PROGRESS

Implementation Phase

Monitoring should be able to respond to 2 questions

Are we taking the actions we said we would take?

Are we making progress on achieving the results that we said we wanted to achieve?

…all this is measured thru Indicators.

Org level

Project level

Examples of Indicators – Telephone lines per 100 persons – Computer usage rate in intervened area – Mobile subscribers per 100 persons (segregated geographically) – Internet availability per 100 persons (based on technology) – Volume of local content developed – Funds spent in each region divided politically, geographically, etc – No of contracts awarded – Subsidy amount disbursed – % of operational expenditures against contribution collected

What to Monitor?

Quality Contracts Schedules Scope Risk

Human Resources Other factors

Who Monitors?

Monitoring Tools

• Periodic Reports • Computer based project management tools.

Project

• Network performance reports e.g: network usage report, billing info, QoS report, call success ratios, etc.

• GIS Maps • Tech. Parameters validation through drive test & other tech. metrics

Basic Design of Monitoring Framework

What is to be measured – indicators?

How it will be measured & recorded?

Who will measure the information?

How frequently info will be measured & reported?

Part 1, Session 3

EVALUATION

What is Evaluation?

EVALUATION - a thorough & independent assessment of activities to determine the extent to which they are achieving stated objectives.

It is a periodic, objective assessment of an ongoing or completed project, program, or policy

Parameters of Evaluation

It relates to specific

It is carried out at discrete

questions regarding

points in time, often

design, implementation

seeking perspective of

and results.

outside experts.

Types of Evaluation – broadly Outcome/Impact Evaluation • investigates nature of relationship between planned inputs & outputs, & their outcomes • assesses, what difference (impact) the intervention made • and how much of the impact is attributable to the Program/Project.

Program Performance Evaluation addresses issues like • relevance to overall goal of org.

• efficiency in delivery of inputs, • effectiveness of approach, • timeliness of approach • other questions regarding project/ program design, management & operational decision making

When Evaluations are conducted?

Program / Project

Start

Baseline

Evaluation

?

Interim

Evaluations

End

Later

Post

Evaluation

for long-term Impact Analysis

When is the Post Evaluation conducted?

Whenever sponsors want to address question like: – In case of a Pilot project: What would be potential development contributions of an innovative new program (before it is started)?

– In case of a successful small-scale Program: To what extent & under what circumstances could it be replicated on a larger scale or with different population groups?

– In case of a completed Program: What has been the contribution of the intervention (made thru the Program)?

How to conduct Impact Evaluation?

Evalua8on E 0 Baseline

-­‐

Evalua8on E C Post Evalua3on

Evalua8on could cover

Project Beneficiary

• Demographic • Socio -­‐ Economic • Technology • Usage pa@ern • Other relevant indicators

Impact= Ec – Eo Approach -­‐ 1 Also called “Before & After method” Start Time

T0

Close Time T C

How to conduct Impact Evaluation?

Evalua8on EEwithout without Evalua8on

Evalua8on Ewith

Project Beneficiary

• Demographic • Socio -­‐ Economic • Technology • Usage pa@ern • Other relevant indicators

Impact= Ewith -­‐ Ewithout

Evalua8on could cover

Approach -­‐ 2

Project Non-­‐Beneficiary

-­‐

Also called “With & Without method” Start Time

T0

Close Time

TC

Evaluation Tools

• Surveys Project

• Beneficiary Assessment • Econometric Analysis- Regression • Case Studies • Cost Benefit Analysis • Cost Effectiveness Analysis

Info. required for Impact Evaluation (example Telecenter) General information about area Mapping of public & private facilities (schools, hospitals, post offices, ‌) Demographic Info. (No. of persons - age group & gender segregated) Number of persons who can access computers (gender-wise) Cost per use of computer, PCO, & other services Persons using computer for educational purpose, health, others Persons using computer for web based earning, Average distance & time from nearest computer center Other uses of computers & ICTs Number of computers per 100 population Number of persons who can access computer and internet Number of persons who know how to use computer

Info. required for Impact Evaluation (example Telecenters) Information on Purpose of using computers Info on Internet & Computers for learning Info on status of Info on status of Info on status of Info on status of

E-commerce E-Health E-governance E-Agriculture

Info on use of computer in social networking Info on economic activities associated with internet & computer Info on status of networking Info on presence of local content in cyber space Info on financial sustainability Info on community participation in management of center

PART-2

Part 2, Session 1

ORGANIZATIONAL CAPACITY ASSESSMENT TEST – One of the ways!

Comparison of 3 scenarios for M&E implementation Scenario 1

Scenario 2

Scenario 3

Sufficient capacity exists within the org.

Capacity exists but not sufficient

Substantial Capacity gaps

• Independent M&E unit • • • • •

• Capacity to execute M&E but with gaps Capacity to execute M&E • M&E plan not M&E plan is available developed Necessary staff available • M&E team deficient Necessary hardware & in terms of HR software are available • Deficiencies in Office-space is available hardware & software

• No M&E set up of any kind • No dedicated human resource • No infrastructure

OCAT*- Sample Test Question-1

Score

What is the organizational structure in context of M&E? Response A separate M&E Unit with clear reporting lines existing, indicate a strong M&E setup viable for in-house M&E A dedicated M&E human resource with no clear reporting line implies a weak M&E system within an organization No M&E system within an organization structure

* Organizational Capacity Assessment Test

Scores 2 1 0

(‌ cont.) OCATQuestion-2

Sample Test Scores

What is the current M&E setup - human, financial, and technical resources?

2.1 Human resources 2.2 Budget (financial allocation) for M&E 2.3 Equipment available for M&E

Response Sufficient Needs Improvement Non-existent

Scores 2 1 0

(….. Cont.) OCAT-

Sample Test

Question-3

Score

What is the rationale for creating M&E within the organization (Management’s perspective)?

Response Clearly defined objectives, and policy exists regarding M&E Management aware about M&E’s importance but no written policy or objectives exist No information available

Scores 2 1 0

(‌.. Cont.) OCAT-

Sample Test

Question-4

Scores

What skills exist within the organization in the following areas; 4.1 Domain Knowledge of Monitoring & Evaluation 4.2 Experience of working and demonstrated capability of managing M&E 4.3 Designing and conducting research/surveys 4.4 Data collection, management, and reporting 4.5 Statistical Analysis 4.6 Knowledge about econometrics 4.7 Using research findings for decision making through recommendations 4.8 Project management 4.9 Development of computer based M&E system Response Sufficient Needs Improvement Non-existent

Scores 2 1 0

(….. cont)

OCAT- Sample Test

• Scoring Recommendations – Maximum attainable score = 28 – Score above 90% indicates sufficient capacity for In-house M&E – Score between 60-89% indicates capacity, which need to be built – Score less than 60% indicates weak capacity, where Outsourced M&E is the most plausible option

Part 2, Session 2

SOURCING OPTIONS

M&E Sourcing Options

OCAT Results

Sufficient Cap.

In-house

Insufficient Cap.

In-house with Cap. Building

Cap. Gaps

Out-source

Steps for M&E implementation in the 3 scenarios Scenario 1

Scenario 2

Scenario 3

Implement M&E inhouse if not already being done

Implement M&E in-house, but: • Acquire deficient HR

Outsource M&E by engaging a professional firm

• Develop M&E team • Develop capacity of team thru professional training • Procure necessary HW & SW • Develop M&E Plan • Execute M&E Plan • Outsource functions which are not feasible or are occasional

How to decide?

Build In-House

Outsource

Possible reasons of selecting one over the other In-house

Out-sourced

• Cost - may be less expensive in long run

• Cost - less expensive to outsource (eg: there isn’t lot of work)

• Required expertise is available

• Lack of in-house capacity too big

• Want direct control over the process

• Outside Providers’ research & specialized know-how exceeds by far

• Desire of better quality • Flexibility in managing system configuration

• External validation is required

• Reliable & competent service providers not available

• A legal requirement?

• Routine nature of task

• To keep in-house manpower small • Notion that third-party M&E is considered more “neutral”

Part 2, Session 3

IN-HOUSE M&E

Structure of a good M&E Unit - Functions

Manager

Survey

Enumerators

Monitoring

Engineers

Field Coordinators

Evaluation

Database Coordinators

Data Analyst

Reporting

Sociologist/ Economist

HR Requirements for In-house M&E (pg 1 of 2) S. No Designation

Experience

Qualification

10+ years in Project Management + Masters Level Degree — Monitoring & Evaluation Management, Social Sciences, & Engineering of similar projects in development sector

No.

1

Manager

2

Monitoring 5+ years in Monitoring Specialist

Masters Level Degree — Management, and Social Sciences

1

3

E v a l u a t i o n 5+ years of conducting Specialist Evaluation

Masters Level Degree — Social Sciences

1

4

S u r v e y 5 + years in conducting Specialist surveys

Masters Level Degree — Management, and Social Sciences

1

R e p o r t i n g (5+ years in Specialist Report-writing

Masters Level Degree — Social Sciences (economics/ sociology or other relevant disciplines)

1

5

1

Pg 2 of 2) HR

Requirements for In-house M&E

S. No Designation Experience

Minimum qualification

6

5+ years of in capturing S o c i o l o g i s t / socio-economic aspects Economist of development interventions

Masters Level Degree — Social Sciences (economics/ sociology or other relevant disciplines)

7

Engineers

3+ years

Bachelors Level — Engineering Telecom

8

Data Analyst

3+ years in conducting data analysis & interpretation of data

Masters Level Degree -Management or Statistics

9

Masters Level Degree — D a t a b a s e 5+ years in developing & Computer Sciences, Database Coordinator managing databases Management

10

F i e l d 3+ years in field coordinators coordination

11

Enumerators

3+ years in enumerating surveys

Nos. 1

Workload dependent 1

1

Bachelors Level Degree -Sociology + other relevant disciplines

Workload dependent

Bachelors Level Degree -Sociology + other relevant disciplines

Workload dependent

Material Requirements for In-house M&E S.No Item 1

Hardware

2

Software

3

Drive Test Equipment

4

Communication Equipment

5

Furniture

6

Vehicles

7

Office

8

Misc.

Quantities

Part 2, Session 4

OUTSOURCED M&E

Selection Process to outsource M&E

Receipt Invitation of applications to prequalify from Firms

Shortlist Evaluation of the preof applications qualified

RfP to the shortlisted

Selection Evaluation of the Outof proposals sourcing Firm!

Task List for the Consultant/Firm • Develop a detailed configuration of M&E unit incl: identification of HR, job descriptions, competency levels & organization structure • Develop M&E Plan to address: What will be done? How it will be done? Who will do it? When it will be done? How much resources are required?

• Build capacity of the staff for M&E; • Develop computer based M&E reporting system, along with user manuals & documentation; • Develop M&E Operational Manual for the staff; • Install hardware & software for M&E & train operators; • Train __ persons to conduct/supervise data-collection process; • Design & conduct baseline surveys in project areas; • Develop impact assessment methodology and & conduct impact assessment;

(.. Cont)

Task List for the Consultant/Firm

• Identify problems with M&E system & come up with solutions to those problems; • Develop transition plan to support USF in taking over M&E • Provide refresher training on M&E to USF staff; • Oversee design & conduct of various M&E activities by USF; • Review the results of studies undertaken & propose improvements; • Assist in data analysis & interpretation; • Review the system & propose amendments if required; • Provide technical support for maintaining computer based M&E system.

• Ensure that staff is receiving adequate support to implement M&E functions & that data collection & analysis is on schedule

If out-sourced, some HR would still be required

At least with 5 basic skills: 1) Project Management 2) Contract Management 3) Financial Controlling 4) Technical Monitoring 5) Socio-Economic Monitoring

Min HR required to monitor Out-sourced M&E S. No Designation Experience

Min. qualification

Nos.

1.

M a n a g e r 5+ years in project M&E management

Masters level — Management, Social Sciences, or Engineering

1

2.

3+ years in capturing socioSociologist/ economic aspects Economist of development interventions

Masters level — Social Sciences (economics/sociology or other similar disciplines)

1

Engineer

Bachelors level — Engineering Electronics / Telecom

Workload dependent

3.

3+ years in field

Part 2, Session 5

PROCUREMENT DOCUMENTATION

List of Important Documents for Pre-qualifications

1. Invitation to Prequalify 2. Instructions to Applicants 3. Prequalification Datasheet 4. Prequalification Evaluation Criteria 5. Prequalification Application Form.

Major parts of RfP for M&E

1. Introduction 2. General Instructions to Bidders 3. Specific Instructions for Technical & Financial parts of Bids 4. Evaluation Criteria (within the bidding docs!)

5. Draft Contract

Structure of TORs • Preamble • Brief Project Description • Scope of Work • Key Personnel • Obligations of Successful Bidder • Time Frame • Deliverables • Payment Modalities

THANKS piftikhar@hotmail.com

Backup Slides

Agenda Part-1 • Session 1- Setting the stage • Session 2- Monitoring • Session 3- Evaluation

TORs for the Evaluation Team/Firm • Evaluation Background and Context • Evaluation Purpose • Evaluation Scope and Objective – What aspects of the intervention are to be covered by evaluation? (time frame, geographic area & target groups to be considered, projects (outputs) are to be included).

– What are primary issues of concern that evaluation needs to address ?

• Evaluation Questions • Evaluation Methodology …continued

TORs for the Evaluation Team/Firm • Evaluation Products – Inception report – Draft Evaluation report – Final Evaluation Report

• Evaluation Team Composition & Competencies • Evaluation Implementation Arrangement • Evaluation Time Frame for the Process • Evaluation Cost – Projected Budget

Planning

Goal Setting

Developing Strategy

Resource Allocation

Outlining Implementation

Sourcing Options

Steps for M&E implementation in the 3 scenarios Scenario 1

Scenario 2

Scenario 3

Recommendations for activating M&E Function

Implement M&E Implement M&E in-house, but: Outsource M&E in-house • Acquire deficient HR • Engage a firm • Execute for provision of • Develop M&E team M&E Plan M&E services • Develop capacity of team thru professional training • Procure necessary HW & SW • Develop M&E Plan • Execute M&E Plan • Outsource functions which are not feasible or are occasional

Possible reasons of selecting one over the other In-House

Outsourced •

•

Cost - may be less expensive in long run

•

Required expertise is available

•

•

Want direct control over the process

•

•

Desire of better quality

•

•

•

Cost - presently less expensive to outsource (eg: there isn’t lot of work) Lack of in-house capacity very big Outside Service Providers’ research & specialized know-how exceeds External validation is required (like third party audit validation)

Flexibility in managing system configuration

•

Reliable & competent service providers not available

•

To keep the in-house manpower small

•

May be a Legal requirement

Routine nature of task

•

Notion that third-party M&E is considered more “neutral” than in-house.

Structure of a typical M&E Unit

Material Resources Required for In-house M&E • • • • • • •

Hardware Software Drive test equipments Furniture Communication Equipment Vehicles Office

Selection Process to outsource M&E

Part2, Session 6

WORKED EXAMPLE FOR EVALUATION OF BIDS

Worked example - Evaluation of Bids Items

Assigned weights

Evaluated Points Rating

1- Qualifications of Bidder (150) a- Experience in similar projects

50

100%

50

b- Experience in similar geographic areas

50

90%

45

c- Prior experience of working with USPF

50

0%

0

a- Understanding of objectives

100

90%

90

b- Quality of methodology

100

100%

100

c- Innovation in proposed approach

50

90%

45

d- Work plan/ time lines

50

80%

40

e- Presentation of the Bid

50

90%

45

f- Partnership with a local firm

50

70%

35

2- Approach and methodology (400)

(‌cont)Worked

example - Evaluation of Bids Assigned weights

Evaluated Rating

Points

a- Team leader

100

80%

80

b- Monitoring specialist

50

70%

35

c- Evaluation specialist

50

90%

45

d- Survey specialist

50

100%

50

e- Sociologist/Economist

50

80%

40

f- Reporting Specialist

50

90%

45

g- Other Staff

100

90%

90

Items 3- Personnel (450)

Total Evaluated Points

835