MONITORING & EVALUATION for UNIVERSAL SERVICE PROGRAMS/PROJECTS
Introduction • Personal – Former CEO USF Pakistan – First employee - built USF Pakistan and ran it successfully for nearly 5 years – Now a consultant in several countries of Asia and Africa
• This Webinar – NOT about M&E itself – Rather about M&E in USOF Projects & Programs! – After an introduction to the concept of M&E, “setting the stage”: how to set up M&E in a Universal Service Organisation
Agenda • Part-1 – Session 1- Setting the stage – Session 2- Monitoring – Session 3- Evaluation
• Part-2 – – – – – –
Session 1- Organizational Capacity Assessment Session 2- Sourcing Options Session 3- In-House M&E Session 4- Outsourced M&E Session 5- Procurement – if outsourced Session 6- Worked Example for Bid Evaluation
Part 1, Session 1
SETTING THE STAGE
Project life
Goals
Objectives
Inputs
Outputs
Focus of Monitoring
Outcomes
Impacts
Focus of Evaluation
Definitions INDICATOR - a metric for measurement of inputs, processes, outputs, outcomes, & impacts for projects, programs, or strategies. INPUTS - the main resources required to undertake activities & to produce outputs.
OUTPUTS - physical &/or tangible goods / services delivered by the project. Describes scope of the project OUTCOME - what the project intends to accomplish by the end of it’s implementation. IMPACT - result of the project, accrued in medium to long term.
Part1, Session 2
MONITORING
Monitoring is an ongoing process by which stakeholders obtain regular feedback on the progress being made towards achievement of their planned objectives.
PLAN
Planning Phase
PROGRESS
Implementation Phase
Monitoring should be able to respond to 2 questions
Are we taking the actions we said we would take? Are we making progress on achieving the results that we said we wanted to achieve?
…all this is measured thru Indicators.
Org level
Project level
Examples of Indicators – Telephone lines per 100 persons – Computer usage rate in intervened area – Mobile subscribers per 100 persons (segregated geographically) – Internet availability per 100 persons (based on technology) – Volume of local content developed – Funds spent in each region divided politically, geographically, etc – No of contracts awarded – Subsidy amount disbursed – % of operational expenditures against contribution collected
What to Monitor? Quality Contracts Schedules Scope Risk
Human Resources Other factors
Who Monitors?
Monitoring Tools
• Periodic Reports • Computer based project management tools.
Project
• Network performance reports e.g: network usage report, billing info, QoS report, call success ratios, etc. • GIS Maps • Tech. Parameters validation through drive test & other tech. metrics
Basic Design of Monitoring Framework
What is to be measured – indicators?
How it will be measured & recorded?
Who will measure the information?
How frequently info will be measured & reported?
Part 1, Session 3
EVALUATION
What is Evaluation? EVALUATION - a thorough & independent assessment of activities to determine the extent to which they are achieving stated objectives. It is a periodic, objective assessment of an ongoing or completed project, program, or policy
Parameters of Evaluation
It relates to specific
It is carried out at discrete
questions regarding
points in time, often
design, implementation
seeking perspective of
and results.
outside experts.
Types of Evaluation – broadly Outcome/Impact Evaluation • investigates nature of relationship between planned inputs & outputs, & their outcomes • assesses, what difference (impact) the intervention made • and how much of the impact is attributable to the Program/Project.
Program Performance Evaluation addresses issues like • relevance to overall goal of org. • efficiency in delivery of inputs, • effectiveness of approach, • timeliness of approach • other questions regarding project/ program design, management & operational decision making
When Evaluations are conducted?
Program / Project
Start
Baseline Evaluation
?
Interim Evaluations
End
Later
Post Evaluation
for long-term Impact Analysis
When is the Post Evaluation conducted? Whenever sponsors want to address question like: – In case of a Pilot project: What would be potential development contributions of an innovative new program (before it is started)? – In case of a successful small-scale Program: To what extent & under what circumstances could it be replicated on a larger scale or with different population groups? – In case of a completed Program: What has been the contribution of the intervention (made thru the Program)?
How to conduct Impact Evaluation? Evalua8on E 0 Baseline
-‐
Evalua8on E C Post Evalua3on
Evalua8on could cover
Project Beneficiary
• Demographic • Socio -‐ Economic • Technology • Usage pa@ern • Other relevant indicators
Impact= Ec – Eo Approach -‐ 1 Also called “Before & After method” Start Time
T0
Close Time T C
How to conduct Impact Evaluation?
Evalua8on EEwithout without Evalua8on Evalua8on Ewith
Project Beneficiary
• Demographic • Socio -‐ Economic • Technology • Usage pa@ern • Other relevant indicators
Impact= Ewith -‐ Ewithout
Evalua8on could cover
Approach -‐ 2
Project Non-‐Beneficiary
-‐
Also called “With & Without method” Start Time
T0
Close Time
TC
Evaluation Tools
• Surveys Project
• Beneficiary Assessment • Econometric Analysis- Regression • Case Studies • Cost Benefit Analysis • Cost Effectiveness Analysis
Info. required for Impact Evaluation (example Telecenter) General information about area Mapping of public & private facilities (schools, hospitals, post offices, ‌) Demographic Info. (No. of persons - age group & gender segregated) Number of persons who can access computers (gender-wise) Cost per use of computer, PCO, & other services Persons using computer for educational purpose, health, others Persons using computer for web based earning, Average distance & time from nearest computer center Other uses of computers & ICTs Number of computers per 100 population Number of persons who can access computer and internet Number of persons who know how to use computer
Info. required for Impact Evaluation (example Telecenters) Information on Purpose of using computers Info on Internet & Computers for learning Info on status of Info on status of Info on status of Info on status of
E-commerce E-Health E-governance E-Agriculture
Info on use of computer in social networking Info on economic activities associated with internet & computer Info on status of networking Info on presence of local content in cyber space Info on financial sustainability Info on community participation in management of center
PART-2
Part 2, Session 1
ORGANIZATIONAL CAPACITY ASSESSMENT TEST – One of the ways!
Comparison of 3 scenarios for M&E implementation Scenario 1
Scenario 2
Scenario 3
Sufficient capacity exists within the org.
Capacity exists but not sufficient
Substantial Capacity gaps
• Independent M&E unit • • • • •
• Capacity to execute M&E but with gaps Capacity to execute M&E • M&E plan not M&E plan is available developed Necessary staff available • M&E team deficient Necessary hardware & in terms of HR software are available • Deficiencies in Office-space is available hardware & software
• No M&E set up of any kind • No dedicated human resource • No infrastructure
OCAT*- Sample Test Question-1
Score
What is the organizational structure in context of M&E? Response A separate M&E Unit with clear reporting lines existing, indicate a strong M&E setup viable for in-house M&E A dedicated M&E human resource with no clear reporting line implies a weak M&E system within an organization No M&E system within an organization structure
* Organizational Capacity Assessment Test
Scores 2 1 0
(‌ cont.) OCATQuestion-2
Sample Test Scores
What is the current M&E setup - human, financial, and technical resources? 2.1 Human resources 2.2 Budget (financial allocation) for M&E 2.3 Equipment available for M&E
Response Sufficient Needs Improvement Non-existent
Scores 2 1 0
(….. Cont.) OCAT-
Sample Test
Question-3
Score
What is the rationale for creating M&E within the organization (Management’s perspective)? Response Clearly defined objectives, and policy exists regarding M&E Management aware about M&E’s importance but no written policy or objectives exist No information available
Scores 2 1 0
(‌.. Cont.) OCAT-
Sample Test
Question-4
Scores
What skills exist within the organization in the following areas; 4.1 Domain Knowledge of Monitoring & Evaluation 4.2 Experience of working and demonstrated capability of managing M&E 4.3 Designing and conducting research/surveys 4.4 Data collection, management, and reporting 4.5 Statistical Analysis 4.6 Knowledge about econometrics 4.7 Using research findings for decision making through recommendations 4.8 Project management 4.9 Development of computer based M&E system Response Sufficient Needs Improvement Non-existent
Scores 2 1 0
(….. cont)
OCAT- Sample Test
• Scoring Recommendations – Maximum attainable score = 28 – Score above 90% indicates sufficient capacity for In-house M&E – Score between 60-89% indicates capacity, which need to be built – Score less than 60% indicates weak capacity, where Outsourced M&E is the most plausible option
Part 2, Session 2
SOURCING OPTIONS
M&E Sourcing Options
OCAT Results
Sufficient Cap.
In-house
Insufficient Cap.
In-house with Cap. Building
Cap. Gaps
Out-source
Steps for M&E implementation in the 3 scenarios Scenario 1
Scenario 2
Scenario 3
Implement M&E inhouse if not already being done
Implement M&E in-house, but: • Acquire deficient HR
Outsource M&E by engaging a professional firm
• Develop M&E team • Develop capacity of team thru professional training • Procure necessary HW & SW • Develop M&E Plan • Execute M&E Plan • Outsource functions which are not feasible or are occasional
How to decide?
Build In-House
Outsource
Possible reasons of selecting one over the other In-house
Out-sourced
• Cost - may be less expensive in long run
• Cost - less expensive to outsource (eg: there isn’t lot of work)
• Required expertise is available
• Lack of in-house capacity too big
• Want direct control over the process
• Outside Providers’ research & specialized know-how exceeds by far
• Desire of better quality • Flexibility in managing system configuration
• External validation is required
• Reliable & competent service providers not available
• A legal requirement?
• Routine nature of task
• To keep in-house manpower small • Notion that third-party M&E is considered more “neutral”
Part 2, Session 3
IN-HOUSE M&E
Structure of a good M&E Unit - Functions
Manager
Survey
Enumerators
Monitoring
Engineers
Field Coordinators
Evaluation
Database Coordinators
Data Analyst
Reporting
Sociologist/ Economist
HR Requirements for In-house M&E (pg 1 of 2) S. No Designation
Experience
Qualification
10+ years in Project Management + Masters Level Degree — Monitoring & Evaluation Management, Social Sciences, & Engineering of similar projects in development sector
No.
1
Manager
2
Monitoring 5+ years in Monitoring Specialist
Masters Level Degree — Management, and Social Sciences
1
3
E v a l u a t i o n 5+ years of conducting Specialist Evaluation
Masters Level Degree — Social Sciences
1
4
S u r v e y 5 + years in conducting Specialist surveys
Masters Level Degree — Management, and Social Sciences
1
R e p o r t i n g (5+ years in Specialist Report-writing
Masters Level Degree — Social Sciences (economics/ sociology or other relevant disciplines)
1
5
1
Pg 2 of 2) HR
Requirements for In-house M&E
S. No Designation Experience
Minimum qualification
6
5+ years of in capturing S o c i o l o g i s t / socio-economic aspects Economist of development interventions
Masters Level Degree — Social Sciences (economics/ sociology or other relevant disciplines)
7
Engineers
3+ years
Bachelors Level — Engineering Telecom
8
Data Analyst
3+ years in conducting data analysis & interpretation of data
Masters Level Degree -Management or Statistics
9
Masters Level Degree — D a t a b a s e 5+ years in developing & Computer Sciences, Database Coordinator managing databases Management
10
F i e l d 3+ years in field coordinators coordination
11
Enumerators
3+ years in enumerating surveys
Nos. 1
Workload dependent 1
1
Bachelors Level Degree -Sociology + other relevant disciplines
Workload dependent
Bachelors Level Degree -Sociology + other relevant disciplines
Workload dependent
Material Requirements for In-house M&E S.No Item 1
Hardware
2
Software
3
Drive Test Equipment
4
Communication Equipment
5
Furniture
6
Vehicles
7
Office
8
Misc.
Quantities
Part 2, Session 4
OUTSOURCED M&E
Selection Process to outsource M&E
Receipt Invitation of applications to prequalify from Firms
Shortlist Evaluation of the preof applications qualified
RfP to the shortlisted
Selection Evaluation of the Outof proposals sourcing Firm!
Task List for the Consultant/Firm • Develop a detailed configuration of M&E unit incl: identification of HR, job descriptions, competency levels & organization structure • Develop M&E Plan to address: What will be done? How it will be done? Who will do it? When it will be done? How much resources are required? • Build capacity of the staff for M&E; • Develop computer based M&E reporting system, along with user manuals & documentation; • Develop M&E Operational Manual for the staff; • Install hardware & software for M&E & train operators; • Train __ persons to conduct/supervise data-collection process; • Design & conduct baseline surveys in project areas; • Develop impact assessment methodology and & conduct impact assessment;
(.. Cont)
Task List for the Consultant/Firm
• Identify problems with M&E system & come up with solutions to those problems; • Develop transition plan to support USF in taking over M&E • Provide refresher training on M&E to USF staff; • Oversee design & conduct of various M&E activities by USF; • Review the results of studies undertaken & propose improvements; • Assist in data analysis & interpretation; • Review the system & propose amendments if required; • Provide technical support for maintaining computer based M&E system. • Ensure that staff is receiving adequate support to implement M&E functions & that data collection & analysis is on schedule
If out-sourced, some HR would still be required
At least with 5 basic skills: 1) Project Management 2) Contract Management 3) Financial Controlling 4) Technical Monitoring 5) Socio-Economic Monitoring
Min HR required to monitor Out-sourced M&E S. No Designation Experience
Min. qualification
Nos.
1.
M a n a g e r 5+ years in project M&E management
Masters level — Management, Social Sciences, or Engineering
1
2.
3+ years in capturing socioSociologist/ economic aspects Economist of development interventions
Masters level — Social Sciences (economics/sociology or other similar disciplines)
1
Engineer
Bachelors level — Engineering Electronics / Telecom
Workload dependent
3.
3+ years in field
Part 2, Session 5
PROCUREMENT DOCUMENTATION
List of Important Documents for Pre-qualifications
1. Invitation to Prequalify 2. Instructions to Applicants 3. Prequalification Datasheet 4. Prequalification Evaluation Criteria 5. Prequalification Application Form.
Major parts of RfP for M&E
1. Introduction 2. General Instructions to Bidders 3. Specific Instructions for Technical & Financial parts of Bids 4. Evaluation Criteria (within the bidding docs!) 5. Draft Contract
Structure of TORs • Preamble • Brief Project Description • Scope of Work • Key Personnel • Obligations of Successful Bidder • Time Frame • Deliverables • Payment Modalities
THANKS piftikhar@hotmail.com
Backup Slides
Agenda Part-1 • Session 1- Setting the stage • Session 2- Monitoring • Session 3- Evaluation
TORs for the Evaluation Team/Firm • Evaluation Background and Context • Evaluation Purpose • Evaluation Scope and Objective – What aspects of the intervention are to be covered by evaluation? (time frame, geographic area & target groups to be considered, projects (outputs) are to be included). – What are primary issues of concern that evaluation needs to address ?
• Evaluation Questions • Evaluation Methodology …continued
TORs for the Evaluation Team/Firm • Evaluation Products – Inception report – Draft Evaluation report – Final Evaluation Report
• Evaluation Team Composition & Competencies • Evaluation Implementation Arrangement • Evaluation Time Frame for the Process • Evaluation Cost – Projected Budget
Planning
Goal Setting
Developing Strategy
Resource Allocation
Outlining Implementation
Sourcing Options
Steps for M&E implementation in the 3 scenarios Scenario 1
Scenario 2
Scenario 3
Recommendations for activating M&E Function
Implement M&E Implement M&E in-house, but: Outsource M&E in-house • Acquire deficient HR • Engage a firm • Execute for provision of • Develop M&E team M&E Plan M&E services • Develop capacity of team thru professional training • Procure necessary HW & SW • Develop M&E Plan • Execute M&E Plan • Outsource functions which are not feasible or are occasional
Possible reasons of selecting one over the other In-House
Outsourced •
•
Cost - may be less expensive in long run
•
Required expertise is available
•
•
Want direct control over the process
•
•
Desire of better quality
•
•
•
Cost - presently less expensive to outsource (eg: there isn’t lot of work) Lack of in-house capacity very big Outside Service Providers’ research & specialized know-how exceeds External validation is required (like third party audit validation)
Flexibility in managing system configuration
•
Reliable & competent service providers not available
•
To keep the in-house manpower small
•
May be a Legal requirement
Routine nature of task
•
Notion that third-party M&E is considered more “neutral” than in-house.
Structure of a typical M&E Unit
Material Resources Required for In-house M&E • • • • • • •
Hardware Software Drive test equipments Furniture Communication Equipment Vehicles Office
Selection Process to outsource M&E
Part2, Session 6
WORKED EXAMPLE FOR EVALUATION OF BIDS
Worked example - Evaluation of Bids Items
Assigned weights
Evaluated Points Rating
1- Qualifications of Bidder (150) a- Experience in similar projects
50
100%
50
b- Experience in similar geographic areas
50
90%
45
c- Prior experience of working with USPF
50
0%
0
a- Understanding of objectives
100
90%
90
b- Quality of methodology
100
100%
100
c- Innovation in proposed approach
50
90%
45
d- Work plan/ time lines
50
80%
40
e- Presentation of the Bid
50
90%
45
f- Partnership with a local firm
50
70%
35
2- Approach and methodology (400)
(‌cont)Worked
example - Evaluation of Bids Assigned weights
Evaluated Rating
Points
a- Team leader
100
80%
80
b- Monitoring specialist
50
70%
35
c- Evaluation specialist
50
90%
45
d- Survey specialist
50
100%
50
e- Sociologist/Economist
50
80%
40
f- Reporting Specialist
50
90%
45
g- Other Staff
100
90%
90
Items 3- Personnel (450)
Total Evaluated Points
835