Page 1

Internal briefing note for HLE consultants

From one High Level Evaluation team to another

Lessons and tips for conducting a HLE evaluation at the ILO

November 2016


Internal briefing note for HLE consultants

The purpose of this document is to share lessons learned and tips on what worked well in conducting a highlevel evaluation (HLE) at the ILO. This documents includes specific pointers categorised by phases of the evaluation process.

Scoping and initial desk review To get acquainted with the functioning of the ILO, its results-based management system, and its reporting system, we strongly advise the consultant to familiarize with the following documents: 1. Results-based Management in the ILO This document provides an overview of the Results-based Management (RBM) system at the ILO. It explains how outcomes, strategies, indicators, targets, and baselines are defined. Moreover, it discusses work planning, and performance monitoring and evaluation within the RMB framework. For more information about how RBM is applied to ILO’s programming framework, see Annex 1. 2. Relevant Strategic Policy Framework (SPF) or Strategic Plan (SP)1 This is ILO’s medium-term planning document. It expresses the strategic orientation of the ILO, providing global outcomes, indicators, and measurements criteria. The evaluation team should look at the overall targets set in the relevant SP and compare them with the relevant Implementation Reports to determine whether the ILO has reached its targets. Find here the SPF 2010-15, the transitional SP 2016-2017, and the SP 2018-21. 3. Programme & Budget (P&B) These documents detail the vision set out in the SPF/SP, specifying strategies and resources needed to achieve results over a 2-year time frame. The targets set in the P&B should match the relevant strategic planning document (SPF/SP). The evaluation team should take a look at whether the P&B is coherent and in line with the relevant strategic planning document. Find here the P&B 2010-2011, the P&B 2012-2013, the P&B 2014-2015, and the P&B 2016-2017. 4. Programme Implementation Reports These documents offer an overview of ILO's performance and programme implementation for a biennium, providing reporting and analysis by strategic objective. These documents detail information about the fate of the CPOs, specifying which ones were measured by the ILO as a “result” to be reported on. The programme implementation report drafted in the last year of an SPF/SP (e.g. 2015) contains a summary of the achievements for the entire SPF/SP period. The evaluation team should examine that summary to validate what ILO has reported. For more information on how results are reported, see the Guidance for the Regions (2014-15) or Annex 4. Find here the Implementation reports for 2010-2011, 2012-2013, and 2014-2015. 5. Decent Work Country Programme (DWCP) The Decent Work Country Programmes are the expression of the ILO’s P&B locally: they organize the work of the Organization in each country around a limited number of country programme priorities and outcomes, which are further detailed in implementation plans. Priorities and outcomes in the DWCPs are defined by constituents in close consultation with the ILO. Country Programme Outcomes (CPOs) act as the building blocks of DWCPs. CPOs are practical manifestations of the ILO’s intentions in a given country and serve as a link between a consented results framework (i.e the DWCP) and the P&B outcomes. In the current system, a CPO can be linked to a single P&B outcome—the primary outcome to which the CPO contributes. However, in practice, a single CPO may contribute to more

1

This document used to be called Strategic Policy Framework (SFP) until 2015 and is now called Strategic Plan (SP).

November 2016


Internal briefing note for HLE consultants

than one P&B outcome. Find here the DWCP for each country. For more information on DWCPs see the ILO Decent Work Country Programme: a practical guidebook or Annex 2. 6. Outcome-Based Workplans (OBW) Outcome-Based Workplans are high-level workplans composed by outcome strategy, country programme outcomes (CPOs), and global products. OBWs set out how the Office will implement the strategy for each Outcome and how the different resources will be allocated accordingly. They provide a basis for the different ILO units (HQ, field offices and Turin) to plan their contribution to the achievement of prioritized CPOs and the delivery of the Global Products. For a diagram showing the relationship between SPF/SP, P&B, OBWs, Implementation Reports and DWCP see Annex 3. After familiarizing with the above documents, the consultant is strongly encouraged to come to Geneva for a week to meet the EVAL team and relevant stakeholders, and to conduct scoping in person. Relevant stakeholders include staff members in the department concerned by the evaluation, as well as representative of the PROGRAM2 and PARDEV3 departments, and constituents. While in Geneva, the evaluation team can hold a group ÂŤ kick off Âť meeting to meet the stakeholders, listen to their expectations, and gather their inputs. From a practical perspective, such meetings help with understanding the breadth of ILO work on any given subject. The added value of in person meetings is that involving stakeholder from the beginning enhances stakeholder buy-in. Additionally, these meetings can help the evaluation team understand what relevant information is available regarding the concerned department. At the ILO the type of information collected and the format in which it is stored can vary across departments. For instance, some departments may have detailed information on their expenditures on technical support to regions, while some other departments may have it in less detail. Even at the same level of detail, information can be stored in different types of reports and/or on different platforms (KSP, departmental i-drive, etc.) To facilitate the gathering of relevant information, EVAL will ask the concerned department to provide the information below, if available: 1. 2. 3. 4.

Records of technical support, disaggregated by intervention type (mission reports, etc.) List of publications List of global products and relevant related documents Expenditure on support provided to constituents (mission reports) disaggregated by country and intervention type 5. List of good practices forums and seminars organized by the department 6. Key events regarding/involving the department (for EVAL/the consultant to attend and interact with stakeholders) 7. List of relevant partners (i.e. interagency working groups)

Evaluation design and Inception report During the design phase of the report, please consider the different contributions and tools that can be provided by EVAL. In previous HLEs, EVAL provided support for desk reviews of decent work country programmes to quantify the demand for interventions related to the theme under evaluation. Another 2

The Strategic Programming and Management Department (PROGRAM) advises on programme planning, resource allocation and implementation reporting. Some key PROGRAM tasks include: preparing the biennial Programme and Budget, Implementation Reports and the Strategic Plan; designing systems and programming tools, analysis and guidance as well as capacity building and training of staff; supporting and advising on the implementation of improvements in internal structures and management systems, including through integrated functional support of IRIS. 3 The Partnerships and Field Support Department (PARDEV) is responsible for resource mobilization, the management of development cooperation activities and public-private partnerships. PARDEV supports ILO field offices and manages day-to-day relations with other international and regional organizations.

November 2016


Internal briefing note for HLE consultants

contribution that can be offered by EVAL is the design and administration of surveys (using EVAL’s official account for Survey Monkey). In the past, surveys have been sent to relevant ILO staff, constituents, and ILO partners. If relevant, make sure the survey reflects the areas in which the team will have to make the final ratings. For instance, be sure to ask clear questions on stakeholder perceptions on each one of the DAC criteria in a clean manner, so that the responses to those questions can be used to serve as input to the ratings, if needed. When choosing which instruments will be employed during the evaluation, we strongly encourage the evaluation team to define ex ante how the different criteria (relevance, coherence, effectiveness, efficiency, impact, and sustainability) will be measured. In particular, it is best to define early how the tools used (e.g. surveys, financial analyses, direct observations) will be factored into the rating of each criteria. For instance, for the HLE on Skills the evaluation team used the perceptions of ILO staff and constituents as two input into the ratings and the rating provided by each evaluation team member (based on field visits, case studies, and interviews) as additional inputs. All four inputs were expressed in a scale from 1 to 64. Finally, for each of the evaluation criteria, a simple average of all inputs was taken and used as overall rating.

Field visits When planning the field visits, the evaluation team should be vocal about who to meet. It is key for the team to meet with the ILO constituents and relevant ILO staff, but also consider meeting with the UN partners, and donors in addition to the stakeholders and beneficiaries. During field visits, try to visit a project location and speak to beneficiaries if possible. EVAL can facilitate in the process of identifying the main stakeholders (country and regional office directors and staff, the concerned specialists in the office, TC projects staff, if relevant.) Further identification is done upon consultation with EVAL, the evaluation team, and the ILO country focal point.

Draft report Once the first draft of the report has been completed, the consultant is strongly encouraged to come to Geneva for one week to present the draft report to relevant stakeholders and receive first-hand feedback. Ideally, set the meeting for the first available working day of that week, and take the rest of the week for any follow-up meeting with stakeholder and for revising the draft report.

At the end of the process If possible, at the end of the week in Geneva, take some time to debrief on the evaluation process with the entire team. Reflecting on this document, brainstorm on what additional tips and lessons learned can be passed on to future evaluation teams.

Timeframes As was indicated in the TOR for this exercise, the HLE process has a specific timeframe as the GB summary document should be transmitted to the Governing Body for its November session. In the ILO this means that the summary document must be handed over to the official ILO document preparation process on August 31st at the latest. This is not a flexible deadline so the team should ensure that milestones are set to ensure that the final executive summary is ready for 31 August.

4

1 being «Highly Unsatisfactory» and 6 «Highly Satisfactory»

November 2016


Internal briefing note for HLE consultants

Annexes Annex 1: Snapshot of the ILO’s Planning Framework At the global level, ILO outcomes are developed and approved by the Organization and are set out in the Strategic Policy Framework (SPF) or Strategic Plan (PS) and in the programme and budget (P&B) documents. The SPF adopts Results-Based Management in identifying the results expected for the planning period, within a framework of outcomes, indicators, and targets. The P&B further specifies the results for the biennium, the strategies the ILO will implement to achieve them, and the resources available to do so. Targets set in P&B documents define the level of achievement intended toward the desired result within a specific timeframe. To support the effective and efficient achievement of results under each P&B Outcome, the ILO has adopted biennial planning in the form of Outcome-Based Workplans (OBWs). OBWs provide the basis for more detailed workplans to be developed by headquarters and field units. OBWs are developed and implemented using existing IRIS functionality, in particular the Strategic Management Module/ Implementation Planning (SM/IP). Each CPO in SM should be classified as either target, pipeline, maintenance or achieved. Each CPO has a description; milestones and major outputs; resources required; links to the relevant P&B outcome indicator; and linked resources. CPOs proposed by countries in a given biennium may or may not be linked to resources. Overall, CPOs are categorised as: 1. Target: CPOs identified for completion during the ongoing biennium(resources can be linked) 2. Pipeline: CPOs on which work will be done during the ongoing biennium but full achievement will not be reached (resources can be linked) 3. Maintenance: CPOs that are important but for which the time is not yet ripe to expect results (no resources linked) 4. Achieved: CPO that is considered completed and reported as such in the Programme Implementation Report.

Annex 2: What is a DWCP? In any given country, Decent Work Country Programmes (DWCPs) are defined by ILO constituents– i.e. the Ministry of Labour and workers’ and employers’ organizations – with support from the ILO Country Office. DWCPs transform country priorities into achievable and measurable outcomes in line with the ILO’s SPF/SP. The following six steps are undertaken to draft a DWCP. The three ILO constituents should be involved throughout these steps: 1. Defining the country context 2. Establishing country programme priorities 3. Defining intended outcomes, their measurement and strategy 4. Planning for implementation 5. Implementation, monitoring and reporting 6. Evaluation and country programme reviews DWCPs usually take about four to six years for successful implementation. They may mobilize significant resources, both locally and from outside sources. Development cooperation projects also support the achievement of country outcomes.

November 2016


Internal briefing note for HLE consultants

Annex 3: Relationship between DWCPs and the ILO’s programming framework5

Annex 4: Reporting results in Implementation Reports6 For each Country Programme Outcome (CPO), reporting information will have to: 1. Include only results that meet the measurement requirements – The focus is on results (outcomes) consistent with the measurement requirements of the P&B indicator to which the CPO is linked. Information that does not meet these requirements shall not be included in the table. 2. Identify clearly the Office contribution to the related result – Identify outputs delivered by the Office to support constituents in achieving the reported results. All reported results should be clearly attributable to ILO contributions. References to outputs supported through specific donors’ extrabudgetary funding and RBSA are encouraged. 3. Refer to new results – CPO results should be new and additional to what has been reported as a result in the previous two biennia. If reported previously, a CPO should not be reported again against the same measurement criteria unless the result is substantively different. It is acceptable to report a result against an outcome indicator in more than one biennium if different measurement criteria are applied.

5 6

Based on ILO Decent Work Country Programme: a practical guidebook, p. 4 Based on Report on ILO Programme Implementation 2014-15 Guidance for the regions p. 3

November 2016

Briefing note for hle consultants for eval comments  

Draft in progress

Read more
Read more
Similar to
Popular now
Just for you