Organisation and Methodology_IPA CBC

Organisation and Methodology for the on-going programme evaluation of the Hungary - Croatia IPA Cross-border Cooperation Programme 2007-2013 and the Hungary - Serbia IPA Cross-border Cooperation Programme 2007-2013

Budapest 21 December 2012 st

Name: HitesyBartuczHollai Eurocomsulting Kft. Address: H-1054 Budapest, Vértanúk tere 1.

Tel: +36-1-319-1790 Fax: +36-1-319-1381 E-mail: info@hbhe.hu Website: www.hbhe.eu

Name: ICG Ex-Ante Kft. Address: H-1023 Budapest, Bécsi út 3-5. V. emelet 56.

Tel: +36-1-321-1773 Fax: +36-1-321-1772 E-mail: info@icg-exante.hu Website: www.icg-exante.hu

Name: European Association for Information on Local Development (AEIDL) Address: Chaussée Saint-Pierre, 260 – B-1040 Bruxelles Tel: +32 2 736 49 60 Fax: +32 2 736 04 34 E-mail: info@aeidl.eu Website: www.aeidl.eu

2

Organisation and Methodology for the On-going Programme Evaluation of the Hungary - Croatia IPA Cross-border Cooperation Programme 2007-2013 and the Hungary - Serbia IPA Cross-border Cooperation Programme 2007-2013

Table of contents 1.

Rationale ................................................................................................................................................................... 5 1.1.

Understanding the assignment ................................................................................................................... 5

1.1.1.

Objective of the Contract ......................................................................................................................... 5

1.1.2.

Key issues affecting successful implementation ...................................................................................... 5

1.1.3.

Current status and perspectives ............................................................................................................... 7

Hungary-Croatia IPA Cross-border Co-operation Programme ................................................................................ 7 Hungary-Serbia IPA Cross-border Co-operation Programme ................................................................................. 8 1.1.4.

1.2.

2.

Summary statements ............................................................................................................................... 9 Risks and assumptions ................................................................................................................................ 9

Strategy ................................................................................................................................................................... 11 2.1.

Our approach ............................................................................................................................................ 11

2.1.1.

Underlying principles of our work .......................................................................................................... 11

An evaluation meeting the highest quality standards .......................................................................................... 11 An evaluation that is useful .................................................................................................................................. 11 An evaluation with involvement and empowerment ........................................................................................... 11 2.1.2.

The pillars of our methodology .............................................................................................................. 12

We design a toolkit which fits the evaluation topic and the programme specificities ......................................... 12 We identify feasible recommendations and support their implementation ........................................................ 12 We design schemes allowing for the parallel evaluation of the two Programmes .............................................. 12 2.1.3.

The evaluation toolkit ............................................................................................................................. 13

In-depth interview ................................................................................................................................................ 14 Focus group interview .......................................................................................................................................... 14 Co-creation workshops (CCWS) ............................................................................................................................ 14 Questionnaire – online interview ......................................................................................................................... 14 Document analysis ................................................................................................................................................ 14 Evaluation statistical analyses (EVALSTAT) ........................................................................................................... 15 Indicator analyses ................................................................................................................................................. 15 Sampling ............................................................................................................................................................... 15 Process modelling using value engineering techniques ....................................................................................... 16 Logical Framework Method (LFM) ........................................................................................................................ 16 Case Study............................................................................................................................................................. 16 2.2.

Activities and related inputs and outputs ................................................................................................. 16 Evaluation Plan ..................................................................................................................................................... 16

2.2.1.

2.2.1.1.

Inception Phase ...................................................................................................................................... 18 Inception report ................................................................................................................................. 18 3

Organisation and Methodology for the On-going Programme Evaluation of the Hungary - Croatia IPA Cross-border Cooperation Programme 2007-2013 and the Hungary - Serbia IPA Cross-border Cooperation Programme 2007-2013

2.2.2.

1st Evaluation Phase ............................................................................................................................... 18

2.2.2.1.

Topic 1: Direct comparison of the Programme objectives and results so far .................................... 18

2.2.2.2.

Topic 2: Assessment of the project selection procedure and criteria ................................................ 19

2.2.2.3.

Topic 3: Evaluation of Programme performance by indicators ......................................................... 20

2.2.2.4.

Topic 4: Revision of the Programme Communication activities based on the Communication Plan 21

2.2.2.5.

First Evaluation Report: ...................................................................................................................... 22

2.2.3.

2nd Evaluation Phase ............................................................................................................................. 22

2.2.3.1.

Topic 5: Contribution to European Policies: Analysis of the Programme contribution to key

European and macro-regional policies ................................................................................................................. 22 2.2.3.2.

Topic 6: Future: analysis and evaluation of Programme deliveries and results in order to provide

recommendations to the Participating Countries to the next programming period (2014- 2020) ...................... 23 2.2.3.3.

Topic 7: Case Studies and Best Practices ........................................................................................... 24

2.2.3.4.

Topic 8: Benefits of the Programme in terms of preparing the non-member states (Croatia, Serbia)

for the absorption of Structural Funds at the time of accession to the European Union .................................... 24 2.2.3.5.

2.2.4.

2.3.

Second Evaluation Report .................................................................................................................. 25 Final phase .............................................................................................................................................. 26

Presentation of Consortium and division of tasks .................................................................................... 27

2.3.1

General approach ................................................................................................................................... 27

2.3.2

The consortium....................................................................................................................................... 27

HitesyBartuczHollai Euroconsulting (Hungary) ..................................................................................................... 27 ICG Ex-Ante (Hungary) .......................................................................................................................................... 28 European Association for Information on Local Development AEIDL (Belgium) .................................................. 29 Sub-contractors .................................................................................................................................................... 30 2.3.3

Division of tasks and responsibilities between the members of the consortium .................................. 31

2.3.4

The team................................................................................................................................................. 33

Team Leader ......................................................................................................................................................... 33 Key Experts ........................................................................................................................................................... 33 Short Term Experts ............................................................................................................................................... 33 2.4.

Support facilities ....................................................................................................................................... 37

2.5.

Sub-contracting arrangements foreseen .................................................................................................. 37 InTER and Karzen i Karzen..................................................................................................................................... 37

3.

4.

Timetable of Activities............................................................................................................................................. 38 3.1.

Timing of activities .................................................................................................................................... 38

3.2.

Work plan.................................................................................................................................................. 40

Log Frame ................................................................................................................................................................ 42

4

Organisation and Methodology for the On-going Programme Evaluation of the Hungary - Croatia IPA Cross-border Cooperation Programme 2007-2013 and the Hungary - Serbia IPA Cross-border Cooperation Programme 2007-2013

1.

Rationale

1.1.

Understanding the assignment

1.1.1. Objective of the Contract The objective of the call for proposals is to design and implement the overall independent assessment of the HU-HR and HU-SR Cross border Cooperation Programmes through a series of evaluation exercises. The two Programmes have to be evaluated separately in line with the ToR and the evaluation plans to be developed respectively for the two Programmes.

1.1.2. Key issues affecting successful implementation The following aspects will be taken into account when designing the strategic approach and defining the methodological tool-kit:

Adapting to new approaches and EU requirements In line with EC Regulation 1085/2006 (article 47 (1)) and EC Working Document No 5, in our understanding the present on-going evaluation is a process taking the form of a series of evaluation exercises. It aims at better understanding outputs and results achieved and progress towards longer-term impacts, as well as recommending, if necessary, remedial actions.

Besides desk research, field work will be emphasized: this approach calls for an intensive involvement of a large group of stakeholders affected by the Programme (local/regional stakeholders and the public as such). This will allow for a better reflection of beneficiaries’ views as well, besides the programme managers’ perspective.

The evaluation exercises will be carried out in a transparent and evidence-based way with factbased conclusions and a clear chain of reasoning. This way, valuable judgements will be directly linked to the specific aspects of the Programmes/projects assessed and to the evaluation criterion used. As a result, evaluation statements will be comprehensive and easy to understand.

Adapting to Specific ToR requirements in terms of content implementation The project (the ongoing evaluation activities) will be structured around four reporting phases, each building on the results of the previous one. The reporting will be carried out separately for the two Programmes.

The evaluation exercises to be carried out will strictly follow the content-related requirements formulated in the ToR, i.e. the evaluation topics. These will be broken down in the Evaluation Questions formulated in the inception phase. The evaluation activities and the consequent reports will therefore clearly refer to the specific topics listed in the ToR. 5

Organisation and Methodology for the On-going Programme Evaluation of the Hungary - Croatia IPA Cross-border Cooperation Programme 2007-2013 and the Hungary - Serbia IPA Cross-border Cooperation Programme 2007-2013

Furthermore, the evaluation methodology and tools will also address the “specific purposes� of the ongoing evaluation listed in the ToR

Adapting to the specific context of the implementation: through careful selection of experts and setting up experienced multi-language management structures the use of different languages and the cultural gap caused by the different working cultures will be addressed This particular project being the first ongoing exercise in the IPA CBC Programmes, the Consortium will support the Management bodies (MAs, NAs, JTS etc.) in the course of the implementation through continuous professional moderation and coaching as well

Setting up management structures allowing smooth and transparent implementation of the project composed of two CBC Programme areas Besides carrying out the specific tasks listed in the ToR (content development), the Consultants have a major role in providing a permanent process-management activity along the project’s implementation to ensure timely and successful achievement of the objectives set in the ToR. Taking into account the size of the project (implementing two parallel evaluation plans), the number of companies and experts involved in the delivery and the complexity of both Programme management structures, the provision of a highly performing central project-management is of key importance: We will ensure qualified human capacity, experienced in the management of sizable complex projects implemented in multi-cultural environment. Also, IT based management tools and techniques will be available providing for an effective and safe project administration Internally, clear division of tasks, transparent internal communication, clear procedural rules, structured project administration will be established to efficiently manage the large team of experts involved Externally, the professional project management unit (PMU), acting as a Secretariat the Programme Management Bodies, will operate a one-channel communication line towards the two parallel Programme administrations comprising of central level steering bodies and the national level country specific bodies presented in the below table:

6

Organisation and Methodology for the On-going Programme Evaluation of the Hungary - Croatia IPA Cross-border Cooperation Programme 2007-2013 and the Hungary - Serbia IPA Cross-border Cooperation Programme 2007-2013

1.1.3. Current status and perspectives Hungary-Croatia IPA Cross-border Co-operation Programme The European Commission approved the Hungary-Croatia IPA Cross-border Co-operation Programme 20072013 on 13 March 2008. The Programme allocated about 52 million EUR Community funding for the 7-year period. The Priorities and Areas of Interventions of the Programme are the following:

Priority 1: Sustainable Environment and Tourism 1.1

Sustainable and Attractive Environment

1.2

Sustainable Tourism in the Mura-Drava-Danube River Area

Priority 2: Co-operative Economy and Intercommunity Human Resources Development 2.1

Co-operative Economy

2.2

Intercommunity Human Resources Development

Priority 3: Technical Assistance Within the Programme three open Calls for Proposals (CfP) have been closed so far; the first CfP was launched in 2009, the second CfP in 2010 and the third one in 2011. Projects supported within the first CfP are approaching their closing stage, therefore conclusions can be drawn regarding the management of a 7

Organisation and Methodology for the On-going Programme Evaluation of the Hungary - Croatia IPA Cross-border Cooperation Programme 2007-2013 and the Hungary - Serbia IPA Cross-border Cooperation Programme 2007-2013

whole project cycle and also on the results of the implemented projects. Projects selected for funding within the second CfP started their implementation phase mostly in autumn 2011, and the implementation of those selected in the 3rd CfP has just started.

The breakdown of funding per Calls for Proposals and priorities is shown in the table below: Funding for

Funding for

Funding for

1st CfP (EUR)

2nd CfP (EUR)

3rd CfP (EUR)

1.1

5 705 449

6 383 094

1 161 576

13 250 119

28,3%

1.2

349 995

0

17 736 280

18 086 275

38,6%

2.1

4 342 285

4 125 926

1 643 773

10 111 984

21,6%

2.2

1 913 630

3 461 227

0

5 374 857

11,5%

TOTAL

12 311 359

13 970 247

20 541 629

46 823 235

100%

Priority

TOTAL Funding Percentage (EUR)

The Hungary-Croatia IPA Cross-border Co-operation Programme invested the most in the tourism development of the Mura-Drava-Danube River Area, with over 18 million EUR awarded to projects within Priority 1.2. Almost half of the available funds were awarded under Priority 1.1 Sustainable and Attractive Environment and Priority 2.1 Co-operative Economy, whereas Development of Intercommunity Human Resources has been granted more than 5,3 million EUR. Hungary-Serbia IPA Cross-border Co-operation Programme The Hungary-Serbia IPA Cross-border Co-operation Programme was adopted by the European Commission on 25 March 2008. The indicative financial allocation (IPA contribution) is about 50 million EUR for the seven-year period of 2007-2013. The Programme offers a wide range of opportunities for the potential beneficiaries on both sides of the border and concentrates on two priorities. The Priorities and Areas of Interventions of the Programme are the following:

Priority 1: Infrastructure and Environment 1.1

Infrastructure for physical connections

1.2

Common responsibility for the environment

Priority 2: Economy, Education and Culture 2.1

Stimulating a synergic economy, tourism and R&D

2.2

Education and culture for a common mind

Priority 3: Technical Assistance Within the Programme three open Calls for Proposals (CfP) have been launched, the first CfP in 2009, the second CfP in 2010 and the third CfP in 2011. Projects supported within the first CfP are mostly 8

Organisation and Methodology for the On-going Programme Evaluation of the Hungary - Croatia IPA Cross-border Cooperation Programme 2007-2013 and the Hungary - Serbia IPA Cross-border Cooperation Programme 2007-2013

approaching their closing stage, therefore conclusions can be drawn regarding the management of a whole project cycle and also on the results of the implemented projects. Projects that were selected for funding within the second call started their implementation mainly by the end of 2011, and those selected in the 3rd CfP have just started their implementation.

Call for Proposals launched

No of applications received

Total requested IPA (Community) funding

Number of funded projects

Total IPA (Community) funding

1st CfP

170

37,2 M EUR

69

17,9 M EUR

2nd CfP

217

43,4 M EUR

68 (selected)

12,3 MEUR

3rd CfP

n.a.

n.a.

58 (selected)

n.a.

Based on the number of funded projects, most projects are funded within Priority 2.1 and Priority 2.2 (40%, and 38% respectively), and only 9% and 13% are funded under Priority 1.1. and Priority 1.2. However, the size of the funds granted to projects within Priority 1.1 exceeds those granted in the other three priorities.

1.1.4. Summary statements Summarizing the above, in the 2007-2013 period mid-term evaluation has been replaced by a more flexible tool, the so called on-going evaluation having a few important distinctive characteristics: instead of taking a "snapshot" of the programme in a given moment it provides continuous feedback to the Programme management, thus it has extended strategic and operational functions. According to this new approach, increased attention should be paid to the new and novel aspects of the evaluation exercise, such as better involvement of stakeholders, making the whole process more clearly evidence based and providing coaching and moderation for the implementing bodies.

All these need a carefully designed implementation in terms of structuring and managing activities in a way that produce transparent implementation procedures, smooth cooperation with the Programme management bodies, clear and professionally structured reports with constructive and self-explanatory recommendations. On the other hand it is clear that the size of the project (two ongoing evaluation processes in parallel) calls for additional experts (a pool of experienced and available short term experts is a must) as well as highly professional and well performing project management.

1.2.

Risks and assumptions

The Consortium along with the two Subcontractors has made due provisions for an effective and successful mitigation of risks that might occur during a nearly two-year long implementation.

9

Organisation and Methodology for the On-going Programme Evaluation of the Hungary - Croatia IPA Cross-border Cooperation Programme 2007-2013 and the Hungary - Serbia IPA Cross-border Cooperation Programme 2007-2013

Based on the assumptions listed in the ToR the use of an appropriate tool-kit is probably the most important success criteria to mitigate risks. In the Strategy we have therefore presented a comprehensive set of methodological tools and instruments that fulfils this criterion: The methodologies and tools proposed are fully in line with the relevant recommendations of different EU Regulations Each tool presented is linked with the specific activities and tasks to be carried out The tools also reflect the previous experience brought by each Consortium member European Association for Information on Local Development (AEIDL) in its quality of helpdesk of the European Evaluation Network for Rural Development will provide a genuine added value contributing with its knowledge regarding state of the art evaluation techniques.

The Consortium shares the opinion of the Contracting Authority that the other main risk to be faced is the weakness of keeping the timeframes by providing a tough coordination between the multi-country management bodies and stakeholders. In our understanding the key factors of efficient risk mitigation lie in the Consultants’: experience in managing similarly sizable and multi-player projects in multicultural environment: setting up appropriate and efficient internal and external cooperation structures; setting up and operating intelligent project administration and monitoring systems to appropriately administer workflow and expert inputs and ensure inclusion of stakeholders/Task Force members knowledge of the Programme regions: mainly through involving subcontracting companies and experts with strong local presence and in-depth knowledge of both the Serbian and Hungarian territories capacity to provide sufficient and good quality human capacities: making available an exhaustive pool of short-term experts, resulting in increased flexibility and ability to act/react even within extremely tight deadlines and high volatility of the workload

10

Organisation and Methodology for the On-going Programme Evaluation of the Hungary - Croatia IPA Cross-border Cooperation Programme 2007-2013 and the Hungary - Serbia IPA Cross-border Cooperation Programme 2007-2013

2.

Strategy

2.1.

Our approach

2.1.1. Underlying principles of our work An evaluation meeting the highest quality standards Independence, impartiality: Evaluation, by nature, predominantly explores problems, opens up questions and makes proposals. For this reason it is of outmost importance that evaluators stay independent and impartial. Also, our consortium will find the right balance between different views and opinions when discussing sensitive issues.

Clashing opinions, validating information: Based on our previous experiences we will collect information from all relevant sources and have them screened from various aspects. This way the reliability of the data used will be higher and the evaluation judgment sounder.

Capitalisation: We are aware of the actions already taken in assessing the outcomes of the Programmes and also of the latest research findings. All new developments shall be built on these existing foundations.

Therefore our intention is not to reinvent the wheel, but to synthesise the outcomes of previous actions, developments, measurements and evaluations, which should create a sound basis for our work.

An evaluation that is useful Practice oriented approach, applicability: the purpose of this project is to enhance the efficiency of the Programmes and thus contribute to the development of the cross-border regions. This clearly means that our recommendations must be of practical use, meaning that they and their financial, organisational consequences have to be easily put into practice.

An evaluation with involvement and empowerment Transparency and awareness-raising: Universal acceptance of the evaluation findings can only be obtained through trust and commitment of the stakeholders. In order to achieve this, we must continuously make stakeholders aware of what is happening and why it is happening during the evaluation process. For this purpose, we use a wide variety of communication tools and techniques to reach stakeholders and keep them informed of the process and the outcomes.

Co-Creation: Co-creation is an elemental basic feature of our consultancy approach: we would rather act as a sparring partner of the key stakeholders, and not as a “know-all�. Co-creation requires partnership: evaluators function on equal terms and in trust based relations with the stakeholders, adopt independent standpoints, and combine psychosocial and management competences with expert knowledge. In practice, this can be done through common planning of evaluation processes, common analysis of findings and content-related discussions based on equal terms.

11

Organisation and Methodology for the On-going Programme Evaluation of the Hungary - Croatia IPA Cross-border Cooperation Programme 2007-2013 and the Hungary - Serbia IPA Cross-border Cooperation Programme 2007-2013

2.1.2. The pillars of our methodology We design a toolkit which fits the evaluation topic and the programme specificities The evaluation of the topics and questions presented in the ToR (to be finalized during the preparation of the Inception report) requires the use of a specific combination of various evaluation methods.

As a first step, we have identified the optimal tools to be used for the evaluation actions under each topic to define the optimal methodological tool-mix. This we have then refined along the following criteria in line with the ToR requirements and our above mentioned working principles: to be in line with the expected timeframe of the evaluation and enable efficient delivery to take into account the parallel evaluation of two different programmes; to ensure the highest possible quality evaluation results; to build on the principles of involvement and empowerment. However, participation in the evaluation process (workshops, individual interviews, focus groups, provision of data, etc.) should not put an excessive burden either on the institutional actors, or on the beneficiaries.

We identify feasible recommendations and support their implementation One of the key benefits of “evaluation during programme implementation� is that it formulates recommendations for improvements along the way to be introduced within the given programme period.

Even though there are no more calls expected in either of the programmes until the end of the period, our evaluation approach is to identify pragmatic proposals that can facilitate the most possible improvements during the programme lifetime.

As stressed by the EC Working Document No. 5, external evaluation experience should be combined with the skills and experience available by the programme stakeholders. We will therefore build on efficiently coordinated participatory methods, especially when proposing amendments or changes. This approach will improve the quality of recommendations and strengthen the commitment of relevant stakeholder groups to the actual introduction of the proposed changes. The specific recommendations (related to either strategy or implementation) resulting from the joint work with stakeholders may be 1) of a more general type, to be introduced in the long run (possibly only in the next programme period) or can be 2) specific "Quick win" recommendations (resulting in improvements in the short run).

We design schemes allowing for the parallel evaluation of the two Programmes As this present evaluation exercise involves the evaluation of two Programmes, we have made due provisions to ensure that the parallel tasks are delivered in the most efficient way and in the highest quality.

As the evaluation topics and questions presented in the TOR are the same for the two Programmes, we have designed a uniform methodological toolkit for both. However, considering the differences in the socio-economic background, the programme content, the implementation details and the stakeholders, the evaluation plans must be tailor made for the two Programmes during the inception phase, in co-creation with the Contracting Authority.

12

Organisation and Methodology for the On-going Programme Evaluation of the Hungary - Croatia IPA Cross-border Cooperation Programme 2007-2013 and the Hungary - Serbia IPA Cross-border Cooperation Programme 2007-2013

The scheduling of actions and tasks, presented in the time table under point 3.1 has been designed with careful consideration of the below critical issues: The overall timeframe of the delivery (start and end dates) is identical for the two programmes, however, in order to smoothen overloaded periods in such cases the tasks necessitating intensive involvement of the experts are scheduled with a time lap Based on the above, the limited human resources, especially key experts, will be smartly allocated between the two programmes Similarly, short-term experts offering an important additional capacity have to be involved in a systematically planned and organized way according to their competences in terms of area and thematic specialization A Joint management structure will be set up for the coordination of the parallel tasks between the two Programmes in terms of expert allocation, external and internal communication and administration.

2.1.3. The evaluation toolkit Primary and secondary data collection tools, as well as quantitative and qualitative methods provide the methodological basis of our proposal. This combination is necessary due to the complexity and depth of the evaluation themes. Our approach is to always use at least two sources of data/information to reach a sufficient level of data reliability.

We intend to follow both an evidence based (i.e. quantitative, based on facts and figures) and an opinion based approach (i.e. subjective sources, collected via interviews, workshops) in order to have the most reliable result and to draw the most reliable conclusions.

The most frequently used data collection tools are document analysis, analysis of statistical data (e.g. IMIS database) and analysis of other general sources such as media, internet, etc. combined with expert panels, interviews, moderated workshops.

Our evaluation toolkit covers the full evaluation cycle:

13

Organisation and Methodology for the On-going Programme Evaluation of the Hungary - Croatia IPA Cross-border Cooperation Programme 2007-2013 and the Hungary - Serbia IPA Cross-border Cooperation Programme 2007-2013

In-depth interview In-depth interviews are carried out in the form of bilateral 30-90 minutes long semi-structured in-depth conversations conducted by trained staff. The purpose is usually to collect specific information on the predefined issues, leaving room for a more detailed insight into specific topics reflecting the interviewee’s perspectives and views.

Focus group interview The focus group interview is a structured discussion method that involves the progressive sharing and refinement of participants' views and ideas. The technique is particularly valuable for analysing themes or fields which give rise to divergent opinions or which involve complex issues that need to be explored in depth. The advantage of using this particular evaluation method is that participants are less influenced by interaction with the evaluators than they might be in a one-to-one interview and it is possible to collect a large amount of qualitative information in a short space of time.

Co-creation workshops (CCWS) The workshop method is generally recommended to be applied in case of complex evaluation tasks, especially when quantitative tools are not applicable (e.g. non quantifiable impacts) and also when participants have different opinions and an independent decision making is necessary.

For co-creation workshops, we use the OPERA and GROUP EXPO tools. OPERA is a new efficient way of working that helps to achieve real change and to release group energy. GROUP EXPO is a specific cocreation workshop tool, applicable in case of large groups.

Co-creation workshops can be carried out as Expert panels involving both experts from the evaluation team and external specialists.

Questionnaire – online interview By means of an online questionnaire respondents fill in a web-based programmed questionnaire. Data collection is therefore automated so there is no need for an interviewer. As for technical facilities, respondents only need internet access and an e-mail address. Answers are systemized by the software and can be easily exported into XLS sheets. Furthermore, the web-based questionnaire allows for making a questionnaire query of complicated questions thanks to the special programmed sequence of questions.

The most frequent use of the online interview will be the beneficiary survey which will be combined with telephone interviews if it is needed.

Document analysis The first step of each evaluation exercise is the analysis of documents relevant to the issues to be assessed.

The document analysis is not just a review of existing documents but it also addresses the internal coherence and linkage of the document under evaluation with other reference documents.

The list of documents to be analysed for each evaluation topic / question will be presented in the evaluation plans.

14

Organisation and Methodology for the On-going Programme Evaluation of the Hungary - Croatia IPA Cross-border Cooperation Programme 2007-2013 and the Hungary - Serbia IPA Cross-border Cooperation Programme 2007-2013

Evaluation statistical analyses (EVALSTAT) Due to the size of the project a huge mass of statistical data should be processed and analysed. Large sample surveys and structured secondary data compiled in data files can easily be processed by means of EVALSTAT (statistical software) providing for a multiple statistical analysis to be carried out based on these data.

The most frequently used analytical methods will be as follows: crosstables, Correlation / Regression / Factor / Cluster analysis.

Indicator analyses In our understanding this tool concerns two main fields: The evaluation of the indicator system and quality of indicators in accordance with specific requirement systems (SMART, QQTTP) Assessing and analysing the indicators’ values

Sampling In order to create a representative basis for evaluation, we will use the statistic sampling method allowing the representation of all relevant aspects of the object of evaluation. Sampling will be the basis for selecting the projects which will go through in-depth analyses under various evaluation topics. Aspects applied for sampling will include project type, project size, beneficiary type and other relevant aspects.

15

Organisation and Methodology for the On-going Programme Evaluation of the Hungary - Croatia IPA Cross-border Cooperation Programme 2007-2013 and the Hungary - Serbia IPA Cross-border Cooperation Programme 2007-2013

Process modelling using value engineering techniques The objective of business process management (BPM) is to give a basis and to develop the efficiency of processes and the operation. The main aim of the tool is to define and analyse the present stage („as is”) in order to give a basis to the necessary future change management activities („to be”).

Values engineering techniques allows to assess the real added value of each procedural steps in the light of the value of the final product.

Logical Framework Method (LFM) The logical framework technique is an exercise in structuring the components of a project or single programme and analysing their internal and external coherence. The product is a logical framework, a formal matrix presentation of the internal functioning of the project / programme, measuring the fulfilment of main objectives, identifying external risks, facilitating the establishment of monitoring and control system of a programme.

Case Study Topic 7 expressively calls for identification and collection of best practices and case studies. In our understanding by analysing selected projects representing the most relevant achievements of the Programme objectives at project level a best case catalogue will be compiled.

Case studies to be elaborated and included in the catalogue will provide for multiple use for the given evaluation: Illustration: justifying conclusions of the evaluation with real cases; Exploration: allowing hypotheses for further studies; Critical analysis: validation of certain findings of the exercise.

Furthermore, the best case catalogue can serve as a methodological support document for other Programmes as well.

2.2.

Activities and related inputs and outputs

Evaluation Plan The objective of the Evaluation Plan is to carry out the on-going evaluation of the Hungary - Croatia IPA Cross-border Cooperation Programme 2007-2013 and the Hungary - Serbia IPA Cross-border Cooperation Programme 2007-2013. As defined in the ToR, the Consortium will produce two Evaluation Plans for the HU- HR and the HU-SRB IPA Programmes respectively, in line with “EC Working Document No.5: Indicative Methods on Evaluation Methods” and in close co-operation with the JTS/MA, to lay down the evaluation's methodology and co-ordination.

The Evaluation Plans include the following key-elements: Methodology and techniques to be adopted within the evaluation process; 16

Organisation and Methodology for the On-going Programme Evaluation of the Hungary - Croatia IPA Cross-border Cooperation Programme 2007-2013 and the Hungary - Serbia IPA Cross-border Cooperation Programme 2007-2013

Evaluation questions communicated by the Programme bodies (will be provided by the JTSs).

However, the contractor should be able to add further questions, if needed; List of documents or types of information required from the Programmes; Timeframe; Tasks of each expert within the working team.

17

Organisation and Methodology for the On-going Programme Evaluation of the Hungary - Croatia IPA Cross-border Cooperation Programme 2007-2013 and the Hungary - Serbia IPA Cross-border Cooperation Programme 2007-2013

2.2.1. Inception Phase 2.2.1.1.

Inception report

Activities

Act 0.1: Kick-off meeting

Act 0.2: Updating the Evaluation Plan Act 0.3: Preparation of Inception Report

Inputs

Methodology/tools used

List of questions provided by JTS; Programme documents and relevant EU

Consultant moderated co-

regulations; other relevant studies; previous evaluation reports

creation workshop

Outputs Finalized evaluation questions; list of documents to be reviewed

Document analysis Act 0.1 outputs

Final Evaluation Plan Drafting

Outputs of Act 0.1 and Act 0.2; relevant documents

Drafting and reviews of JTS

Inception Report

2.2.2. 1st Evaluation Phase 2.2.2.1.

Topic 1: Direct comparison of the Programme objectives and results so far

Activities

Inputs

Act 1.1: Examination of

Selection of relevant ETC / national / regional programme documents

programme relevance

Hungary-Croatia IPA CBC Programme 2007-2013, Hungary-Serbia IPA CBC Programme 2007-2013

Benchmarking ; Analysis of socio-

Recent macro-economic studies, strategy documents, relevant statistical data at territorial level 18

Methodology/tools used

Outputs

Comparative document

State of Play report on

analysis

Programme objectives and results

Organisation and Methodology for the On-going Programme Evaluation of the Hungary - Croatia IPA Cross-border Cooperation Programme 2007-2013 and the Hungary - Serbia IPA Cross-border Cooperation Programme 2007-2013

economic situation and changing needs; SWOT and strategy

Focus group of applicants – 1 per programme

Focus group

Expert panel to recreate the SWOT and the logframe of the programmes based on the current status of the area, comparison of the original objectives and strategy with the updated ones

Expert panel

Act 1.2: Comparison of objectives and results: Thematic scopes, geographical scope, partnerships, cost

Existing programme level databases and the document analysis of Act 1.1, allowing for analysis of data / information regarding the key topics under evaluation question 1.2 identified in the ToR

efficiency,

IMIS and project progress reports

cross-border impact,

EVALSTAT method

sustainability, evaluation of sources vs. emerging needs, demands

2.2.2.2.

Topic 2: Assessment of the project selection procedure and criteria

Activities

Act 2.1: Evaluation of application procedure (from call to submission)

Inputs

Methodology/tools used

Programme Documents (Project implementation Handbook, Application packages of calls)

Document analysis

Documents related to information days and seminars (agenda, FAQ, minutes)

Process modelling using value engineering techniques

Evaluation documentation and reports Members of the JMCs (min 2 prs/committee), Project managers of the JTS/and or Information points (4 prs – 2 interviews/ Programme), 19

In depth interviews

Outputs Critical analysis of application procedure and recommendations (e.g. on call schedule, accessibility, support mechanisms, administrative and content criteria)

Organisation and Methodology for the On-going Programme Evaluation of the Hungary - Croatia IPA Cross-border Cooperation Programme 2007-2013 and the Hungary - Serbia IPA Cross-border Cooperation Programme 2007-2013

Programme Planners (2 prs – 1 interview /Programme) Target group: evaluators (1 group/ Programme), consultants specialised in grant application drafting (1 group/programme), Beneficiaries (1 group/programme)

Focus Group interviews

Sampling: all intervention areas covered by evaluators carrying out large numbers of evaluations (coordinated with the Programming Bodies)

Act 2.2: Evaluation of assessment procedure (selection)

Documents related to the selection and preparation of the experts participating in the pre-selection and evaluation (selection criteria, training or orientation of evaluation, selection process documents)

Process modelling using value engineering techniques

Minutes of the Monitoring Committees related to evaluation IMIS : Quantitative data related to both approved and rejected applicants, beneficiaries

Database analysis

Cost data related to the selection process (if available)

2.2.2.3.

Critical analysis of assessment procedure and recommendations (e.g. on consistency of objectives, efficiency of selection criteria, contribution of administrative/formal requirements to expected results)

Topic 3: Evaluation of Programme performance by indicators

The programme indicator systems are designed to support the continuous monitoring activity. While the monitoring can only apply the indicators, evaluation can go beyond, and assess the performance of the indicator system as a measurement tool. The objective of this evaluation topic is to contribute to the establishment of a better indicator system.

Activities

Inputs

Methodology/tools used

Outputs

Act 3.1: Project based overview of Programme performance

Project financial and administrative data available from IMIS

Data analysis; EVALSTAT

Comprehensive overview analysing programme financial performance by priority area, country, sector, type of beneficiary

Act 3.2: Indicator based evaluation of programme

Programme and project level indicators (comparison of baseline, targeted, time-proportional and current indicator values)

Indicator analysis

In-depth review of

20

Programme performance

Organisation and Methodology for the On-going Programme Evaluation of the Hungary - Croatia IPA Cross-border Cooperation Programme 2007-2013 and the Hungary - Serbia IPA Cross-border Cooperation Programme 2007-2013

performance

against indicators, identification of gaps and low performing areas; Evaluation of the current indicators system:

Act 3.3: Assessing current Indicator system vs. SMART and QQTTP criteria.

Focus groups: planners and implementation officers Programme and project level indicators; Programme objectives; output of Act 3.2

Focus group interviews, LogFrame

recommendations to strengthen the link between project and Programme level indicators and improvement of measurement potential

2.2.2.4.

Topic 4: Revision of the Programme Communication activities based on the Communication Plan

Activities Act 4.1: Overall assessment of effectiveness of Programme level communication (comparison of objectives with concrete activities and results)

Inputs

Methodology/tools used

Communication plans of the Programmes and reports on activities and results

Document analysis

Analysis of publications and promotional tools of the Programmes Analysis of media coverage of the two Programmes Website audit, analysis of web statistics of Programme websites

Act 4.2: Assessment of visibility of the Programmes at EU and participating country level

Outputs

Document analysis

Web-based and traditional questionnaire surveys with applicants who had submitted project proposals

Questionnaire survey

In depth interviews (personal and telephone) with selected applicants as well as the JTS / MA

In-depth interview

21

Evaluation report on Programme communication performance and visibility

Organisation and Methodology for the On-going Programme Evaluation of the Hungary - Croatia IPA Cross-border Cooperation Programme 2007-2013 and the Hungary - Serbia IPA Cross-border Cooperation Programme 2007-2013

2.2.2.5.

First Evaluation Report:

Activities

Inputs

Act 0.4: Preparation of Draft First

Outputs of Topics 1-4

Methodology/tools used

Evaluation Report and discussing with Programme bodies

Drafting Interviews with programme bodies

Outputs

Draft First Evaluation Report

Act 0.5: Elaboration of First Evaluation Report incorporating comments and feedback from

Feedback of Programme bodies

Drafting

Final First Evaluation Report

Programme bodies

2.2.3. 2nd Evaluation Phase 2.2.3.1.

Topic 5: Contribution to European Policies: Analysis of the Programme contribution to key European and macro-regional policies

Activities

Act 5.1: Analysis of Contribution to European Policies (additional deliveries provided, bridges to policy deliveries)

Inputs

Methodology/tools used

First evaluation report (output of first evaluation phase) reflecting the status of Programme performance; Europe2020, Cohesion Policy, European Integration Agenda, Macroregional strategies such as Danube Region Strategy, other relevant publications of DG Enlargement

Expert panel with key persons responsible for the selected policy / 22

Document analysis of key features / objectives of relevant policies / strategies; Keyword analysis (to what extent the keywords of the relevant policy and strategy documents are present in the Programme strategies, calls and application forms) Expert panel

Outputs

Analysis identifying the level of contribution of the two Programmes to European policies, listing additional deliveries

Organisation and Methodology for the On-going Programme Evaluation of the Hungary - Croatia IPA Cross-border Cooperation Programme 2007-2013 and the Hungary - Serbia IPA Cross-border Cooperation Programme 2007-2013

strategy areas and representatives of Programme Bodies

2.2.3.2.

Topic 6: Future: analysis and evaluation of Programme deliveries and results in order to provide recommendations to the Participating Countries to the next programming period (2014- 2020)

Analysis and appraisal of the programme results is of key importance in terms of the planning of future programmes, thus one of the focuses will be on causalities so that end users of the Evaluation Report could fully understand what interventions and what mechanisms worked and why , and what were less successful. Results will not only be identified in terms of indicators planned and achieved but identified as results by the beneficiaries, and programmes managers. The topic is complex since it covers both thematic results of the programmes as well as management issues, thus the analysis will also synthesize these two aspects.

Activities

Inputs

Methodology/tools used

Outputs

Outputs of topics 1 to 5 and the First Evaluation Report Monitoring Reports, Annual Programme Reports Act 6.1 Strategic assessment of Programmes and their implementation, including evaluation of current management structures (division of tasks, daily operation, management)

Act 6.2: Country-specific (HU, SR, HR) analysis of Programme

Documents describing management structure, division of tasks (eg: audit trails, internal guides, etc)

Document analysis and Drafting

Focus groups: key players of Programmes , members of the JSC (min 2 prs/committee), project managers of the JTS, members of other management functions (minimum 4 interviews/Programme), beneficiaries (1 group meeting/Programme)

Focus groups

Previous CBC-programme results, other CBC programmes (e.g. HungaryRomania Cross-Border Co-operation Programme 2007-2013, HungarySlovakia-Romania-Ukraine ENPI Cross-border Cooperation Programme 2007-2013).

Comparative analysis

Outputs of Act 6.1, major strategic documents and development concepts and plans of the involved countries

Document analysis

Target groups: 330 project owners

Questionnaire survey 23

Future oriented comprehensive study listing joint results and problems and hot topics for the next programming period and recommendations for better performing management structures

Country-specific Evaluative Reports of Programme deliveries and results and lists of recommendations,

Organisation and Methodology for the On-going Programme Evaluation of the Hungary - Croatia IPA Cross-border Cooperation Programme 2007-2013 and the Hungary - Serbia IPA Cross-border Cooperation Programme 2007-2013

deliveries and results

2.2.3.3.

Workshop focus groups: smart mix of planners and implementers at national/Programme level, including representatives of different policies covered by the Programme priorities (e.g tourism, economy, environment, infrastructure, culture, education, etc): 4 thematic workshops/Programme

one per country CCWS

Topic 7: Case Studies and Best Practices

Activities

Inputs

Methodology/tools used

Outputs

Desk research of existing documents and reports

Act 7.1: Collection of Best Practices Compilation of Case Studies

Preliminary selection of good practices compiled by the JTS based on pre-defined criteria Information gained from beneficiaries and Programme management bodies during Topics 1-6 Outputs of Topics 1-6

Final selection and analysis of 4 projects per Programme priority based on a pre-defined template for carrying out the case studies Phone interviews based on a pre-defined questionnaire template

Collection of Best Practices and a list of Case Studies (best practices potentially covering a wide range of areas, 16 case studies).

On site visits to selected beneficiaries to cover strategic issues having added value for the evaluation

2.2.3.4.

Topic 8: Benefits of the Programme in terms of preparing the non-member states (Croatia, Serbia) for the absorption of Structural Funds at the time of accession to the European Union

Both in case of Croatia and Serbia, the EU programmes also have a special role to contribute to the preparation of these states for future accession. Although national programmes under pre-accession funds have targeted facilities for this purpose, the cross-border cooperation programmes provide the opportunity of collaboration with a MS, and therefore promote know-how transfer. The evaluation shall assess the scale and scope of this contribution.

24

Organisation and Methodology for the On-going Programme Evaluation of the Hungary - Croatia IPA Cross-border Cooperation Programme 2007-2013 and the Hungary - Serbia IPA Cross-border Cooperation Programme 2007-2013

Activities

Inputs

Methodology/tools used

Outputs of Topics 1-7

Document analysis

Focus group interviews: with Croatian and Serbian Programme bodies and beneficiaries of the Programmes. The interviews will explore their perception and experiences on benefits, know-how transfer and awareness

Focus group interview

Act 8.1: Comprehensive Study on Programme operation and

Questionnaire survey

benefits for Croatia and Serbia in

Comparative analysis comparing the impacts on beneficiaries and similar organisations outside the programme area (control groups)

terms of preparing for Structural Funds

Questionnaire survey involving beneficiaries and control groups outside the programme area

In depth interviews: with key officers of the First Level Control bodies in Croatia and Serbia to explore the developments in relation to the legal and management framework, experiences, awareness and problems encountered

2.2.3.5.

Outputs

Lessons learnt from the implementation of the current Programmes for the IPA countries: joint and country-specific experiences and recommendations/ statements for Serbia and Croatia Evaluation Report on Croatia’s special benefits regarding the use of IPA experiences in shifting to Structural Funds regulation

In depth interview

Second Evaluation Report

Activities

Inputs

Methodology/tools used

Outputs

First Evaluation Report

Document analysis

Draft Second Evaluation

Interview with Programme bodies

Interview

Report

Act 0.6: Monitoring the implementation of all recommendations made in previous phases and assessment of the impact of previous reports 25

Organisation and Methodology for the On-going Programme Evaluation of the Hungary - Croatia IPA Cross-border Cooperation Programme 2007-2013 and the Hungary - Serbia IPA Cross-border Cooperation Programme 2007-2013

Act 0.7: Preparation of Draft Second Evaluation Report and

First Evaluation Report

discussing with Programme

Interview with Programme bodies

Drafting

bodies

Final Second Evaluation Report

Act 0.8: Elaboration of Second Evaluation Report incorporating comments and feedback from

Feedback of Programme bodies

Drafting

Inputs

Methodology/tools used

Programme bodies

2.2.4. Final phase Activities Act 0.9: Monitoring the implementation of all recommendations made in previous phases and assessment of the impact of previous reports Act 0.10: Preparation of Draft Final Report and discussing with Programme bodies Act 0.11: Elaboration of Final Report including comments and feedback from Prog. bodies

First Evaluation Report

Outputs

Document analysis

Second Evaluation Report

Interview

Interview with Programme bodies

Draft Final Report First Evaluation Report

Drafting

Second Evaluation Report Presentation and discussion

Interview with Programme bodies Feedback of Programme bodies

Drafting

26

Final Report

Organisation and Methodology for the On-going Programme Evaluation of the Hungary - Croatia IPA Cross-border Cooperation Programme 2007-2013 and the Hungary - Serbia IPA Cross-border Cooperation Programme 2007-2013

2.3.

Presentation of Consortium and division of tasks

2.3.1 General approach Based on the provisions given by the ToR, the professional team executing the ongoing evaluation of the Hungary-Croatia and Hungary-Serbia IPA Cross Border Cooperation Programmes needs to meet a set of criteria both in terms of appropriate knowledge and skills, and in terms of human capacity and management capabilities. In this sense in our understanding the following factors are the key drivers of the successful implementation: First-hand experience at consortium level with the cross-border areas, the thematic fields of the Programmes and preparedness regarding state of the art evaluation methods and tools also in the changing EU policy context Availability of a critical mass of professionals coming from all sides of the borders, equipped with the necessary methodological tool-kit and relevant area/thematic knowledge, together with Sound and result-oriented process management operated by committed and experienced senior people, including appropriate and transparent communication between stakeholders during the entire period of the implementation

2.3.2 The consortium Our team is formed to best perform the expected tasks through: Setting up a Consortium composed of expert companies based in Hungary and Brussels with highly relevant complementary experience, complemented by Subcontractors bringing in country and border-region specific knowledge from Serbia and Croatia.

Providing key experts perfectly fitting the requirements detailed in the ToR complemented by a large pool of quality short term experts with experience in evaluation in the given cross-border areas and in a variety of thematic fields covered by the Programmes.

Due to the size and intensity of the workload, safe project implementation requires a high flexibility from the implementation team: our consortium complemented by the two subcontractor companies will at any time have the necessary human resources in terms of quantity and quality to provide a buffer capacity to the project.

HitesyBartuczHollai Euroconsulting (Hungary) HitesyBartuczHollai Euroconsulting, the Consortium Leader, has over 22 years of experience in providing TA and consulting services for the design and implementation of EU financed regional development programmes and projects. The vast knowledge and expertise acquired makes HBH Euroconsulting one of the leading regional consulting firms in the field of developing and managing ETC (Interregional, Transnational and Cross-border Cooperation) projects in Hungary and in the Central European Region. In the context of regional development HBH Euroconsulting has built up valuable sector-related knowledge as 27

Organisation and Methodology for the On-going Programme Evaluation of the Hungary - Croatia IPA Cross-border Cooperation Programme 2007-2013 and the Hungary - Serbia IPA Cross-border Cooperation Programme 2007-2013

well, including economy, transport, environment (especially energy efficiency and renewable energy) demography and cultural heritage.

Clients include public authorities at different levels (central governmental institutions, regional bodies and municipalities), universities and knowledge institutes, public foundations and private companies from all over Europe. Services provided to these clients include a wide range of expertise. The most relevant services in the context of this evaluation exercise are: Programme and project monitoring and evaluation services related to the implementation of large scale TA projects for Hungarian central government bodies (ROP MA, NDA, Ministry of Health, ESF MA) Multiannual TA services for Central bodies (NDA, Ministry of Economy, ROP MA) Design and implementation of ETC projects for CBC and transnational programmes (HU-SRB, HURO, Central Europe, SEE, NWE) with a high success rate (more than 80%) Regional and Urban Development, Spatial Planning HBH Euroconsulting has developed in the past 22 years to become a solid company working with a permanent team of 20 employees and a large and well-equipped office infrastructure. The multinational staff (including native Romanian, German, French, Dutch speakers) operates permanently in a bilingual (English and Hungarian) working environment and has knowledge of further languages such as Croatian, Spanish, Italian, Greek. HBH will therefore contribute to the successful implementation of this project by setting up a professional project management unit (PMU) operating a transparent internal communication within the Consortium, clear procedural rules and structured project administration.

ICG Ex-Ante (Hungary) ICG Ex-Ante has been active in the field of strategic and management consulting as a Hungarian consulting company for almost 10 years now, with assignments mostly related to programmes of the Structural Funds and earlier the pre-accession funds. The company is an integral part of an international group working with over 100 experienced consultants, operating in offices located in 6 countries of Central and South-Eastern Europe. As part of the multinational expert groups set up within the ICG group, Ex-Ante acquired a unique Central and South-East European knowledge and expertise and has become familiar with the implementation methods of Structural Funds in other Member States of the EU as well, the respective border areas in particular. ICG Ex-Ante has a vast record of successfully implemented projects in the fields of: Evaluation of EU funded Programmes Management services for EU financed projects and Programmes Development strategies Urban and regional development Its clients are international organisations, central administrative bodies, public institutions, local governments, NGOs and enterprises. ICG Ex-Ante supports them in implementing their development projects and in establishing the pre-conditions for sustainable and efficient operation.

28

Organisation and Methodology for the On-going Programme Evaluation of the Hungary - Croatia IPA Cross-border Cooperation Programme 2007-2013 and the Hungary - Serbia IPA Cross-border Cooperation Programme 2007-2013

ICG Ex Ante operates with a well prepared staff possessing significant knowledge and experience concerning the theoretical background and practical implementation of the various research methodologies and data collection techniques in their area of expertise.

The added value of ICG Ex Ante within the consortium comes principally from its experience of evaluating Programmes and state of the art knowledge and practical experience with evaluation methodologies that constitute the basis of the tasks to be performed in the present project. The methodologies to be used in this project are theoretically well established (following the guidelines set by the European Commission) and have also been continuously improved by the company to strengthen the functional usefulness of the evaluations.

European Association for Information on Local Development AEIDL (Belgium) Established in 1988, the European Association for Information on Local Development (AEIDL) is a multicultural and multilingual non-profit organisation specialising in the management and coordination of transnational networks; the study and evaluation of European programmes on local and regional development; the dissemination of knowledge and methodological tools, particularly in the fields of territorial development and the evaluation of public policies.

The permanent core team of AEIDL comprises of experts in rural and urban development, regional policy, the environment, social policy, that is complemented by an extensive network of external partners and experts in every country of the European Union. This ensures the best combination of content-related expertise, geographical coverage and communication capacity. It also facilitates the development and management of inter-cultural exchanges, which is a key competence of AEIDL. In addition to the provision of services to administrations of different Member States (evaluations of development programmes, provision of networking expertise, training in the local development approach, support for programming, etc.), AEIDL has over 23 years of experience in coordinating contracts for the European Commission. It currently works with the Environment, Employment and Social Affairs, Agriculture, Maritime Affairs and Fisheries, Mobility and Transport, and Regional Policy DGs. In the context of this Evaluation Project, AEIDL will contribute with its extensive experience of working on evaluation of public policies. Since 2009, AEIDL is a member of the EEIG ‘RUREVAL’ together with MetisEurope (Austria), in charge of coordinating the network of evaluators in the field of rural development. It also conducted several smaller evaluation studies in the field of local development, regional development, LEADER, EQUAL, etc. More recently (2012), AEIDL has been awarded the contract to undertake the evaluation of the LEADER programme in Aquitaine, France, in which cooperation and cross border issues are assessed by using a benchmarking method (comparisons with Austria, Sweden and other European regions). This evaluation study will provide guidance for the local development dimension in the future cohesion policy.

29

Organisation and Methodology for the On-going Programme Evaluation of the Hungary - Croatia IPA Cross-border Cooperation Programme 2007-2013 and the Hungary - Serbia IPA Cross-border Cooperation Programme 2007-2013

Sub-contractors

Institute for Territorial Economic Development (Serbia) InTER – Institute for Territorial Economic Development was established in 2006 as an independent nongovernmental think tank in Serbia with the mission of promoting and advancing sustainable socio-economic territorial development in the Western Balkans, including cross-border initiatives.

Clients of InTER include international organisations, central administrative bodies, public institutions, local governments, NGOs and enterprises and its main fields of expertise are support and assistance in: Assessment of cross-border cooperation project proposals to be implemented in CBC Programmes Carrying out mid-term and final evaluations of long-term programs and single-projects for European Commission, Sida, ADA, UNDP, and others (altogether more than 13 organisations) Designing policies and programmes on territorial development Capacity building for the public sector (principally public administration organisations and local governments), private enterprises and civil society organisations Identifying funding opportunities, as well as supporting the preparation of project proposals and implementing projects also in CBC context InTER has also acquired significant experience in the implementation of the latest practices, procedures and regulations in the above listed areas and the ability, through its multi-disciplinary approach, to meet and adapt to the specific needs of clients.

InTER's key contribution to the current evaluation work will include its in-depth knowledge of the HungarySerbia border region and the considerable expertise in setting up teams of thematic experts bringing various expertise linked to programme and project evaluation: InTER has a long-lasting cooperation and working relationship with experts involved into project evaluation of the present HU-SR CBC Programme.

Karzen i Karzen (Croatia) KARZEN I KARZEN is a Croatian private consulting firm operating since April 2007 in the area of regional development, preparation and implementation of EU funded projects, strategic planning, institution development, education, as well as project management, monitoring and evaluation.

The firm provides technical assistance to national, regional and local governments, intergovernmental agencies, SMEs, regional development agencies and civil society organisations.

Karzen i Karzen’s activities are concentrated on the following key focus areas: socio-economic and human resource development, cross-sectorial development/cooperation, social innovation: challenging old methods and creating new tools, practices and models to socioeconomic challenges, by bridging technical innovations with human outcomes project development for CBC Programmes

30

Organisation and Methodology for the On-going Programme Evaluation of the Hungary - Croatia IPA Cross-border Cooperation Programme 2007-2013 and the Hungary - Serbia IPA Cross-border Cooperation Programme 2007-2013

project evaluation including CBC project proposals Karzen i Karzen is committed to provide the best possible contribution to the implementation of the present evaluation project driven by the following working principles: Local knowledge: practical and up-to-date knowledge of the local economic and policy context and actors in the HU-HR region Participative culture: interpreting participation as an end in itself, thus taking the time to develop and enhance the skills, knowledge and abilities of individuals.

Adopting a “co-creation” approach: open up active and tailor-made communication channels at the grassroots level to ensure all voices are actively involved in crafting the intended outcomes Inclusiveness: working with leaders from a broad range of communities and organizations and consciously working to foster cooperation with groups that have been “absent from the table” so far.

2.3.3 Division of tasks and responsibilities between the members of the consortium In our experience when implementing large, multi-player assignments clear-cut division of tasks between professional players is of key importance. Based on lessons learnt in similar projects the guiding principles of the internal cooperation are as follows:

Lead Contractor principle As Lead Contractor, HBH Euroconsulting will take the overall responsibility regarding the implementation of the project, including process management & monitoring, project administration, facilitation and technical support and acting as the Secretariat for the Programme Management Bodies.

It will also be responsible for setting up and operating a QC team together with AEIDL assuring due quality control over the whole implementation team.

In communications with the Contracting Authority we also follow the “single communication channel principle”, meaning that all communication will be channelled through the LC who will take care of the further internal distribution and allocation of information.

Competence based division of tasks HBH Euroconsulting: further to management tasks HBH Euroconsulting delegates the Team Leader (TL) and thus will fulfil the overall thematic coordination both towards key and short-term experts and the two management bodies behind the Programmes. It will bring in its particular project level expertise acquired through the design and management of over 40 ETC projects over the last 6 years.

ICG Ex-Ante is highly experienced in Programme evaluation: it has carried out more than 10 exante, mid-term and final evaluations over the past 5 years. ICG Ex-Ante will therefore contribute 31

Organisation and Methodology for the On-going Programme Evaluation of the Hungary - Croatia IPA Cross-border Cooperation Programme 2007-2013 and the Hungary - Serbia IPA Cross-border Cooperation Programme 2007-2013

with its hands-on experience in the design and daily implementation of the work plan and delegate a highly senior key expert for the daily implementation as a token to carrying out this ongoing evaluation exercise as smoothly as possible.

AEIDL’s a strategic role in the project implementation lies in providing a continuous methodological and Quality Control support over the evaluation process as well as tools and techniques and the final products of the 4 evaluation phases.

Subcontractors will have an important and visible contribution to implementing the project, bringing in their particular expertise and area knowledge regarding Croatia and Serbia. They will provide complex professional support activities such as collection and processing of data, carrying out field research, organising meetings, workshops and interviews. Also, they will provide practical support for the implementation in the form of office infrastructure, organisation, mobilisation, public involvement, effective communication and dissemination in their respective countries. As an additional contribution, one of the key experts proposed for this project has been delegated by InTER. Furthermore, Consortium members and Subcontractors have carefully selected short term experts to cover specific sector-related thematic areas, including regional development, economy, innovation, tourism, transport, environment and cultural relations.

Balanced workload between parties The commonly agreed division of tasks will result in a balanced participation of the members of the implementation team (Consortium members and Subcontractors) in the whole process of implementation keeping in mind that subcontracting cannot exceed the 20% threshold.

Multi-country working culture The successful implementation of the project needs continuous interactions among Consortium members, Subcontractors and experts. Besides the high level command of English, all team members and experts are familiar with working in multinational teams and multicultural environment.

Daily operation of the Implementation Team The daily operation of the Implementation Team will be based on clearly defined internal procedural and communication rules. Progress of the implementation will be closely monitored by monthly project meetings organized by the Team Leader (reporting to the Consortium management) and assessed by quarterly evaluation meetings of the Consortium members represented at management level.

Subcontractors will be invited to meetings whenever their presence is important. Regular meetings with the Client will be held to discuss progress and contractual issues, including eventual deviations or necessary modifications due to changes in the project environment.

32

Organisation and Methodology for the On-going Programme Evaluation of the Hungary - Croatia IPA Cross-border Cooperation Programme 2007-2013 and the Hungary - Serbia IPA Cross-border Cooperation Programme 2007-2013

2.3.4 The team The quality and preparedness of the key experts is the real key to successful project implementation. The experts selected by the Consortium are all senior consultants having state of the art methodological and management competences in order to “navigate” the project within the predefined frames in a smart way and with high flexibility.

Team Leader Hen Gerritse, a Dutch citizen is a European expert specialised in regional development in general and in ETC programme design and implementation in particular for 30 years. He drafted two transnational programmes (NWE and North Sea) and assisted the development of the Interreg IVC Programme. As an expert of INTERACT he moderates workshops/training events. He is also a strategic advisor for several cooperation projects based on his in-depth knowledge on EU policies with a strong focus on territorial cooperation. Having worked for EU (NWE JTS) and national public authorities (Dutch Ministry of Housing), he was in charge of the monitoring of CBC and transnational programmes.

Key Experts Zoltán Barna-Lázár is a Hungarian regional development expert specialised in programme evaluation. His 7 years of experience in evaluation and in monitoring of various EU programmes encompassed evaluator activities regarding both the HU-SR and HU-HR CBC Programmes. Furthermore he assisted the preparation and implementation of several CBC and transnational projects (HU-RO, HU-UA, HU-HR, SEE, CE). Based on his 12 years working experience in regional development, he is highly specialised in economy (including SME development), tourism, transport and human resource development.

Zoran Krtinic is a Serbian expert, with more than 6 years of experience in programme development supporting the Serbian National Authority in designing and implementing the HU-SR IPA CBC Operational Programme. As a CBC expert, Zoran has also supported project applicants and beneficiaries in project development, implementation (procurement in particular) and networking activities in the CBC region. He is a skilled evaluator, he has carried out the evaluation of grant project proposals in HU-SR IPA CBC Programme, and provided technical assistance to the EU Delegation to Serbia in all procurement matters relating to the evaluation of tenders. Besides his excellent English and native Serbian/Croatian language skills, he has a high level command of Hungarian.

Short Term Experts A pool of Serbian, Croatian and Hungarian short-term experts with competences in project and programme evaluation, a sound area-related knowledge and complementary thematic experience is available to provide inputs for specific parts of the project. Most of them used to work together previously, thus creating a common working culture will be easy.

The following table provides the list of the short term experts with their competences:

33

Organisation and Methodology for the On-going Programme Evaluation of the Hungary - Croatia IPA Cross-border Cooperation Programme 2007-2013 and the Hungary - Serbia IPA Cross-border Cooperation Programme 2007-2013

Name

Country

Relevant language

Exp. (yrs)

Relevant experience

Evaluation experience in the area

Key areas of expertise

Project pipeline for IPA/EU SF – Europeaid/125567/D/Ser/HR Suzana Kovacevic

Croatia

SR, HR, EN,

15

Project management, development, evaluation

IPA II CBC HU-HR IPA IV Human Resources Development OP

Economy, labour market, social inclusion, gender issues

IPA II CBC HR-BiH Ivana Markanovic (Vlasic) Mirna Karzen

Croatia

SR, HR, BI, EN

Croatia

SR, HR, BI, EN

11

Programming, programme evaluation, project assessment

23

Programming, programme evaluation, project development, capacity building

41

Programming, programme monitoring, programme evaluation, capacity building

IPA HR -MNE CBC Programme – 11 projects IPA HU - HR CBC Programme – 36 projects

LLP and Grundtvig Programme

HU-SR CBC Programme – 61 projects (2012) Gordana Lazarevic

Serbian

HR, EN, SI, SR

Progress Programme – 22 projects HU-SR CBC Programme – 26 projects (2011) RO-SR CBC Programme – 34 projects

Jasna Hajder

Dragisa Mijacic

Mirjana Strugar

Serbian

HR, SR, EN

Serbian

HR, SR, EN

Serbian

HR, SR, EN

12

Planning, Programming, Project development and management

Provincial Programme on local economic development in Vojvodina HU-SR CBC Programme (2011-2012) EU IPA CBC programmes:

13

Programme evaluation (midterm and final), Project assessment, Project development, training

HU-SR CBC Programme

9

Programme evaluation, project assessment, financial monitoring

BG-SR, SR-HR, SR-MNE, SR-RO - altogether 84 projects EU Commission, Sida, ASA, UNDP – 13 programmes and singleproject HU-HR CBC Programme LEDIB (local economic development) Programme 34

Ecology, environment, infrastructure, social infrastructure Tourism/ecotourism, socioeconomic development, business development, SMEs, natural resources Socio-economic development, civil society issues, environment, people to people actions Local economic development, human rights and democracy, civil participation, gender issues, ethnic issues Socio-economic development, rural development, cluster development Infrastructure, research, tourism, environment

Organisation and Methodology for the On-going Programme Evaluation of the Hungary - Croatia IPA Cross-border Cooperation Programme 2007-2013 and the Hungary - Serbia IPA Cross-border Cooperation Programme 2007-2013

UNDP MIR2 Programme Ferenc Szilágyi

Gabriella Somogyi

Ferenc Tátrai

Hungary

Hungary

Hungary

HU, EN

HU, EN

HU, EN

HU-SR CBC Programme

21

Monitoring, project assessment

6

Project assessment, project development and management

HU-SK CBC Programme

HU-SR CBC Programme – 60 projects

42

Programme management, project assessment, project development

HU-HR CBC Programme

HU-HR CBC Programme

HU-HR CBC Programme SI-HU CBC Programme

Water management, thermal waters, environment, sewage management Tourism, marketing, environment, urban development, cultural heritage Human resource development, labour market, vocational training, innovation, business incubation

HU-SK-UA CBC Programme AT-HU CBC Programme Boglárka Bakos

Hungary

HU, EN

17

Programming, project development, project evaluation,

HU-RO CBC Programme SI-HU CBC Programme

Tourism, urban rehabilitation

HU-HR CBC Programme HU-SR CBC Programme West-Transdanubian OP HU-SK CBC Programme HU-HR CBC Programme Balázs Wächter

Hungary

HU, EN

15

Programming, programme evaluation (mid-term and exante), project evaluation, project development

HU-SR IPA CBC Programme HU-RO CBC Programme ROP Hungary Economic development projects under the Hungarian GrowthPoles HU-SK-UA Neighbourhood Programme AT-HU Interreg IIIA proposals 35

Socio-economic development, urban development, HR development, tourism

Organisation and Methodology for the On-going Programme Evaluation of the Hungary - Croatia IPA Cross-border Cooperation Programme 2007-2013 and the Hungary - Serbia IPA Cross-border Cooperation Programme 2007-2013

LEADER in Aquitaine (F) (October 2012 – April 2013) ESF Expert Evaluation Network for METIS-DG EMPL ESF technical assistance to transnational cooperation and Katalin Kolosy

Belgium

EN, HU

30

Programming, programme/project evaluation, monitoring

innovation for DG EMPL Children’s rights in Belgium and Luxemburg (DG Justice) LEADER+ “One-stop-mobility-shops” (PROGRESS) Member of the evaluation team for the EQUAL project Insertion-emploi.net (FR-MDP-2001-10095)

36

agriculture, gender, humanrights, entrepreneurship

Organisation and Methodology for the On-going Programme Evaluation of the Hungary - Croatia IPA Cross-border Cooperation Programme 2007-2013 and the Hungary - Serbia IPA Cross-border Cooperation Programme 2007-2013

2.4.

Support facilities

The Consortium and its member companies dispose of experienced staff assuring the necessary flexibility for assisting and back-stopping the project implementation at administrative level: daily project administration, facilitation of external and internal communication including liaising with Programme stakeholders and other organisations involved, preparation and administration of meetings, interviews, workshops, appropriate filing and documentation of materials received and produced at the various phases of implementation. All staff members have an excellent command of English and one of the languages of the CBC Programme.

The companies forming the Consortium offer comfortable and state of the art office facilities with free office spaces and workstations, several meeting/workshop rooms and high quality technical working facilities (laptops, projectors) for the implementation of the project. In terms of location, both Hungarian consortium members are situated in the centre of Budapest, while InTER and Karzen i Karzen have similarly fine office infrastructures in Belgrade and Zagreb respectively. Furthermore the team can also make use of the office infrastructure provided by the short-term experts. These facilities located in the heart of the border region (Szeged, PĂŠcs) are available for all kinds of field-related project events as well.

2.5.

Sub-contracting arrangements foreseen

InTER and Karzen i Karzen Sub-contracting arrangements will aim at bringing in their particular expertise and area knowledge regarding Croatia and Serbia, firstly through providing complex professional support activities (collection and processing of data, carrying out field research, preparing and moderating meetings, workshops, interviews) and secondly by providing practical support for the implementation (office infrastructure, organisation, mobilisation, public involvement, effective communication and dissemination in their respective countries).

As an additional contribution, one of the key experts proposed for this project (Zoran Krtinic) has been delegated by InTER.

37

Organisation and Methodology for the On-going Programme Evaluation of the Hungary - Croatia IPA Cross-border Cooperation Programme 2007-2013 and the Hungary - Serbia IPA Cross-border Cooperation Programme 2007-2013

3.

Timetable of Activities

3.1.

Timing of activities

The timing, sequence and duration of the proposed activities, taking into account the mobilisation time is presented in the table below:

38

Organisation and Methodology for the On-going Programme Evaluation of the Hungary - Croatia IPA Cross-border Cooperation Programme 2007-2013 and the Hungary - Serbia IPA Cross-border Cooperation Programme 2007-2013 Nr. Of Months 0 Calendar Years 2013 Calendar Months 3

Topics & Reports

1 2013 4

2 2013 5

3 2013 6

4 2013 7

5 2013 8

6 2013 9

7 2013 10

8 2013 11

9 2013 12

10 2014 1

11 2014 2

12 2014 3

13 2014 4

14 2014 5

15 2014 6

16 2014 7

17 2014 8

18 2014 9

19 2014 10

20 2014 11

Topics 1.1. Examination of programme relevance 1. Direct comparison of the Programme objectives and results so far 1.2. Comparison of objectives and results

2. Assessment of the project selection procedure and criteria

2.1. Evaluation of application procedure 2.3. Evaluation of assessment procedure 3.1. Project based overview on programme performance

3. Evaluation of programme performance by indicators

3.2. Indicator based evaluation of programme performance 3.3. Assess current Indicator system

4. Revision of the Programme Communication activities based on the Communication Plan

4.1. Overall assessment of effectiveness of Programme level communication 4.2. Assessment of visibility of the programme

5. Contribution to European Policies: Analysis of the programme 5.1. Analysis of Contribution to European Policies contribution to key European and macro-regional policies 6. Future: analysis and evaluation of 6.1. Strategic assessment of programmes and implementation programme deliveries and results in order to provide recommendations 6.2. Country-specific analysis of prog. deliveries and results

7. Case Studies and Best Practices

7.1. Collection of Best Practices, Compilation of Case Studies

8. Comprehensive Study on programme operation and benefits for Croatia and Serbia in terms of preparing for SF

8.1. Comprehensive Study on programme operation and benefits for Croatia and Serbia in terms of preparing for Structural Funds

Reports Inception Report First Evaluation Report Second Evaluation Report Final Report Legend Hungary - Croatia Programme main activities

Hungary - Croatia Programme review

Indicative timing for meetings, workshops, expert panels, interviews, etc.

Hungary - Serbia Programme main activities

Hungary - Serbia Programme review

Submission of drafts and reports

39

21 2014 12

22 2015 1

Organisation and Methodology for the On-going Programme Evaluation of the Hungary - Croatia IPA Cross-border Cooperation Programme 2007-2013 and the Hungary - Serbia IPA Cross-border Cooperation Programme 2007-2013

3.2.

Work plan

The following work plan gives a detailed overview of the resources required for the implementation of project activities. Indicative expert days are calculated for each key expert and for short-term experts.

40

Organisation and Methodology for the On-going Programme Evaluation of the Hungary - Croatia IPA Cross-border Cooperation Programme 2007-2013 and the Hungary - Serbia IPA Cross-border Cooperation Programme 2007-2013

41

Organisation and Methodology for the On-going Programme Evaluation of the Hungary - Croatia IPA Cross-border Cooperation Programme 2007-2013 and the Hungary - Serbia IPA Cross-border Cooperation Programme 2007-2013

4.

Log Frame

Overall objective

Objectively verifiable indicators

Sources of Verification

Assumptions

(Subsequent evaluation reports)

Remaining good neighbourhood relations between the relevant countries

Sources of Verification

Assumptions

Number of approved projects; number of agreed partnerships; Improved quality of current and future cross-border cooperation programmes between Hungary and Croatia / between Hungary and Serbia

Purposes To draft an independent assessment of the on-going performance of the Programmes and projects with regard to the relevance, efficiency, effectiveness, impact and sustainability.

Improvement of the implementation process of the Programmes.

quantity of allocated money; Quantity of funds allocated, etc. (to be verified by the expost evaluation and evaluations in the 2014– 2020 period) Objectively verifiable indicators

Inception Report Reports approved

First Evaluation Report

Contracting Authority prepared for future evaluation exercises

Second Evaluation Report Final Evaluation Report

Availability of the Contracting Authority’s experts

Sources of Verification

Assumptions

Inception Report First Evaluation Report

Timely availability and acceptable quality of required inputs;

Second Evaluation Report

Availability of stakeholders’ groups;

Final Evaluation Report

No substantial changes in the planned

To coach and prepare the experts of the Contracting Authority during the first evaluation exercise for future programmes.

Results Programmes’ and projects’ progress reviewed; Relevance of the Programmes and projects checked; Efficiency and impact of the Programmes is measured;

Objectively verifiable indicators

Reports produced/developed Application of the recommendations.

Sustainability is assessed;

42

No substantial changes in the planned implementation of the Programmes

Organisation and Methodology for the On-going Programme Evaluation of the Hungary - Croatia IPA Cross-border Cooperation Programme 2007-2013 and the Hungary - Serbia IPA Cross-border Cooperation Programme 2007-2013

Recommendations for future interventions are made;

implementation of the Programmes;

Contracting Authority prepared for future evaluation exercises.

Readiness of the decision makers to discuss the strategic development of the evaluation process; Intermediate decisions made and in a timely manner by the decision makers.

Activities (as proposed by our Consortium)

Means

Expenses

Assumptions

Altogether 540 expert days

Availability of stakeholders‘ groups for discussion and clarifications

Expert’s analytical and creative work

As detailed in present offer’s activities, methodology and inputs and outputs matrix (see chapter no 2.2. Activities)

Organisation and facilitation of stakeholders interventions (interviews, workshops) Experts drafting textual proposals and finalising documents Experts presenting and discussing proposals at meetings

Preconditions Evaluation of the bids and selection of the Contractor by early March 2013

43