Final Evaluation Report May, 2012
By:
ATLANT KIS Project Evaluation SECTION ONE
Contents
1.- Introduction 1.1.- Structure of the Evaluation Report 1.2.- Evaluation objectives 1.3.- Evaluation constraints 2.- Contextualization 2.1.- Transnational Territorial Co-operation Programme - Atlantic Area: Interreg IV B 2.2.- ATLANT KIS Project: definition; scope; objectives and actions; target public and organisation and management structure. 3.- Methodological approach 3.1.- Types of Evaluation 3.2.- Evaluation criteria and key questions 3.3.- Gathering information tools 3.4.- Timeline and field research
Seminario Final - 27 de Marzo de 2012 PĂĄgina 2 de 134
Final Report – May 2012
ATLANT KIS Project Evaluation Contents (II)
SECTION TWO 4.- Evaluation o the ATLANT KIS Project’s Implementation 4.1.- Project's envisaged activities physical execution analysis 4.2.- Project’s management and co-ordination processes analysis 4.3.- Achievement analysis of the below criteria o 4.3.1.- Consistency and relevance
o 4.3.3.- Efficiency
o 4.3.2.- Effectiveness
o 4.3.4.- Comunication and Visibility
5.- Evaluation on the results achieved by ATLANT KIS project 5.1.- Achievement analysis of the below criteria o 5.1.3.- Sustainability
o 5.1.1.- Impact o 5.1.2.- Satisfaction /Expectations
Seminario Final - 27 de Marzo de 2012 Página 3 de 134
Final Report – May 2012
ATLANT KIS Project Evaluation Contents (III) 5.2.- The specific case of the region of Navarra
SECTION THREE 6.- Conclusions related to the criteria and issues raised in the evaluation. 7.- Answers to the Evaluation key questions 8.- Recommendations 9.- Appendix
Seminario Final - 27 de Marzo de 2012 Página 4 de 134
Final Report – May 2012
ATLANT KIS Project Evaluation
Section one
Seminario Final - 27 de Marzo de 2012 Página 5 de 134
Final Report – May 2012
ATLANT KIS Project Evaluation
1. Introduction
Seminario Final - 27 de Marzo de 2012 Página 6 de 134
Final Report – May 2012
ATLANT KIS Project Evaluation
1.1.- Structure of the Evaluation 1. Introduction Report
Seminario Final - 27 de Marzo de 2012 Página 7 de 134
Final Report – May 2012
ATLANT KIS Project Evaluation With the purpose of structuring the available information in a selective manner, this report is divided into three sections, detailed below: Section one: To bring into context, the Co-operation Programme - Atlantic Area: Interreg IV is previously analysed, more specifically, its geographical scope, description and development phases and the areas of planning and management. Then, the ATLANT KIS project is presented. Apart from that, this section explains the methodological approach used for the project evaluation as well as the job’s constrains. Section two: This section contains, on the one hand, an analysis of the processes and actions carried out to implement the ATLANT KIS project -establishing the grade of physical execution of the activities, and a management and coordination process analysis- and a study on the fulfilment criteria through the execution phase of the project. On the second hand, this section offers an assessment of the results reached and effects caused by the project in accordance with the objectives and expectations set.
Seminario Final - 27 de Marzo de 2012 Página 8 de 134
Final Report – May 2012
ATLANT KIS Project Evaluation
Section three: This section compiles the main conclusions reached after analysing each evaluation criteria (in both implementation and results areas), which make it possible to answer the two key questions posed in the evaluation. Finally, this part includes a series of recommendations to be considered by the involved partners, that may give cause for reflection and serve as a guide for future projects.
Seminario Final - 27 de Marzo de 2012 PĂĄgina 9 de 134
Final Report – May 2012
ATLANT KIS Project Evaluation
Introduction 1.2.-1. Evaluation objectives
Seminario Final - 27 de Marzo de 2012 Página 10 de 134
Final Report – May 2012
ATLANT KIS Project Evaluation
Overall Objectives: To evaluate the implementation of the activities and actions carried out from June 2009 to May 2012 on the ATLANT KIS project and give an overview of the main results achieved and the effects brought about for both internal and external beneficiaries.
Seminario Final - 27 de Marzo de 2012 PĂĄgina 11 de 134
Final Report – May 2012
ATLANT KIS Project Evaluation
Specific Objectives: • • • •
Monitor achievement of goals and objectives Describe the life cycle of the project throughout the reporting period Reflect on management ’s execution performance Provide useful information orientated towards decision-making for designing and executing of new projects and/or future actions.
Seminario Final - 27 de Marzo de 2012 Página 12 de 134
Final Report – May 2012
ATLANT KIS Project Evaluation
1.3.- Evaluation process 1. Introduction constrains
Seminario Final - 27 de Marzo de 2012 Página 13 de 134
Final Report – May 2012
ATLANT KIS Project Evaluation The non-existence of
a previous evaluation -Conceptual Evaluation or Design Evaluation- limits the
possibility to study the actual consistency and pertinence of the ATLANT KIS project, that is, its internal logic. This type of study is intended to analyse the excellence of the diagnosis executed prior to the initiation of the intervention, verify the existence of clearly defined and measurable objectives, analyse the connexion with the problems observed and finally, to examine the logic of the type of intervention designed, from both programme’s internal perspective and in comparison to other policies’ and programmes’ (benchmarking).
The evaluation process was assisted by the participation and cooperation of KIS suppliers from all the regions involved, with the exception of Galicia. A list containing KIS suppliers having participated in any action promoted by the partners in their territories was requested with the purpose of knowing their opinion, expectations, motivation and grade of satisfaction on their participation on the project and on its resultant tools and products. In the cases in which the list was not provided, the suppliers’ details were obtained among those suppliers registered in the KIS platform. Nevertheless, in the case of Galicia, no supplier from the region was registered in the platform. Therefore, they were unable to cooperate with the evaluation.
Seminario Final - 27 de Marzo de 2012 Página 14 de 134
Final Report – May 2012
ATLANT KIS Project Evaluation
Systematicity and information gathering: No unique data. As a result of the analysis carried out on information gathering for the evaluation, the evaluators consider that, despite the project coordinator’s efforts (Iniciativas Innovadoras) to elaborate a homogeneous procedure of data collection for all entities, this does not to cover all aspects of the global monitoring of the intervention (actions and participants), as it was noticed that neither all organisations collected the same information, nor the same fields were systematised; the manner in which the information was processed and the level of dissemination, also varied from one to another. That is, the mechanisms started do not guarantee, either homogenously or systematically, an appropriate monitoring of the physical execution and the impact of the project.
Evaluation process tight schedule >> This prevented the evaluation team from gathering all associated partners’ opinions, with the exception of those located in the region of Navarre.
Seminario Final - 27 de Marzo de 2012 Página 15 de 134
Final Report – May 2012
ATLANT KIS Project Evaluation
2. Contextualization
Seminario Final - 27 de Marzo de 2012 Página 16 de 134
Final Report – May 2012
ATLANT KIS Project Evaluation
2.1.- Transnational Territorial Co1. Introduction operation Programme - Atlantic Area: Interreg
Seminario Final - 27 de Marzo de 2012 Página 17 de 134
Final Report – May 2012
ATLANT KIS Project Evaluation Mean goal >> the attainment of significant and tangible advances in transnational cooperation with aim of furthering the
cohesive, sustainable and balanced territorial development of the Atlantic Area (with the exception of the regions of the Azores and Madeira and the Canary Islands, the Atlantic regions included in the territory are Spain, France, Ireland, Portugal and United Kingdom) and its maritime heritage.
The programme is financed through ERDF for the period 2007-2013.
The "Atlantic Area 2007-2013" programme is the fourth transnational cooperation programme in the area and seeks to: enhance the Atlantic maritime heritage; valorise the maritime resources of the Atlantic area; contribute to the emergence of new economic activity clusters; promote accessibility and logistic conditions; contribute to the balanced and sustainable development of the Atlantic area. Differences in comparison to previous programmes >> It focuses on concrete achievements, the exchange of experience as well as the transfer of know how and cross fertilisation between projects that address similar issues. Whereas, the previous programmesSeminario were limited Final to studies exchange information. - 27orde Marzoofde 2012 PĂĄgina 18 de 134
Final Report – May 2012
ATLANT KIS Project Evaluation The Operational Programme "Atlantic Area 2007-2013" is structured along the following priorities: PRIORITY
OBJECTIVES
1. Promote transnational entrepreneurial and innovation networks
• the development of knowledge transfers between companies and research centres; • the enhancement of competitiveness and innovation capacities in the maritime economy of the Atlantic area; • the stimulation of economic conversion and diversification by promoting regional potential.
2. Protect, secure and enhance the marine and coastal environment sustainably
• the improvement of maritime safety; • the sustainable management and protection of the resources of marine spaces; • the exploitation of the renewable energy potential of the marine and coastal environment of the Atlantic area; • the protection and promotion of natural spaces, water resources and coastal zones.
3. Improve accessibility and internal links
• the promotion of the interoperability and continuity of existing transport networks as well as sea, road, rail and air intermodality; • the promotion of short sea shipping (SSS) and cooperation between ports.
4. Promote transnational synergies in sustainable urban and regional development
• the pooling of resources and skills in the field of sustainable urban and rural development in the Atlantic area; • the increase of the influence of cities and regions and their attractiveness trough networking; • the conservation and promotion of the Atlantic cultural heritage.
5. Technical assistance
• Financial support is available for administration, monitoring, evaluation and control of the programme.
Seminario Final - 27 de Marzo de 2012 Página 19 de 134
Final Report – May 2012
ATLANT KIS Project Evaluation
1.ATLANT Introduction 2.2.KIS Project
Seminario Final - 27 de Marzo de 2012 Página 20 de 134
Final Report – May 2012
ATLANT KIS Project Evaluation ATLANTIC-KIS aims to provide an answer to the first priority of the Operational Programme (OP): : Promote transnational entrepreneurial and innovation networks; contributing thus to the “Cohesive development of the Knowledge Economy” challenge. Specifically the project falls under Objective 1.1. leading to the “Development of knowledge transfers between companies and research centres”.
Overall goal>> To enhance the Knowledge and Technology Transfer and innovation processes on SMEs through the promotion and co-operation of Knowledge Intensive Services (KIS) at the Atlantic Area.
Long term goal >> To contribute to the development of Clusters of KIS at the Atlantic Area, that might help identify the area as an excellent one in the supply of KIS.
Length of the project>> From 1 June 2009 to May 31 of 2012.
Seminario Final - 27 de Marzo de 2012 Página 21 de 134
Final Report – May 2012
ATLANT KIS Project Evaluation
Specific Objectives of the project: 1.
To design a methodology for the mapping of regional and transregional demand and supply of KIS, transferable to other regions.
2.
To develop and disseminate 7 Audits on KIS demand and supply from involved regions.
3.
To design a methodology for the identification and transfer of Best Practices at the regional policy level, enabling the promotion and dynamisation of KIS.
4.
To identify, exchange and disseminate to other EU regions Best Practices resulting from successful regional experiences on the field of KIS promotion and dynamisation.
5.
To transfer the Best Practices identified to the regional policies of involved regions, and to experiment new models and approaches based upon the latter, aimed at the promotion and dynamisation of KIS.
6.
To create stable communication links among KIS of involved regions, to foster their networking and cooperation by the development of a collaborative platform, open to all KIS from the Atlantic Area.
Seminario Final - 27 de Marzo de 2012 PĂĄgina 22 de 134
Final Report – May 2012
ATLANT KIS Project Evaluation
Geographical Scope >> The project involves 8 partners from 5 different countries (7 European regions) constituted as a Consortium: COUNTRY
INSTITUTION
REGION
Dirección General de Empresa e Innovación (Gobierno de Navarra)
Navarra
Consellería de Innovación e Industria (Xunta de Galicia)
Galicia
FRANCE
Bretagne Développement Innovation
Bretaña
PORTUGAL
Agência de Innovaçao
UNITED KINGDOM
Devon and Cornwall Business Council
Devon y Cornualles
South and Eastern Regional Assembly
Southern and Eastern
SPAIN
IRELAND
Westbic Borders, Midlands and West Regional Assembly
Border, Midland and Western
Seminario Final - 27 de Marzo de 2012 Página 23 de 134
Final Report – May 2012
ATLANT KIS Project Evaluation Work Plan (matrix of the project) Activities
Partner leader of the activity
1. Preparation Activities
Actions
Follow-up indicators
1.
Development of the project idea, taking into consideration the Programme. Identification and contact of relevant partners . Design and agreement of the proposal, following the Applicant's Handbook.
- 1 Project Proposal submitted. - Consortium of 8 partners agreed. - 8 Co-financing declarations agreed.
2. 3.
Design of management and co-ordination system and tools Project Co-ordination Quality Management
1 Partnership agreement signed. 1 Steering Committee 1 Technical Committee constituted. 1 Project Implementation Guide. 1 Intranet. 6 Consortium Meetings. 4 Progress Reports and 1 Final Report. 1 final external Evaluation Report.
1. 2. 3.
Definition of Methodology for Audit Development of Audit Dissemination of Audit
1 Methodology agreed on the definition of KIS, the scope of the study, and the outreach tools. 7 regional Mappings on KIS demand and supply. More than 500 KIS contacted. 1 Audit Report on 4 languages. Página 24 de 134 7 Dissemination Workshops: 300 part.
2. 3. 1.
2. Management and Co-ordination
3. Audit on KIS demand and supply on involved regions
Dirección General de Empresa e Innovación (Gobierno de Navarra)
Seminario Final - 27 de Marzo de 2012
Final Report – May 2012
ATLANT KIS Project Evaluation Work Plan (matrix of the Project) (II) Activities
Partner leader of the activity
Actions
Follow-up indicators
4. Benchmarking Study on Best Practices for KIS promotion
Bretagne Développement Innovation
1.
Good Practices Methodology definition 2. Identification and analysis of Good PracticesKIS Project Evaluation ATLANT 3. Exchange of Best Practices and development of new models 4. Selection of Good Practices to transfer 5. Dissemination of Best Practices and Lessons Learnt on experimentation
1 Methodology Guide for identification of Good Practices and Transfer check. 60 Good Practices identified. 1 Exchange Workshop: 30 part. 1 Guide on 4 languages. 7 Action Plans on GPs transfer. 7 Dissemination Workshops: 300 part.
5. Transfer of Best Practises and new models on the promotion of KIS
Consellería de Innovación e Industria (Xunta de Galicia)
1. 2.
7 Transfer Implementation Plans and Evaluation Reports. Good Practices effectively transferred: 20. Stakeholders involved: 500. 7 Seminars: 300 part. Nº of new clusters and networks : 3. Página 25level: de 13420. Nº of new links at transnational Final Report – May 2012
Transfer framework definition Transfer and new models implementation
Seminario Final - 27 de Marzo de 2012
ATLANT KIS Project Evaluation Work Plan (matrix of the Project) (III) Activities
Partner leader of the activity
Actions
Follow-up indicators
6. Atlantic Area KIS Platform
South and Eastern Regional Assembly
1.
Platform services definition and development. Platform launch and maintenance Transnational KIS Market Place event Platform Sustainability study
1 Virtual Platform in 4 languages. Visits to Platform: 10,000. Nº of registered users: 500. Nº of enquiries received: 100. 1 KIS Market Place: 150 part. Nº of networks: 25. Sustainability Scheme and Action Plan: 1
7. Promotion, Dissemination and Exploitation Activities
Agência de Innovaçao
1. 2. 3.
Communication Strategy and Plan Development of communication tools Implementation and monitoring of Communication Plan Final Dissemination Seminar
1 project website : 500 visits/month. 4 E-Newsletter: 1,000 recipients. 1 Project Brochure: 1,200 copies. 1 Final Seminar: 150 part. Articles published: 20. Presentation at other EU fora: 12. 10 Press Conferences: 100 media reached.
2. 3. 4.
4.
Seminario Final - 27 de Marzo de 2012 Página 26 de 134
Final Report – May 2012
ATLANT KIS Project Evaluation
Target audience
DIRECT BENEFICIARIES
Partners involved in the project KIS suppliers Associated partners involved in the project Knowledge and technology sources (universities, technological centers, higher education bodies…)
INDIRECT BENEFICIARIES
From public sector (PA) KIS demanders
From private sectors (SMEs, Chambers of Commerce, business associations)
Seminario Final - 27 de Marzo de 2012 Página 27 de 134
Final Report – May 2012
ATLANT KIS Project Evaluation Bodies for management and coordination of the Project
Management>> Lead partner of the project = DG Enterprise, Department of Innovation (Government Navarre)
Project coordinator>> external technical assistance (Iniciativas Innovadoras, S.L.)
TECHNICAL COMMITTEE (TC)
STEERING GROUP (SG) Composed by 1 representative from each partner . Tasks: to give strategic advice and guarantee the adequacy to the general objectives. 6 meetings in conjunction with the TC. Managing ERDF, Environmental and Equal Opportunities Authorities from host regions were invited to the SG.
Composed by all partners. Tasks: to agree on the contents, work plan and monitoring of the programmed activities; validation of outputs; assurance of the involvement of all partners in shared tasks, accomplishment of the objectives and goals and conflict resolution among partners. 6 meetings.
Seminario Final - 27 de Marzo de 2012 PĂĄgina 28 de 134
Final Report – May 2012
ATLANT KIS Project Evaluation
3. Methodological approach
Seminario Final - 27 de Marzo de 2012 Página 29 de 134
Final Report – May 2012
ATLANT KIS Project Evaluation
3.1.- Methodological Approach of 1. Introduction the evaluation: types of evaluation
Seminario Final - 27 de Marzo de 2012 Página 30 de 134
Final Report – May 2012
ATLANT KIS Project Evaluation Analysis Approach PROCESS (Tasks and Actions taken to achieve results)
2 different analysis approaches
=
RESULTS (outputs and outcomes)
=
EVALUATION OF IMPLEMENTATION
EVALUATION OF RESULTS
Seminario Final - 27 de Marzo de 2012 Página 31 de 134
Final Report – May 2012
ATLANT KIS Project Evaluation Methodological Approach of the evaluation Types of Evaluation •
Evaluator-based:>> external evaluation.
•
Content-based:>> evaluation of implementation and outcome evaluation.
•
Based on the time perspective:>> intermediate evaluation as it is conducted previous to the end date of the project: 31 May 2012 and ex post evaluation.
•
Purpose-based:>> formative evaluation –it is intended to be a tool which enables the evaluator to gather useful feedback to improve future actions (recommendations)- and summative evaluation -gives an overview of the tasks carried out while starting-up and implementing the project by the various agents involved, with a view to render a final and conclusive judgment on the results –products- achieved-.
Seminario Final - 27 de Marzo de 2012 Página 32 de 134
Final Report – May 2012
ATLANT KIS Project Evaluation
Some clarifications on the methodology approached The coincidence in time of submission of the project’s intermediate-evaluation report and the project’s scheduled finish date (May 2012), makes it truly difficult to asses the impact and effects reached, as a longer period of time would be necessary to determine if the changes and effects anticipated to result from the implementation of the intervention have made satisfactory progress.
Seminario Final - 27 de Marzo de 2012 Página 33 de 134
Final Report – May 2012
ATLANT KIS Project Evaluation
3.2.- Key Questions that this 1. Introduction EVALUATION aims to answer
Seminario Final - 27 de Marzo de 2012 Página 34 de 134
Final Report – May 2012
ATLANT KIS Project Evaluation
Concerning evaluation of implementation Has the project been developed in accordance with the management, execution and accessibility terms foreseen? Concerning evaluation of results Has the project produced a desired effect/changes on the project’s beneficiaries? What is the immediate impact?
The strategy-formulation tool used to carry out the evaluation is the EVALUATION MATRIX. The evaluation criteria have been based on the following key issues:
Seminario Final - 27 de Marzo de 2012 Página 35 de 134
Final Report – May 2012
ATLANT KIS Project Evaluation
CONSISTENCY: A measure which verifies appropriate connection between the project’s objectives and the strategic tools established. RELEVANCE: The extent to which the project's objectives and actions are pertinent in relation to the framework in which they are developed. EFFECTIVENESS: The extent to which the activities and indicators are fulfilled when analysing management and internal coordination processes and with no regard to inputs. EFFICIENCY: A measure of how economically resources/inputs (time) are converted to results.
Seminario Final - 27 de Marzo de 2012 Página 36 de 134
Final Report – May 2012
ATLANT KIS Project Evaluation
TRANSFERABILITY/VISIBILITY: The extent to which the project has been visible in those areas involved. Transferability assesses performance of communication actions for dissemination in order to determine if they have succeeded in transferring objectives, activities, good practices and results.
IMPACT: Term to describe the effects of the project. Impacts are the changes produced, which can be either foreseen, as the achievement of overall objectives, or unforeseen.
Seminario Final - 27 de Marzo de 2012 PĂĄgina 37 de 134
Final Report – May 2012
ATLANT KIS Project Evaluation
SATISFACTION: The acceptance level of all partners involved as well as both direct and indirect beneficiaries with regard to the project. It addresses various issues: drawing up and implementation processes and outcomes achieved.
SUSTAINABILITY: The extent to which the project's positive impacts can be expected to last after its termination
Additionally, the evaluation includes an assessment on management and monitoring procedures in the Atlankis Project.
Seminario Final - 27 de Marzo de 2012 PĂĄgina 38 de 134
Final Report – May 2012
ATLANT KIS Project Evaluation
1. Introduction 3.3.-Gathering 3.3.Information Tools
Seminario Final - 27 de Marzo de 2012 Página 39 de 134
Final Report – May 2012
ATLANT KIS Project Evaluation Objective >> Encouraging participation among partners and beneficiaries of the implemented actions,
in order to gather information on their
participation, experiences, assessment and level of satisfaction upon outcome being achieved.
Partners’ questionnaire
Associated partners’ questionnaire
KIS suppliers’ questionnaire
Seminario Final - 27 de Marzo de 2012 Página 40 de 134
Final Report – May 2012
ATLANT KIS Project Evaluation
3.4.-Timeline(evaluation phases 1. Introduction & work shedule)
Seminario Final - 27 de Marzo de 2012 Página 41 de 134
Final Report – May 2012
ATLANT KIS Project Evaluation • Phase I : Analysis of Project Information Requirements Primary Sources Analysis
Secondary Sources Analysis
• Project intranet
•Council Regulation (EC) No 1083/2006 of 11 July 2006
•Atlant KIS Website
•Transnational Cooperation Operational Programme "Atlantic Area 2007-2013"
• KIS platform (www.kis4smes.com)
• Spain’s National Strategic Reference Framework 20072013
• Project candidacy form
• A variety of studies and reports on KIS: •COTEC 2011 report • Technology Knowledge Intensive Based Services (TKIBS). A competitiveness tool for businesses and regions • “El uso de las PYMEs de servicios intensivos en conocimiento. Factores relacionados e implicaciones de política “.
• Reports to JTS (Joint Technical Secretariat) • Balanced Scorecard •Meeting Minutes • Outputs: Regional Diagnosis of Supply and Demand; BPs Identification Report; Several guides and PPT among others.
Seminario Final - 27 de Marzo de 2012 Página 42 de 134
Final Report – May 2012
ATLANT KIS Project Evaluation • Phase II: Matrix evaluation elaboration phase • Identification of both criteria and key questions in accordance with the envisaged time to perform the evaluation.-meeting with Iniciativas Innovadoras (lead partner-technical assistance) 22nd February 2012-. • Phase III: Planning of applicable tools for data collecting and field researching • Quantitative techniques Type
Sample
Participation
120
32
Partners’ questionnaire
8
8
Associated partners’ questionnaire
2
2
KIS suppliers’ questionnaire
Seminario Final - 27 de Marzo de 2012 Página 43 de 134
Final Report – May 2012
ATLANT KIS Project Evaluation • Phase IV: Preparing Outputs
• Exploitation and analysis of the information gathered through the surveys. • Drawing up and presentation of the evaluation outputs: • A report on preliminary conclusions. • A performance indicator report (performance-based and outcome-based). • A final evaluation report. • A database including compiled information from the questionnaires.
Seminario Final - 27 de Marzo de 2012 Página 44 de 134
Final Report – May 2012
ATLANT KIS Project Evaluation 2012
TIMELINE OF THE EVALUATION PROCESS
FEBRUARY
ACTIONS TAKEN (By week)
W4
W5
MARCH W1
W2
W3
APRIL W4
W1 W2
W3
MAY W4
W1
W2
W3
W4
PHASE I. Designing and validation Workshop (Discussion on the different approaches and criteria that will lead the methodology and more specifically both the evaluation matrix and the validation of the evaluation work plan). Sources analysis Tool designing for data collection
PHASE II. Execution: field research Surveying process Phone and mailing reminder
PHASE III. Evaluation ending Findings analysis Preliminary conclusions’ presentation Preliminary conclusions ‘ validation by partners
22nd March 27th March
Drawing up and rendering of evaluation outputs 2nd May
Final Evaluation Report ‘s presentation Draft validation by lead partner Drawing up and presentation of the finalSeminario evaluation Final report
- 27 de Marzo de 2012
28th May
ATLANT KIS Project Evaluation
Section two
Seminario Final - 27 de Marzo de 2012 Pรกgina 46 de 134
ATLANT KIS Project Evaluation
4. Evaluation of the ATLANT KIS Project’s Implementation
Seminario Final - 27 de Marzo de 2012 Página 47 de 134
ATLANT KIS Project Evaluation
4.1. Project's envisaged activities physical execution analysis
Seminario Final - 27 de Marzo de 2012 Pรกgina 48 de 134
ATLANT KIS Project Evaluation
Note: The data shown in this document were analysed and assessed in April 2012, thus the outcome obtained can be considered as partial and not definitive, as project completion is programmed for May 2012, when all planned activities are expected to be completed.
Seminario Final - 27 de Marzo de 2012 Pรกgina 49 de 134
ATLANT KIS Project Evaluation Degree of fulfillment of the project work plan (I) ACTIVITIES
ACTIONS 1. Development of the project idea
1. Preparation Activities
2. Partnership constitution 3. Design and agreement of the proposal 1. Design of management and co-ordination system and tools
2. Management and Co-ordination
2. Project Co-ordination 3. Quality Management
HIGH AVERAGE
X X X X X X X
4. Fulfillment of contractual obligations with Managing Authority 1. Definici贸n de la metodolog铆a del diagn贸stico 3. Audit on KIS demand and 2. Development of Audit supply on involved regions 3. Dissemination of Audit 1. Good Practices Methodology definition 4. Benchmarking Study on Best Practices for KIS promotion
2. Identification and analysis of Good Practices 3. Exchange of Best Practices and development of new models 4. Selection of Good Practices to transfer: 5. Dissemination of Best Practices and Lessons Learnt on experimentation
Seminario Final - 27 de Marzo de 2012
LOW
X X X X X X X X P谩gina 50 de 134
ATLANT KIS Project Evaluation Degree of fulfillment of the project work plan (II) ACTIVITIES
ACTIONS
5. Transfer of Best Practises 1. Transfer framework definition and new models on the 2. Transfer and new models implementation promotion of KIS
HIGH AVERAGE
X X X X
1. Platform services definition and development 2. Platform launch and maintenance:
6. Atlantic Area KIS Platform 3. Transnational KIS Market Place event
X X
4. Platform Sustainability study 1. Communication Strategy and Plan
7. Promotion, Dissemination and Exploitation Activities
X X
2. Development of communication tools
X
3. Implementation and monitoring of Communication Plan 4. Final Dissemination Seminar
LOW
X
Seminario Final - 27 de Marzo de 2012 Pรกgina 51 de 134
ATLANT KIS Project Evaluation
About the objectives achievement‌
Most of the partners consider the ATLANT KIS project to be effective. 87% of them consider the general objective achieved and give an overall score of 7.6 to the degree of achievement of the specific objectives.
This statement does not coincide with the perspective of the KIS suppliers>> (only 41% of KIS suppliers consider the general objective achieved).
Seminario Final - 27 de Marzo de 2012 PĂĄgina 52 de 134
ATLANT KIS Project Evaluation GRADE OF EXECUTION OF THE ATLANT KIS ACTIVITIES (May 2009 -April 2012)
72%
100% 80%
HIGH
60%
MEDIUM
24%
40%
4%
20%
LOW
0%
HIGH
MEDIUM
LOW
This appreciation about the effectiveness is reinforced by the high level of physical execution of the project, given that 81% of the monitoring markers have been fulfilled. Regarding the degree of execution of the actions, 72 % have been carried out at a high level of execution, 24% at a medium level and 4 % at a low level.
Seminario Final - 27 de Marzo de 2012 Pรกgina 53 de 134
ATLANT KIS Project Evaluation
4.2.- Management and monitoring process analysis
Seminario Final - 27 de Marzo de 2012 Pรกgina 54 de 134
ATLANT KIS Project Evaluation
Based on the obtained data, it is possible to state that the Lead Partner has performed a strategic and decisive rol during the project life cycle. The partners assess at an excellent level of fulfilment, the tasks relating to the Project Coordination and the Secretary tasks. The remaining assessed tasks (project planning, implementation and monitoring) obtain a good level of fulfilment. As for the aspects that could be improved, some partners claimed that the Lead partner should have had more control and resolution over those partners who didn’t fulfil their tasks and deadlines, taking into account that, in some activities, the organizations had to work in a parallel way, which has caused some delays, non-fulfilments and scarce outcomes in some actions. In any case, the great majority of partners answered affirmatively to this question: Would you use again, on a second stage of the project, the Lead partner services?
Seminario Final - 27 de Marzo de 2012 PĂĄgina 55 de 134
ATLANT KIS Project Evaluation
It can be said that, as part of the coordination tasks, appropriate tools have been created (intranet, templates, forms‌) for the correct technical and administrative project management. All of them have been positively assessed by partners. The use of these tools has been increased all along the project. More specifically, the use of the Intranet as a means of storing files and exchanging information. This tool was rather underused at the beginning of the project.
Seven Consortium meetings have been adequately held. Partnership is satisfied with both the number of meetings and the communication mechanisms used among partners (an overall score of 8 out of 10). Each Consortium meeting has been led by a host partner covering the meeting costs .
Seminario Final - 27 de Marzo de 2012 PĂĄgina 56 de 134
ATLANT KIS Project Evaluation Date
Place
Entidad anfitriona
July 2009
Pamplona (Spain)
Dirección General de Empresa e Innovación (Gobierno de Navarra)
March 2010
Rennes (France)
Bretagne Innovation
October 2010
Bruselas (Belgium)
February 2011
Santiago de Compostela (Spain)
May2011
Cork (Ireland)
South and Eastern Regional Assembly (SERA)
October 2011
Exeter (UK)
Devon and Cornwall Business Council (DCBC)
March 2012
Oporto (Portugal)
Dirección General de Empresa e Innovación (Gobierno de Navarra) y Fundación Galicia Europa Consellería de Innovación e Industria (Xunta de Galicia)
Agência de Innovaçao (ADI)
All efforts have been made to make coincide the great majority of meetings (Rennes, Brussels, Santiago, Cork and Exeter) with the most important workshops (brainstorming and discussion for GP selection; a regional transfer programme presentation …), and/or with project dissemination events (KIS market place held in Cork and Atlant-KIS Final Conference in Oporto). All of them have enabled partners to deal with the various issues raised while becoming the most important coordination and communication tools among partners and, at the same time, a good opportunity for partners to come together to network and exchange experience with projects beneficiaries (KIS suppliers, associated partners, and innovation centres among others).
Seminario Final - 27 de Marzo de 2012 Página 57 de 134
ATLANT KIS Project Evaluation
On the other hand, partners said that business meetings had served them to reach the necessary agreements for effective progress making.
INTERNAL PROJECT CONSTRAINTS … After analysing the answers given by partners to the questionnaire, we conclude that, overall, there are not significant internal constrains affecting project implementation. If anything, the more highlighted issues seem to be the fact that the ATLANT KIS is “too ambitious” (2.7 out of 5) and “partners’ lack of experience in projects of this nature.” (2.6 out of 5).
Seminario Final - 27 de Marzo de 2012 Página 58 de 134
ATLANT KIS Project Evaluation
4.3.- Achievement analysis of the below criteria
Seminario Final - 27 de Marzo de 2012 Pรกgina 59 de 134
ATLANT KIS Project Evaluation
4.3.1.- CONSISTENCY and RELEVANCE analysis
Seminario Final - 27 de Marzo de 2012 Pรกgina 60 de 134
ATLANT KIS Project Evaluation
Overall, the partners consider the project’s objectives to be clearly defined but perhaps, too ambitious. The planned measures in order to achieve the outcomes and objectives are adequate. However, a greater dedication and effort to some of them has caused unbalanced outcomes. The schedule has distributed the tasks in a coherent way, although some partners consider the number of activities to be excessive with regard to the scheduled execution periods. On the other hand, all partners agreed that the envisaged actions and activities meet the specific needs of their regions; however, the Lead partner highlights that due to the fact that the contents of the programme were defined prior to the regional diagnosis, there was little room for adjustment in accordance with the regional needs expressed in the aforementioned diagnosis.
Seminario Final - 27 de Marzo de 2012 PĂĄgina 61 de 134
ATLANT KIS Project Evaluation
Following what it was mentioned before and as set out in the chart, 87% of the partners assessed the ATLANT KIS project as “pertinent”, in the sense that it is considered to be valid and useful for the future.
PARTNERS ASSESMENT ON THE GRADE OF PERTINENCE OF THE ATLANT KIS PROJECT MUCHO
BASTANTE
SUFICIENTE
POCO
NADA
13% 37%
50%
The factors which encouraged partners to participate in the project were: Learn from other regions how to identify and develop pilot projects, through other EU regions GPs knowledge transfer and exchange, focused on the development and promotion of KIS to encourage SMEs innovation and growth. Gain knowledge on other innovation systems from the regions involved. Create regional and transregional networks.
Seminario Final - 27 de Marzo de 2012 Página 62 de 134
ATLANT KIS Project Evaluation
Compare the various regions’ situation with those having a similar economic and sectorial
structure. Enhance communication among different agents: SMEs and KIS. Own region knowledge sharing. KIS sector analysis and knowledge. These expectations are in line with the overall goal and the specific objectives of the project. However, no aspect related to the overall long-run goal is mentioned: “the development of Clusters of KIS in the Atlantic Area”, it is for this reason, that the goal set for this project may be considered overly ambitious.
Seminario Final - 27 de Marzo de 2012 Página 63 de 134
ATLANT KIS Project Evaluation As to the KIS suppliers’ answers … The project objectives are clearly understood and identified by all beneficiaries. Once the answers to the following questions have been analysed: “Overall, could you indicate what you have heard or you know about the ATLANT KIS Project? What are the project’s objectives?, the below conclusions have been rendered: • Out of the total KIS suppliers surveyed, 34% answered in line with the overall goal: enhance the Knowledge and Technology Transfer processes on SMEs through the promotion and co-operation of KIS. • 30% focused their answers on the specific objectives: They stated that the project aims to: create a virtual platform for KIS supply and demand matching, disseminate GPs in the KIS area, boost co-operation among SMEs to develop strategic interests for products and services; help companies’ growth through innovation and raise awareness of KIS demand and companies’ resources . • 16% identified the ATLANT KIS project as a European project joining 5 countries of the Atlantic Area. • Only one out of the all respondents highlighted the overall long-run goal : “development of Clusters of KIS in the Atlantic Area”.
Seminario Final - 27 de Marzo de 2012 Página 64 de 134
ATLANT KIS Project Evaluation
4.3.2.- EFFECTIVENESS analysis
Seminario Final - 27 de Marzo de 2012 Pรกgina 65 de 134
ATLANT KIS Project Evaluation PARTNERS ASSESMENT ON THE GRADE OF EFFICIENCY OF THE ATLANTKIS PROJECT
Out of all partners, 62% reported that the ATLANT KIS project is efficient, with 50% gauging that objectives had been “highly achieved” and the remaining 12% claiming that the objectives had been “achieved”.
38% 50%
A Cluster, made up of universities and firms from the
12%
medical technology industry in the region of BMW Assembly, has been created as a result of the ATLANT KIS project, The KIS market place celebrated in Cork along with
VERY MUCH
MUCH
SOME
LITTLE
NOTHING
the
development
of
the
virtual
platform
(kis4smes), have given rise to the creation of a new KIS Cluster at the transnational level.
Seminario Final - 27 de Marzo de 2012 Página 66 de 134
ATLANT KIS Project Evaluation Partners assesment on project’s achievement of overall goal and specific objectives Partners thinking the ATLANT KIS overall goal is achieved
ASSESMENT ON THE ATLANT KIS ACHIEVEMENT OF THE SPECIFIC OBJECTIVES
1
HALF EVALUATION (1 TO 10 SCALE) YES
To develop a KIS platform aimed at matching the supply and demand of KIS across the Atlantic Area.
6,6
NO
7
7,8
To develop and disseminate 7 Audits on KIS demand and supply among participants involved .
8,5
Out of all partners, only one reported that “it is too soon to gauge whether the overall goal has been achieved...”
Seminario Final - 27 de Marzo de 2012 Página 67 de 134
ATLANT KIS Project Evaluation
KIS suppliers assessment on project’s achievement of the overall goals Chart showing percentage of KIS suppliers thinking the ATLANT KIS overall goal is achieved
YES
41% NO
47%
NOT KNOWS/ NOT ANSWERS
12%
Seminario Final - 27 de Marzo de 2012 Página 68 de 134
ATLANT KIS Project Evaluation
About the objectives achievement‌ Most of the executant partners consider the ATLANT KIS project to be effective. 87% of them consider the general objective achieved and give an average score of 7.6 to the degree of achievement of the specific objectives. This statement does not coincide with the perspective of the KIS suppliers>> (only 41% of KIS suppliers consider the general objective achieved).
Seminario Final - 27 de Marzo de 2012 PĂĄgina 69 de 134
ATLANT KIS Project Evaluation
Added-value of the ATLANT KIS project for… Partners reported that the programme has brought them and KIS demanders, as direct beneficiaries of the project, the added-value described below:
PARTNERS
KIS DEMANDERS
•Gain Knowledge in local KIS supply and demand
• KIS platform (kis4smes)
• Transregional networking and experience exchange.
•Gain knowledge in the various types of KIS services offered in their regions and the benefits of using them.
•Research Vouchers Scheme •Enhance relationship between Policymakers on the innovation field and Public Agencies.
•Raise awareness of their KIS specific needs
•Pilot experiences/GPs testing.
Seminario Final - 27 de Marzo de 2012 Página 70 de 134
ATLANT KIS Project Evaluation Added-value of the ATLANT KIS project for… The ATLANT KIS added-value for KIS suppliers varies depending on the group surveyed:
KIS SUPPLIERS (main beneficiaries of the project) According to Partners •Services promotion through the KIS platform • KIS market place • Pilot tools for sector development.
•Gain knowledge in the most demanded KIS services • Possibility to be in contact with other suppliers and find new partners.
According to KIS Suppliers (18% of those surveyed stated that, the ATLANT KIS project has brought NOTHING into their company so far) •Raise awareness of the importance of KIS and the possibilities they offer. •Advertisement through the KIS platform • Gain knowledge in their company’s KIS needs • Possibility to gain new clients and increase revenue
• Network with the various agents and/or potential partners for global projects development. • Access to key service providers • Gain knowledge in the variety and volume of KIS services available in their region. • Financial support for SMEs growth
Seminario Final - 27 de Marzo de 2012 Página 71 de 134
ATLANT KIS Project Evaluation
4.3.3.- EFFICIENCY analysis
Seminario Final - 27 de Marzo de 2012 Pรกgina 72 de 134
ATLANT KIS Project Evaluation Valuation of the grade of efficiency of the project ATLANT KIS according to the partners VERY MUCH SOME NOTHING
MUCH LITTLE
Out of all partners, 63% gauged the project as “efficient”. Inputs to carry out the various actions were considered to be “average”. However, partners struggled to justify expenditure.
Grade of the partners' agreement regarding the following aspects of the efficiency of ATLANT KIS VERY SATISFIED
37% 63%
SATISFIED
AVERAGE
DISATISFIED
My organisation had no problems to economically justify the expenditure involved
3
There have been sufficient resources available to perform the actions in terms of human team
3
There have been sufficient resources available to perform the actions in terms of material
2
There have been sufficient resources available to perform the actions in terms of economy
2
VERY DISATISFIED
3
1 4
5
1 1 1
6
Seminario Final - 27 de Marzo de 2012 Página 73 de 134
ATLANT KIS Project Evaluation Timeline and deadlines It is noted that initial delays forced some partners to ask for up-to-six-month extensions. The communication activities were the most delayed ones due to -according to ADI, the partner responsible for the initiation of the activity- internal cost-cutting measures, which impeded personnel allocation. It is for this reason that at the last Consortium Meeting held in Oporto on 28th March 2012 it was agreed to outsource an associated partner (SEBIC) to accomplish this task. SEBIC will also try to follow indicators and fulfil the objectives set for the last stage of the project. Principal constraints>> reporting to the STC and Managing Authority was slower than expected caused by the online certification platform and the European Authorities’ late response. In some cases, partners’ performance has been unbalanced. This, has posed problems and delays in some particular actions that were programmed to start sequentially.
Seminario Final - 27 de Marzo de 2012 PĂĄgina 74 de 134
ATLANT KIS Project Evaluation
Project financial management The lack of flexibility of the computer application to validate expenditure led to continuous budget modifications adjusting the expenses to the corresponding budget headings established in the agreed budget. Consequently, three budget modifications were requested –the last one has been introduced along the current month, April 2012-. The first to modifications have mainly enabled performance of activity nº.5, (Transfer of Best Practices). STC flexibility to approve changes is positively assessed. However, it should be remarked that changes were made at a slow pace. Difficulties were encountered for project’s financial management co-ordination, they were caused by the different First Level Controls timeframe set up in each region.
Seminario Final - 27 de Marzo de 2012 Página 75 de 134
ATLANT KIS Project Evaluation
4.3.4.- COMMUNICATION AND VISIBILITY analysis
Seminario Final - 27 de Marzo de 2012 Pรกgina 76 de 134
ATLANT KIS Project Evaluation Analysing the answers given by the partners of the ATLANT KIS project… The activities related to project communication and dissemination carried out within the various regions have been truly unbalanced, no only in terms of the number of de actions put into place, but also in terms of coverage and reaching the target public. Whereas five partners believe that the actions carried out in their regions were “sufficient and clear” , three of them think that they were “insufficient” (BMW Assembly, SERA and ADI). The great majority of partners consider the grade of coverage of the project to be “acceptable”(with the exception of one of the partners involved who rated it “high”), however, they have pointed out that this is due to the goods produced, rather than the actual promotion and dissemination of the project. Seminario Final - 27 de Marzo de 2012 Página 77 de 134
ATLANT KIS Project Evaluation
Among all communication tools used in the project, the partners involved have experienced various visibility levels: Low rated tools: “Digital newsletter”, “Press conferences” and “Project’s website”
Average rated tools: “Articles in press”, “Project presentation in other EU forums “ and “Project’s brochure” Tools having more impact and visibility: “Seminars ”, “Workshops organised in the various regions participating in the project” and “KIS market place held in Cork”
Seminario Final - 27 de Marzo de 2012 Página 78 de 134
ATLANT KIS Project Evaluation
Conclusions to evaluation of the organisation responsible for project’s communication and dissemination . Overall, in reference to the tools developed along the project, they achieved an “acceptable” level of satisfaction (with the exception of one of the partners who thinks they are insufficient). It should be said that, detailed monitoring of the regional and transregional activities carried out was missing, as well as detailed information for the rest of the partners. Partners believe that external co-operation from professional organisations is needed. They also suggest that an organisation working on online promotion and dissemination (e.g. twitter, facebook) be considered for future interventions, this could be done throughout consortium, clusters...
Seminario Final - 27 de Marzo de 2012 Página 79 de 134
ATLANT KIS Project Evaluation Assessing answers from KIS suppliers involved in the ATLANTKIS project… Half of communication through which KIS suppliers met ATLANT KIS Direct contact from an projectinstitution 2% 5%
Workshop/regional forum
2% 2%
Project’s website 11%
More than 50% of the KIS suppliers surveyed stated that they had been able to join the ATLANT KIS project because of the dissemination job done and contacts made by the partners.
33% Attendance at the KIS Conference held in Cork Project’s platform (www.kis4smes.com)
11%
16%
18%
However, as set out in the chart on the left, the action of
gaining
beneficiaries
through
the
various
Project’s brochure
communication tools developed has been limited. Only
Project’s e-newsletter
6% of the respondents were aware of the project
Article in press
existence through brochures, digital newsletters, articles
Other
in press...
Seminario Final - 27 de Marzo de 2012 Página 80 de 134
ATLANT KIS Project Evaluation Assessing answers from KIS suppliers involved in the ATLANTKIS project… Grade of knowledge of the KIS platform (www.kis4smes.com) YES
It should be remarked the important dissemination job done by partners for KIS platform promotion, this is proved by the
3% NO
22%
fact that out of all KIS suppliers surveyed, 75% know of the platform existence.
NOT KNOWS / NOT ANSWERS
75%
Seminario Final - 27 de Marzo de 2012 Página 81 de 134
ATLANT KIS Project Evaluation
5. Evaluation on the results achieved by ATLANT KIS project
Seminario Final - 27 de Marzo de 2012 Pรกgina 82 de 134
ATLANT KIS Project Evaluation
5.1.- Achievement analysis of the below criteria
Seminario Final - 27 de Marzo de 2012 Pรกgina 83 de 134
ATLANT KIS Project Evaluation
5.1.1.- IMPACT analysis
Seminario Final - 27 de Marzo de 2012 Pรกgina 84 de 134
ATLANT KIS Project Evaluation Assesment on the ATLANT KIS impact according to partners VERY MUCH ENVIRONMENTAL IMPACT
MUCH
12,5%
12,5%
LITTLE
NOTHING
12,5%
12,5%
ECONOMIC IMPACT
25,0%
25,0% 37,5%
37,5%
NOT KNOWS/NOT ANSWERS 37,5%
37,5%
POLITIC IMPACT SOCIAL IMPACT
SOME
25,0% 25,0%
12,5%
12,5% 12,5%
12,5%
50,0%
According to the partnership, the ATLANT KIS project has made a “high” political impact, while the social impact is believed to be rather “moderate” (37.5% of the partners rated the impact “high” and 25% “average”). The environmental impact of the project is, overall, the lowest rated, as 62.5% of the partners rated it “very poor”. As shown on the chart above, there are different points of view as to the economic impact of the project is concerned (half of the partners stated that the project had “very high” or “high” economic
Final - 27 de Marzo de 2012 impact, while the other half Seminario rated it “average”).
Página 85 de 134
ATLANT KIS Project Evaluation ATLANT KIS EFFECTS As the project is not yet finished, it is technically too early to evaluate the ATLANT KIS effects in terms of impact. Therefore, the ATLANT KIS benefits must be assessed, in terms of efficiency, as outputs derived from project participation (gain KIS knowledge and its possibilities, make contacts and exchange experiences with other SMEs and entities; a chance for promotion in the industry). However, there are some possible effects that, according to the stakeholders surveyed (partners, KIS suppliers and associated partners) start to show:
Seminario Final - 27 de Marzo de 2012 Pรกgina 86 de 134
ATLANT KIS Project Evaluation ATLANT KIS effects for… PARTNERS •50 % of the partners said that after the ATLANT KIS project, new opportunities of exchanging ideas have arisen among them: • Galicia >> Cooperation among clusters while sharing synergy and taking as reference other projects, in which Galicia, Bretagne and Ireland have been involved, e.g. AT Clusters. • Attempts to hire teaching staff and experts from Exeter for BP exchange. • Agreement with Porto Business School to run the course “Innovation Managers”. • BMW Assembly >> Co-operation with DCBC •75 % of the partners stated that the ATLANT KIS project has made an impact on the creation of public policies and programmes in their regions: • DCBC >> Redefine their innovation strategy to support the growth of potential KIS companies and other industries under the experience of some pilot projects already implemented and transferred. •Navarra >> 4th Technology Plan in Navarra • Galicia >> The project have enabled this region to update the Innovation Managers Network of Galicia Programme set up in 2010 (Rede Xiga). A Dirección Xeral (The General Secretariat )is currently going through a transition period and, consequently, La Axencia de Innovación de Galicia (The Innovation Galician Agency) will be shortly implemented. One of its first missions will be the implementation of the “Registro de Agentes del Sistema Gallego de Innovación.” (“Agents of the Galician Innovation System Registry”). • ADI >> They expect to adopt some of the identified BPs in any future programmes.
Seminario Final - 27 de Marzo de 2012 Página 87 de 134
ATLANT KIS Project Evaluation ATLANT KIS effects for… KIS SUPPLIERS • The ATLANT KIS project has had no effect on the principal beneficiaries of the project in terms of business growth so far: • None KIS supplier has received any service request or query via the KIS platform. • None agreement has been signed between KIS demanders and suppliers after their registration in the platform. • There is only one supplier who stated that they had been able to contact other supplier as a result of their participation in the programme. • Some of the suppliers surveyed agreed that, as a consequence of their participation in the ATLANT KIS programme: • Awareness of the importance of these services has increased. • Their services have been widely promoted.
Seminario Final - 27 de Marzo de 2012 Página 88 de 134
ATLANT KIS Project Evaluation
5.1.2.- SATISFACTION analysis
Seminario Final - 27 de Marzo de 2012 Pรกgina 89 de 134
ATLANT KIS Project Evaluation
The partners involved are satisfied with the overall performance of the project. Yet, there are important issues, among the various aspects evaluated, that should be pointed out. Of the total number of items assessed, the greatest level of satisfaction was found at performance of coordination, organisation and management tasks, with an overall score of 8.2 out of 10. Partners are specially satisfied with the distribution of tasks and duties arranged to achieve project’s success, more specifically with the procedures and methods used for information gathering.
Seminario Final - 27 de Marzo de 2012 PĂĄgina 90 de 134
ATLANT KIS Project Evaluation
With another high score (7.8), partners expressed their level of satisfaction in reference to “Outcome and achievements reached after implementation of the actions and activities scheduled”, and “level of co-operation and experience exchange reached ” (7.3). “Communication
and
dissemination
strategies”
(5.6)
is
the
lowest
rated
item.
It should be said that there is no consensus among partners in regard to the communication and dissemination strategies defined for the project as 3 out of 8 of the partners involved rated it “poor” (scores ranking between 1 and 3 out of 10), 2 of them “good” and the remaining partners “very good ” or “excellent”.
Seminario Final - 27 de Marzo de 2012 Página 91 de 134
ATLANT KIS Project Evaluation
The final beneficiaries are highly satisfied with the activities promoted throughout the project. The overall score obtained concerning their satisfaction level is 7.2 with the items being rated -from highest to lowest- as follows: -Contactable institution in charge of the project in my region -The objectives and goals pursued by the project meet my business sector’s needs -Innovative nature of the activities -Length and timetable of the workshops and seminars run in your region -Accessibility to the information, material and products processed during the project -Opportunity to exchange experiences with other companies from my region
Seminario Final - 27 de Marzo de 2012 PĂĄgina 92 de 134
ATLANT KIS Project Evaluation
5.1.3.- SUSTAINABILITY analysis
Seminario Final - 27 de Marzo de 2012 Pรกgina 93 de 134
ATLANT KIS Project Evaluation The great majority of partners (five out of eight) thinks that the Atlankis project is sustainable and has a “high level “ of continuity. ATLANT KIS has produced outputs lasting beyond project life such as handbooks and methodologies, however, the Sustainability Study aimed in activity nº6, whereby continuity of the virtual KIS platform after project’s completion is to be raised, has not yet been conducted by partners. As regard to transferability, 50% of the partners believe that the ATLANT KIS project is “highly” transferable to other regions. PARTNERS’ ASSESMENT ON THE GRADE OF TRANSFERABILITY OF THE ATLANT KIS VERY MUCH
MUCH
12,5% 12,5%
SOME
LITTLE
NOTHING
25%
50%
Seminario Final - 27 de Marzo de 2012 Página 94 de 134
ATLANT KIS Project Evaluation
5.2.- The specific case of the region of Navarra
Seminario Final - 27 de Marzo de 2012 Pรกgina 95 de 134
ATLANT KIS Project Evaluation
The DG Enterprise and Department of Innovation from Navarra Government was mainly assisted by two entities acting as associated partners for the development of the ATLANT KIS project in the region: The European Business and Innovation Centre of Navarra (CEIN) and the Navarra Association of Consulting (ANEC).
Previous
experience:
While,
CEIN
had
participated
in
other
programmes
assigned
to
Interreg
(EURIS, Organza, ICT Value…) before the ATLANT KIS project, ANEC had no previous experience. Project involvement (specific actions): ANEC has been involved in the development, implementation and dissemination of activity nº 6: KIS transregional co-operation platform (www.kis4smes.com), by advising and acting as a link between other key entities and beneficiaries, as well as drawing up reports/studies and arranging events and workshops. More precisely, they have participated in the platform’s launching and maintenance activities related to act 2 by performing the following actions : They presented the use and operation of the platform at the KIS Regional Forum held on 8th February 2012 in Navarra Factori (CEIN), while encouraging the attendees to join it.
Seminario Final - 27 de Marzo de 2012 Página 96 de 134
ATLANT KIS Project Evaluation Promotion and dynamisation actions: ANEC has provided access to the platform through a link on ANEC’s website as well as on the main business and innovation portals: business portal “Navactiva”; Navarra’s innovation portal “Navarrainnova”; “Navarra.es”; business press “Nueva Gestión”. ANEC has participated in the Transregional Market place as well as in the platform sustainability study. They believe that the objectives related to these actions are accomplished as far as the platform facilitates communication among SMEs while helping internationalisation and online work and gives KIS suppliers visibility. Constraint >> attract KIS demanders to the platform In addition to this, ANEC has supported the activity nº 3 Audit on KIS demand and supply on involved
regions, by delivering two presentations in two different fora: • 18th May 2010. KIS Sector Navarra Discussion Forum: “Regional Diagnosis of Supply and Demand in the KIS sector in Navarra”, at CEIN’s premises •16th February 2011: KIS Sector Discussion Forum: “Transregional Diagnosis of the KIS Sector”, at CEIN’s premises.
Seminario Final - 27 de Marzo de 2012 Página 97 de 134
ATLANT KIS Project Evaluation
CEIN: CEIN has participated in activities nº 4 and nº 5, identification and implementation of GPs. More precisely, they have contributed to the GPs Guide and the implementation and transfer of the three GPs which were identified by the region of Navarra. The actual implementation status of these GPs is as follows: 1.
Business Accelerators (transferred by DCBC): It consists of a course to foster business growth and mentoring of a limited number or companies (6 companies). Run by the Instituto de empresa, this course offered high quality contents, speakers and mentoring network . The course comprised two phases: •A training programme: A five-day training programme was held (from 7th to 16th November) from 9 .00 to 18.30. Number of participants: 12 companies. • Mentoring: Out of the 12 participants, 6 were involved in this activity. They were selected following companies’ greater growth potential criteria. By means of an individualised meeting, each company and its mentor defined the mentoring process objectives, the working procedure and the milestones follow-up. This activity is aimed at: review the business plan; enhance marketing and commercial plans; open new distribution channels and guide companies throughout their internationalisation process. Mentoring was carried out from December 2011 to March 2012.
Seminario Final - 27 de Marzo de 2012 Página 98 de 134
ATLANT KIS Project Evaluation 2. Research Vouchers Scheme: They aim to promote cooperation between SMEs and knowledge centres to foster innovation and companies’ development throughout technology transfer. Firms can apply a basic research voucher for research service assistance to technological experts. The vouchers can cover advice and technical research. (They
provide up to €5,000). Voucher recipients assist companies with processes and procure product analysis to boost productivity by providing technical knowledge. Firms are obliged to put proposals into practice within three months from the study completion. A total of 5 firms participated in this scheme. This BP finishes in May 2012.
3. IDEAGEN: A structured brainstorming session allows participants to generate entrepreneurial ideas and also network within the larger group situation. They are led by innovation experts. Firms, technology centres, universities and entrepreneurs are brought together in a dynamic and interactive online forum. A session is planned for the 24th April 2012 and will focused on new software technologies. Experts from the Public University of Navarra will lead the activity and, it is expected that entrepreneurs working on new business ideas and willing to set their own businesses attend the session.
Seminario Final - 27 de Marzo de 2012 Página 99 de 134
ATLANT KIS Project Evaluation Opinion of the associated partners from navarra about the added-value of the ATLANT KIS project for… ASSOCIATED PARTNERS
KIS SUPPLIERS (main beneficiaries of the project)
•Greater visibility as an entity representing the sector
• Closer links between SMEs, Technology centres and Universities.
• Gain knowledge on european projects
Identity ideas through three creative challenges: • What can be done to jointly create an EXPERT NETWORK of use to companies? • What can be done to learn more about TCs and Universities’ know-how and thereby help business innovation? • What role can our company play to CONNECT ideas and entrepreneurs?
• Network with other entities and regions • Broader knowledge of KIS supply and demand in the region. • Broader knowledge of KIS research centres and teams •Learn from the implemented BP
• A growing awareness of the importance of the KIS sector
Seminario Final - 27 de Marzo de 2012 Página 100 de 134
ATLANT KIS Project Evaluation
Assesment on their participation on the ATLANT KIS project Both entities CEIN and ANEC rated the overall work carried out by the entity they have cooperated with with a 7.3 on a scale of 10, being the more valued aspect the “level of involvement of the partner in the project” (8.5), followed by the “level of communication with the partner” and the “coordinating activities carried out by the partner” (both items were rated with 8).
The activity planning in which they have been involved is highly rated (7.2 out of 10), being the more valued aspects: the “activity planning”; “activity designing: development and contents”; “the adequacy of inputs to goal achievements” and the “time scale and envisaged planning adaptation” (all items were rated with 8 out of 10).
Seminario Final - 27 de Marzo de 2012 Página 101 de 134
ATLANT KIS Project Evaluation Regarding the overall assesment of associated partner on the ATLANT KIS project… 7
Opportunity to exchange experiences with other with other knowledge centres… 8
Direct contact with the institution in charge of the project in my region. 8,5
Usefulness of guides and other material produced throughout the project. 7
Project advertisement 6,5
Accessibility to the information, material and products processed he project. 7,5
Length and throughout timetable of the workshops and seminars run in your region 8
Planning and arrangement of the activities in your company has been involved. 8
Innovative nature of the activities. 6,5
The objectives and goals pursued by the project meet my KIS business sector’s… 6,5 0
2
4
6
8
10
The associated partners from Navarra rate the ATLANT KIS project with an overall score of 7.3 out of 10. Seminario Final - 27 de Marzo de 2012 Página 102 de 134
ATLANT KIS Project Evaluation ATLANT KIS Strengths and areas of improvement according to the associated partners from Navarra... STRENGTHS
AREAS OF IMPROVEMENT
• The project has raised awareness of a sector which was unknown for the great majority of agents and enterprises in Navarra.
• To enhance innovation and creativity for workshops and seminars.
• It has enhanced estate promotion of the KIS companies, this sector did not receive public support up to now.
• Delays in some activities.
• The project is well structured and planned.
• Low level of attraction of KIS demanders.
• The dissemination of KIS has made it possible to bring them to the public and acknowledge that they are services related to innovation and knowledge.
• To boost dissemination and promotion of the project. The communication process all along the project has been slow . The web page has not been developed until the project was well under way.
• The synergy created among participants (entities and companies).
• There should have been more meetings for KIS companies fostering networking and joint projects working, among others, not only in transatlantic areas but also in the region of Navarra.
Seminario Final - 27 de Marzo de 2012 Página 103 de 134
ATLANT KIS Project Evaluation
Regarding the impact of the ATLANT KIS in Navarra‌ The DG Enterprise and Department of Innovation of
Navarra Government, states that the
ATLANT KIS project has made an impact in public policies within the Navarra area, more specifically in the development of the 4th Technology Plan of Navarra.
Seminario Final - 27 de Marzo de 2012 PĂĄgina 104 de 134
ATLANT KIS Project Evaluation
Section Three
Seminario Final - 27 de Marzo de 2012 Pรกgina 105 de 134
ATLANT KIS Project Evaluation
6.- Conclusions related to the criteria and issues raised in the evaluation.
Seminario Final - 27 de Marzo de 2012 Pรกgina 106 de 134
ATLANT KIS Project Evaluation
PROJECT MANAGEMENT and COORDINATION
Seminario Final - 27 de Marzo de 2012 Pรกgina 107 de 134
ATLANT KIS Project Evaluation
The assessment team considers the coordinating organization to have made an important effort in order to develop a homogenized procedure for all the organizations of information collection (monitoring system).
Nevertheless, this should be revised for future interventions since, despite fulfilling the minimum information requirements, it does not allow to cover all the aspects relating to a comprehensive monitoring of the intervention (actions and participants), as well as not incorporating the gender perspective.
Seminario Final - 27 de Marzo de 2012 Pรกgina 108 de 134
ATLANT KIS Project Evaluation With regard to the indicators, which are understood as project advance and progress measures, it can be confirmed that their level of suitability is rather one of a general type (physical and financial execution). That is to say, quantitative indicators of fulfilment have been envisaged but no qualitative indicator has been designed. It can also be said that two different tools have been observed to asses the degree of fulfilment of indicators for information gathering: the Balanced Scored Card and the indicator suite of the project’s application form. This has given rise to misunderstandings related to the project’s performance and results, therefore, the measurement and knowledge of the project effects in the varied territories have been limited. On the other hand, it does no exist a global criterium which allows partners to understand the indicators set, this has led to important differences in information gathering. Seminario Final - 27 de Marzo de 2012 Pågina 109 de 134
ATLANT KIS Project Evaluation
EFFECTIVENESS
Seminario Final - 27 de Marzo de 2012 Pรกgina 110 de 134
ATLANT KIS Project Evaluation
The assessment team states that when all the data has been collected is the moment in which the results of the project start to be visible. Up to the date of handing in this report, the degree of advance and progress regarding the physical execution of the measures is “adequate”. In general, the degree of achievement is acceptable. However, if we analyse and compare each of the activities in the project, significant differences in advance and progress have become evident, being even clearer in some of the regions.
Seminario Final - 27 de Marzo de 2012 Página 111 de 134
ATLANT KIS Project Evaluation
EFFICIENCY
Seminario Final - 27 de Marzo de 2012 Pรกgina 112 de 134
ATLANT KIS Project Evaluation After analysing the aspects in which this criterium is specified, the assessment team, considers the AtlantKis project to be efficient and well-measured with regard to the fact that the available resources have been enough to carry out the scheduled activities. It is not possible to make the same value judgment about the pace of project development since some delays in the initial stage caused some of the expected products not to be finished at the time of the final seminar (especially, those related to the activity no 7: communication and dissemination). The ineffectiveness of the administrative systems of account submission and accountability is confirmed as the main reason for the mismatches which have been detected in the project and the unbalanced performance of the partners. Seminario Final - 27 de Marzo de 2012 Pรกgina 113 de 134
ATLANT KIS Project Evaluation
CONSISTENCY and RELEVANCE
Seminario Final - 27 de Marzo de 2012 Pรกgina 114 de 134
ATLANT KIS Project Evaluation
The assessment team confirms that the good coordination of project objectives, scheduled actions and expected outcome is adequate, to a large extent, for the specific context and needs of the involved regions, not only from the partners’ perspective but also from the KIS suppliers’.
However, in a higher level of planning there are certain doubts whether the Atlankis performance strategy will enable the accomplishment of the overall goal in the long run: “contribute to the development of Clusters of KIS in the Atlantic Area, which will help to ensure that the area is recognised for its ecellence in the field of KIS”. It seems that, the way this goal is stated, the key question fitting into this criterium can not be answered: Which visible or measurable outcome will show that the project’s goal is accomplished? Seminario Final - 27 de Marzo de 2012 Página 115 de 134
ATLANT KIS Project Evaluation
SATISFACTION and EXPECTATIONS
Seminario Final - 27 de Marzo de 2012 Pรกgina 116 de 134
ATLANT KIS Project Evaluation
The evaluation team states that, the partners along with the beneficiaries and the associated partners assessed a “high level� of satisfaction as to management, performance and results achieved is concerned.
It became clear that the goods produced by the project have met their expectations, although some of them (especially the KIS platform) will require that a strategy be developed to ensure their future continuance.
Seminario Final - 27 de Marzo de 2012 PĂĄgina 117 de 134
ATLANT KIS Project Evaluation
COMMUNICATION AND VISIBILITY
Seminario Final - 27 de Marzo de 2012 Pรกgina 118 de 134
ATLANT KIS Project Evaluation
From the evaluation team side, we can confirm that there is a lack of an actual Communication Plan being more effective and suited to the regional and inter-regional framework and in which partners specify: objectives, expected results, indicators, activities, reach and/or media foreseen to make the project visible. Various communication tools have been delineated as it was anticipated. Nevertheless, we think that they could have been re-defined to ensure accomplishment of the effects foreseen. Our view is that criteria should have been clearly stated in advance. That is, “what?” and “for what purpose?”, as well as “use” and “effects and impacts foreseen”, in both coverage and visibility areas .
Seminario Final - 27 de Marzo de 2012 Página 119 de 134
ATLANT KIS Project Evaluation
IMPACT
Seminario Final - 27 de Marzo de 2012 Pรกgina 120 de 134
ATLANT KIS Project Evaluation ATLANKIS PROJECT RESULTS AND EFFECTS PARTNERS 50 % of the partners involved stated that new opportunities to share experiences have arisen as a result of the ATLANT KIS project. 75 % of the partners involved stated that the ATLANT KIS project made an impact on the development of new schemes and policies in their region. KIS SUPPLIERS To date, the ATLANT KIS project has not caused any effect on the major beneficiaries of the project in terms of business development. It can be observed that , in regard to the impact and effects attributed to the project, they seem to be more noticeable within the framework of the development of new insights, tools and methodologies , as well as within experience exchange among partners.
Seminario Final - 27 de Marzo de 2012 Pรกgina 121 de 134
ATLANT KIS Project Evaluation The impact and effects derived from the implementation of the ATLANKIS project would mainly remain among the institutions (project partners), being less noticeable for the beneficiaries in each of their territories. Yet, most long-term effects/impacts, could only be verified several years after the end of the project. It will be necessary for 2/3 years to have passed to be able to confirm if the success factors, which are associated with the actions carried out along the project, have evolved as successfully as it was expected. It is for this reason that, the benefits attributed to the ATLANTKIS project should be considered, in terms of efficiency, as the resultant outputs (gaining greater KIS knowledge; networking and experience exchange opportunities among SMEs and entities; advertising ... ). At this stage of the project, it is not possible to come to a solid and reliable conclusion with regard to either the outcome reached in the long term or the sustainability of the results. Seminario Final - 27 de Marzo de 2012 Pรกgina 122 de 134
ATLANT KIS Project Evaluation
SUSTAINABILITY
Seminario Final - 27 de Marzo de 2012 Pรกgina 123 de 134
ATLANT KIS Project Evaluation
Having received feedback from partners, beneficiaries and associated partners, based on the answers obtained from surveys, the evaluation team can state that: The lack of inputs and funding represents the weakest point of the Atlankis project.
New cooperation formulas as well as new tools and methodologies which may contribute to innovative solutions beyond the life of the project (mainly guides and the KIS platform), along with knowledge transferability to different frameworks and a key sector within the Atlantic area, are the project’s biggest success factors. Seminario Final - 27 de Marzo de 2012 Pågina 124 de 134
ATLANT KIS Project Evaluation
7. Answers to the key questions
Seminario Final - 27 de Marzo de 2012 Pรกgina 125 de 134
ATLANT KIS Project Evaluation Concerning evaluation of implementation
Has the project been developed in accordance with the management, execution and accessibility terms foreseen? Answer:
The objectives and goals set have been largely achieved, however there have been different levels of involvement and performance among partners. In addition to this, different responses concerning the grade of execution of the envisaged activities have been observed. On the other hand, variations in the budget and schedule have forced partners to excessive dedication to paperwork and setting-up time. This fact has significantly limited the top performance of certain activities, from the physical and financial execution perspective.
Seminario Final - 27 de Marzo de 2012 Pรกgina 126 de 134
ATLANT KIS Project Evaluation Concerning outcome evaluation
Has the project or programme produced a desired effect/changes on the project’s beneficiaries? What is the immediate impact? Answer:
The impact and effects that the implementation of the Atlant Kis Project has produced on stakeholders are more noticeable among institutions (partners involved), the effects have been less noticeable, in the short term, among KIS suppliers (as direct beneficiaries) in each of their territories. As for KIS demanders, we do not have any information about their opinion on the level of influence that the project has exercised over them.
Seminario Final - 27 de Marzo de 2012 PĂĄgina 127 de 134
ATLANT KIS Project Evaluation CONCLUSIONS OVERVIEW LEVEL OF ACHIEVEMENT OF THE EVALUATION CRITERIA SYMBOL
ANALYSIS APPROACH
CRITERIA (KEY ISSUES) CONSISTENCY RELEVANCE
Evaluation of Implementation
EFFECTIVENESS EFFICIENCY TRANSFERABILITY/ VISIBILITY IMPACT
Outcome Evaluation
ACHIEVEMENT /SCOPE
SATISFACTION Seminario Final - 27 de Marzo de 2012 SUSTAINABILITY
MEANING
Satisfactory level of achievement and scope of the evaluation criteria
There are some aspects that could be improved
This criterion requires immediate attention, as there are various threats that may put the achievement of the foreseen objectives, results and/or effects at risk Página 128 de 134
ATLANT KIS Project Evaluation
8. Recommendations
Seminario Final - 27 de Marzo de 2012 Pรกgina 129 de 134
ATLANT KIS Project Evaluation In general terms: To carry out a design assessment of any future action in order to guarantee that the objectives and strategy are COHERENT AND APPROPRIATE in the context(s) in which they are developed. Likewise, this is going to allow to make the appropriate modifications and reorientations, thus the achievement of the expected outcomes and their contribution to the objectives and specific characteristics of the Community Initiative INTERREG IV Atlantic Space will be of a greater scope and adjustment. Cooperation and experience learning among different territories with similar geographical characteristics are possible and generate good results. With reference to the information, tracking and assessment systems, to improve observation tools (results and impact indicators) and data collection techniques (quantitative and qualitative) in order to get them to be more suitable and appropriate to answer the key questions, taking into account both our own information availability and others’. It will be necessary to design them from the beginning of the program and as broken down as possible, taking into account the principle of equal opportunities.
Seminario Final - 27 de Marzo de 2012 PĂĄgina 130 de 134
ATLANT KIS Project Evaluation In specific terms: To compile a TERMS GLOSSARY at the beginning of the project, with the aim of making the main and most common concepts which are used in the project easier to understand and favoring the usage of a common language among partners of different nationalities. In the project draft process, to spend more time on a detailed definition of objectives, activities (including feasibility and sustainability) and budget, with the aim of avoiding subsequent modification and rescheduling requests which slow down the start-up of the project. To improve the administrative processes of management and certification in order to ease and speed up the report and justification tasks. Put more effort into the project communication and spreading actions in order to ensure a wider visibility of the project, from a coverage and impact criterium. (the success in the design of powerful working tools such as the KIS platform is blurred because of its scarce use and impact in terms of visibility).
Seminario Final - 27 de Marzo de 2012 Pรกgina 131 de 134
ATLANT KIS Project Evaluation
For similar projects to ATLANT KIS, in which sequential working activities are proposed, it is necessary that mechanisms be established so as to ensure that deadlines are strictly met and a backlog is not produced for the rest of the activities . In addition to this, it is necessary that, when monitoring and managing, a certain timeline flexibility be foreseen for the tasks to be carried out in order to have other alternatives available as well as to be able to move forward in the event that the project has not yield any results in its first stages. Consortium meetings should represent the main partnership meeting point, however, this should not be the only means of communication and cooperation among partners. The added-value of co-operational projects should be displayed, by all or some partners, in a greater number of communication spaces, in which specific issues, related to project execution, be tackled in view to obtain a shared and global orientation. Special attention should be focused on time gap between meetings in order to pursue cooperative dynamics.
Seminario Final - 27 de Marzo de 2012 Pรกgina 132 de 134
ATLANT KIS Project Evaluation
Once the objective of performing a “Sustainability study” on the ATLANT KIS project is accomplished- this has determined the preservation of the KIS platform for at least a year after project’s completion- it is recommended that both the sustainability concept and scope of the project should be defined. In addition to this, a programme should be drawn up establishing objectives and key challenges and serving as a base for project’s sustainability, it should also include partners’ commitments. In addition to this, it will be necessary to establish some action lines and measures, and implement a management and feasibility model.
Seminario Final - 27 de Marzo de 2012 Página 133 de 134
ATLANT KIS Project Evaluation
9. Appendix
Seminario Final - 27 de Marzo de 2012 Pรกgina 134 de 134