QUALITY ASSURANCE
REPORT MAY 2016
This project has been funded with the support from the European Commission. This publication reflects the views only of the author, and the Commission cannot be held responsible for any use which may be made of the information contained therein.
Table of content Table of content ...................................................................................................................................... 1 Evaluation Methodology and Process ..................................................................................................... 3 Objectives and Scope of the Quality Assurance in the LLL-HUB project ............................................. 3 Methodology for the Evaluation of the LLL-HUB Project .................................................................... 4 Implementation of the Evaluation Process ......................................................................................... 6 Planning of the Evaluation activities ................................................................................................... 8 Evaluations Instruments ...................................................................................................................... 8 Evaluation deliverables ....................................................................................................................... 8 Management of Impacts, Value creation and Sustainability .............................................................. 9 Vision of the consortium partners ........................................................................................................ 10 Early vision from the 1st questionnaire ............................................................................................. 10 Intermediary vision (from the 2nd questionnaire) ............................................................................. 15 Advanced vision from the 3rd questionnaire ..................................................................................... 21 Interviews of coordination and WP leaders ...................................................................................... 25 Appreciation of the methodology ......................................................................................................... 26 Appreciation of the functioning of the Actors Networks ...................................................................... 27 Appreciation of the core WPs: LABS, FORUMS, AGORA ....................................................................... 28 Appreciation of the transversal WPs: Communication – Valorisation – Exploitation - Sustainability .. 28 Appreciation of the main deliverables .................................................................................................. 30 Appreciation of the AGORA from the part of external Stakeholders.................................................... 31 LLL-HUB Findings ................................................................................................................................... 32 An innovative architecture to explore the LLL policies in different regions and countries in Europe ........................................................................................................................................................... 32 The LLL- LABS ................................................................................................................................. 32 The LLL-FORUMS ........................................................................................................................... 33 The LLL-AGORA .............................................................................................................................. 33 Regarding "Lifelong Learning" ........................................................................................................... 33 The comparative report ................................................................................................................ 34 About recognising LLL as a specific domain needing specific structures and resources .............. 34 Diversity of the LLL Policies and practices in Europe and how to help them converge .................... 34 Outcomes of evaluation ........................................................................................................................ 35 Satisfaction ........................................................................................................................................ 35 Results ............................................................................................................................................... 35 The methodology .......................................................................................................................... 35 The LLL-HUB architecture .............................................................................................................. 36 The National State of Play and the Comparative Report .............................................................. 36 LLL-HUB
Evaluation Report (Final Version –May 2016)
1
The innovative practices and learners' stories .............................................................................. 36 Impact................................................................................................................................................ 36 Local level ...................................................................................................................................... 36 National level................................................................................................................................. 37 European level ............................................................................................................................... 37 Lessons learnt .................................................................................................................................... 37 The idea of Life Long Learning Pathways ...................................................................................... 38 How to achieve (another step towards) convergence of LLL policies in Europe .......................... 42 Appendices ............................................................................................................................................ 44 Appendix 1 – General Questionnaires................................................................................................... 44 Early vision......................................................................................................................................... 44 Intermediary vision ........................................................................................................................... 46 Advanced vision................................................................................................................................. 48 Appendix 2 – Meeting evaluation ......................................................................................................... 51 Kick-Off .............................................................................................................................................. 51 Lisbon ................................................................................................................................................ 52 Krakow ............................................................................................................................................... 54 Sofia ................................................................................................................................................... 57 Appendix 3 – LABS, FORUMS and AGORA evaluation ........................................................................... 61 LABS ................................................................................................................................................... 61 FORUMS ............................................................................................................................................ 61 AGORA ............................................................................................................................................... 61 Appendix 4 – Report from the external evaluation .............................................................................. 65 Appendix 5 - List of interesting practices collected by the LABS and used do illustrate the different documents ............................................................................................................................................. 86
LLL-HUB
Evaluation Report (Final Version –May 2016)
2
Evaluation Methodology and Process Objectives and Scope of the Quality Assurance in the LLL-HUB project The LLL-HUB project LLL-HUB has three identified objectives: Foster a shared meaning of lifelong learning: the project primarily seeks to create a transnational lifelong learning (LLL) community of understanding and values through comparative research and common spaces for dialogue. To encourage a mutual recognition of LLL peers is the first step to promote fruitful exchanges, better commitment and cooperation. Enable a cross-sectorial, multi-stakeholders cooperation: to build efficient synergies and see the big picture would be cost-effective inasmuch as complementarity between initiatives identified during the research phase will permit to avoid duplication thanks to a better knowledge and dissemination of European tools. Structure the first transnational policy learning based on a genuine research and dialogue at regional/national level on EU LLL strategies: the results of the EU Year of Citizens 2013 encourages us to rethink stakeholders’ adherence to European LLL strategies. The first step for their greater involvement is to give a voice to regional and national relevant stakeholders and empower them so that they can propose effective solutions and to convince policy-makers that their input is crucial. Based on these objectives, (FREREF, work package leader of the LLL LABS identified 3 functions for the LLL-HUB and the LLL-LABS in particular (see Research Methodology for details): In such a project, the idea of Quality is intended as "Quality assurance", in the sense that the project quality is built all along the project. The Quality (Assurance) Plan is the reference document. The quality procedures are embedded in the management processes. The quality evaluation process is implemented within the evaluation process of the project.
Quality Assurance vs Project management Quality Assurance is meant to support Project Management and enable a smooth unrolling of the project, as far as possible. Thus it provides with insights on how to (among other aspects) manage risks and possible conflicts; provide with deliverables and outcomes of outstanding quality; ensure the efficiency of stakeholders' management. The Quality Assurance Process includes a full internal and external Evaluation Process.
The Project Vademecum A simplified and synthetic document, called the project Vademecum, gathers all key information regarding the project. It is a valuable resource for partners, as it enables them to constantly refer to their activities, due dates and deliverables. The Vademecum is a reference document for all project participants.
Risks management The risk assessment is part of the quality assurance, and the risk management is part of the project management tasks. An initial risk assessment is presented in this document. A risk review must be done regularly. A more detailed chapter is devoted to risk management (chap. 7).
LLL-HUB
Evaluation Report (Final Version –May 2016)
3
Conflict management The management of conflicts is done at the closest range to the conflict itself, meaning that one tries to solve the problem where it occurs. For example, if conflict occurs between two individuals, it is the responsibility of these people to try and come to an agreement. A mediator may intervene. If the two people belong to the same partner organisation, it is then the responsibility of the partner to deal with it. If the conflict occurs within a WP, it is the responsibility of the WP leader to manage it. If the conflict cannot be solved at the "ground" level it might be escalated to one of the Committees, or to the Project Management.
Deliverables, Results and Outcomes Each deliverable, results or outcome (also called "products") is under the responsibility of its owner, who is generally the WP Leader of the WP from which the product is supposed to be issued. This responsibility may be delegated to a given partner or individual, with the agreement of the project management. Any kind of event that could jeopardise the production of a product must be reported as soon as possible to the WP Leader and if necessary to the project management, who will undertake the necessary actions to find a solution. The graphic standards for the deliverables and presentations are part of the graphic charter of the project. They are provided by the project management and must be carefully respected. They include the project logos, the Commission logo and disclaimer, and any other requirements stated by the project management. The due date of the products are gathered in a specific table in the Project Vademecum.
Stakeholders' management Stakeholders' management is one of the key tasks of the project management, but it also requires the commitment of all project members. The external stakeholders include the European Commission, the local, regional and national bodies in relation with the project, the target audiences, etc. They are carefully identified by each partner on their territories. The external stakeholders play a key role in appreciating the impact and value of the project products.
Methodology for the Evaluation of the LLL-HUB Project Evaluation is the process through which the participants are able to appreciate the project, its unrolling, its management, and its capacities to fulfils – and even overpass - its objectives, issue the required products, create the expected value and impacts. The evaluation process is the main pillar of the implementation of the Quality Assurance.
Nature of the Evaluation Evaluation enables to give an appreciation through the systematic and regular examination of the resources, realisations and products of the project vis-à-vis what was forecasted in the beginning of the project – for the most part what is stated in the candidacy and agreed by the commission. Thus it enables to investigate on a permanent basis the value created by the project. For example, it gathers, reports and enables to share information about the innovation potential, quality level, relevance of actions, and capability to sustain the impacts of the project. Evaluation is undertaken within an ambitious framework. Nevertheless, its actions are adjusted depending on the resources, the time constraints, the availability of information and people. The kind of evaluation implemented in the LLL-HUB project is mainly formative, meaning that its aim is to help all project participants to improve the project along its whole unrolling. It is neither an audit, nor a simple mirror of what is happening. LLL-HUB
Evaluation Report (Final Version –May 2016)
4
It is undertaken in a spirit of benevolence. It first evidences what is going well in the project in order to capitalise on achievements and potentials to work positively on improvements, according to the spirit of Appreciative Inquiry.
Evaluation principles as they are implemented in the LLL-HUB Project The Evaluation Methodology is designed according to principles emerging from Research works in the area and comprising four pillars: Action Research, Qualitative Evaluation, Appreciative Inquiry and Actor Network Theory.1. As it is implemented in the LLL-HUB project, the evaluation is:
qualitative
It uses qualitative evaluation methods together with quantitative information when they are available. Information is mainly collected through fours processes: open interviews, questionnaires, participative observations and analysis of documents.
systemic and longitudinal
It concerns the project as a whole system embedded in and interacting with its environment, as well as its different subsystems; it takes place all along the project, from beginning to end.
contextualised
It is tailored for the project. It takes into account the specific conditions of its unrolling. It takes place within the project works. Nevertheless, it provides means to overlook what is happening, to sit back and reflect in order to make better decisions, solve problems and provide continuous improvement in an efficient way.
enabling
It does not intend to standardise or impose sanctions in any way. It is meant to support, facilitate, optimise the project unrolling and maximise its impacts and value. It lets the participants realise the value of what they are doing. It enables the project management to better value and disseminate the project products, and thus promote them towards the external stakeholders.
participative
It requires the full participation of all project participants (and even possibly some external stakeholders when available) through the participative evaluation activities. It favours and sustains the dialogue between project actors, and enables them to express their concerns, interests and viewpoints. Its respects the fundamental rights of the persons, the state of the art discretion, and the specific rules established for the project (by making the data anonymous, for example). Evaluation takes place all along the project. It contributes to optimise the project by enabling participants to develop and sustain a common vision of the advancement of works. It gives a longitudinal feed-back thus enabling the most accurate reactivity for project management. Nevertheless, it is not possible to have a permanent process. The evaluation activities are limited and selected. They are reported periodically to the project participants, fostering debates and deliberation. They make an equal space to all participants' interests and viewpoints, meaning that everyone who wants to express themselves will be allowed to.
Goals and Outcomes of the Evaluation The main goals of evaluation are to: inform; Latour, B., (1999) On Recalling ANT, in Actor network Theory and After, John Law and John Hassard editors; Blackwell Publishing, Oxford, UK. Law, J., (1992) Notes on the Theory of the Actor-Network: ordering, strategy, and heterogeneity, Systems Practise, 5(4), pp379-393 1
LLL-HUB
Evaluation Report (Final Version –May 2016)
5
help in decision making; support communication between people; sustain motivation and participation; enable continuous improvement of the project quality; allow to measure what is produced, and what is their impacts; provide a view of the sustainability of the project products outside and after the project life.
It provides information, indicators and recommendations for the different kinds of project actors: project participants in general, project management, policy and decision makers, target audiences and field actors, and even project environment and external stakeholders. Thus, for example, evaluation will check that: the deliverables are relevant and suitable, the expected impacts well stated as well as the means to reach them; the project members feel in line with the project unrolling and productions, with an ongoing or renewed motivation; the actions undertaken are well aligned with the objectives, in order to reach or even overpass them; the project management gets all elements (beyond the project management tools, which are not part of the evaluation) to appreciate what is happening in the project, in order to keep a steady course in a positive atmosphere of mutual satisfaction; in the event of problems occurring, the responsible bodies are well notified and able to react quickly in the interest if the project.
Implementation of the Evaluation Process Evaluation follows a methodological process adapted to the LLL-HUB project. This includes: a set of principles based upon the results of research and practices stemming from field experience; a process structured in steps aligned with the project unrolling; a number of instruments enabling and supporting observation, data collection and analysis, appreciation of what is going on and reporting to partners; and specific resources as planned in the project. Evaluation implements the Quality Assurance Plan in a concrete way, through ensuring that the project is in the best condition to reach, and even overpass, its goals for the entire satisfaction of all stakeholders.
Internal and external evaluation in the LLL-HUB In LLL-HUB the evaluation is processed both at internal level (i.e. activities are performed by the partner in charge of evaluation) and at external level (i.e. the activities are performed by an external entity). This enables to develop a comprehensive view of what is happening with a twofold analysis point of view: from the internal standpoint, a well embedded process for working with systematic activities (questionnaires, evaluation of meetings, …) where it is easy to make data anonymous; from the external standpoint, a more distanced stance for dealing with person-to-person activities (interviews, participatory observations) which enables a more liberated speech from the participants' part.
LLL-HUB
Evaluation Report (Final Version –May 2016)
6
Limits of the Scope of Evaluation Taking into account the resources allocated by the project, evaluation has necessarily to limit its activities, while keeping its ambitions at a high level of standards. There are two kinds of limits: regarding the nature of activities, and regarding the quantity of activities. There is a clear frontier between project management and evaluation. Evaluation is in no way in position to make any decision of project management. In case of an existing problem evidenced by evaluation (i.e. by the project members' appreciation), evaluation might suggest ideas (again issued from the members' suggestions), and recall escalation procedures. But in the end the decision belongs to the project management instances. In the same spirit, the financial management of the project belongs exclusively to the project management. Nevertheless, evaluation might attract the attention of management about scheduling or resources issues. As for the quantity of activities, the limitations are the following: limitation of the participative observations: participation to the partners' meetings and possibly some field visits if interesting and/or necessary; limitations of questionnaires: three general questionnaires ("early vision", "steady vision" and "advanced vision") and one questionnaire for appreciation of each partners' meeting; limitation of the number of interviews: project coordination (3 interviews), WP leaders (at least one interview for each); the interviews might be done as possible during the plenary meetings in order to optimise the travel costs, or by audio conference; and limitations of the inspected documents to the deliverables. Criteria and Indicators for the Evaluation Elements that will be appreciated are the following: satisfaction of participants regarding the project unrolling; degree of completion; efficiency of actions; relevance of decisions; quality of the project management (problems and conflicts management, risk management, support to participants, facilitation, valorisation of the work done and efforts made, internal and external communication management, representation of the project towards thee external stakeholders, etc.) quality of the products; sustainability of the project productions; efficiency of impacts management: target audiences, institutions/organisms, decision and policy makers, at local, Regional, national and European level; relevance and efficiency of the dissemination and valorisation activities; and capability of the project to open perspectives for the future. In the kind of evaluation developed in the LLL-HUB project, the main criteria of evaluation is the perception by the project actors of the state of the project. The main indicator is thus the degree of appreciation of the persons gathered through the interviews and the questionnaires. In addition, some "objectives" criteria may be used to complete the appreciative ones, such as: quantity of tangible productions; appropriateness of deliverables productions and compliance with the planning; number of events (meetings, workshops, conferences, etc.); number of external publications, participation in research conferences, broadcasting shows, etc.; number of participants in the FORUMS and AGORA; LLL-HUB
Evaluation Report (Final Version –May 2016)
7
etc.
In every case, the criteria are defined collectively with the project participants and the actors implied in the actions.
Planning of the Evaluation activities The activities are linked up to the project ones in order to ensure a harmonious unrolling, to take the least amount of time to the participants, to facilitate (rather than disturb), and minimise the efforts and expenses. The scheduling of activities is available in the Evaluation Action Plan. The plenary meetings are key moments which rhythm the implementation of the project and thus become key control points regarding: risks reviews and actions to undertake if necessary; impacts management review and necessary actions; and valorisation and dissemination review and actions to be taken. It is recommended that each plenary meeting agenda gives explicitly time and space for these reviews and that the elements are written in the meeting accounts in order to be able to appreciate their implementation and follow up. The Piloting Committees enable to make decision in the short term, to prepare and debrief the plenary meetings and other projects activities, and to maintain the internal cohesion among partners.
Evaluations Instruments Participatory Observations Participatory Observation enable to associate the evaluators – both internal and external - to the project works, while leaving them their specific stance. They are important to have a good grasp of the project, its unrolling and distinctive atmosphere, of the actors' network and their driving interests. There is no bias to that, because the evaluation is appreciative and participative.
Questionnaires Questionnaires are based upon a set of open questions in order to gather the opinion and judgement of the persons answering upon the subjects in questions. They are more viewed as "substitute for interviews" (written interviews instead of administrated interviews) as it is not possible to interview each participant several times along the project.
Interviews Interviews are realised as often as possible during the plenary meetings. They might also be done using audio conference devices.
Documents The review of documents is limited mainly to the deliverables and to the documents that should be published outside of the project (external communication supports).
Evaluation deliverables Evaluation produces one contractual deliverable which is a final report at the project level. In addition, each questionnaire will form the basis to a presentation which will be commented and discussed in the plenary meeting to take place after the gathering of information, so that every project actor will be aware of how the project is appreciated by all members. If necessary, the project management and/or the WP leaders could be warned about specific issues regarding the whole project or one or other WP if the evaluation team thinks that there might be some risks occurring. LLL-HUB
Evaluation Report (Final Version –May 2016)
8
Management of Impacts, Value creation and Sustainability The management of impacts is a key aspect of the value creation within a European project like LLLHUB. An impact is not something which is recorded after the fact. It has to be built in, right from the beginning of the project. Thus it is the role of evaluation, together with project coordination, to sensitise project partners and to accompany them all along the project in order to have them put the maximum attention to the question of "inflicting" impacts on the field. All activities undertaken are possible levers to multiply, enhance and enlarge projects impacts. The issue of impact is important because it is the way to evidence the fact the project creates value for its stakeholders. The value creation is central; the impacts are the concrete materialisation of it. The management of impact is included in the evaluation management, in the sense that a review of impacts is included in each general questionnaire, and the interviews of project coordination and WP leaders. The management of impact is also closely related with the management of Dissemination (WP6) and Exploitation (WP7): the first general evaluation questionnaire enables to make explicit the forecasted impacts at all level and for each target audience. It also enables to share the possible means of actions necessary to reach the goals; the second general questionnaire enables to start making an inventory of the first recorded impacts, how encouraging they are, and how to continue developing the efforts; the last general questionnaire is a place to attract attention about how to continue the efforts after the project in order to sustain the impacts and create more value.
LLL-HUB
Evaluation Report (Final Version –May 2016)
9
Vision of the consortium partners Early vision from the 1st questionnaire
LLL-HUB project - Early vison of the partners The main words here are "National" and "LLL". So, at least the object of the project is clear (it is about LLL), and the first concern are the survey at national level.
KSF
Innovative character of the project Open approach Successful seminars/conferences/ publications on partners' level Final conference (must be successful) Respect of local/national authorities Visibility of the project (non-participants would like to be counted in) Internal and external communication Practical conclusions together with political recommendations Capacity of all partners to grasp the objectives and implement new working methods/practices in their region/country Involvement of National Stakeholders
Pitfalls
Diverse levels of concerns for the partners (e.g. "practical" vs "political") Changes … Not enough attention from public bodies (government, EU, …) Not enough participation (experts, surveys, forums, Agora…) Too "theoretical" approach Too diverse approaches of LLL
LLL-HUB
Evaluation Report (Final Version –May 2016)
10
Be able to align partners' interests to avoid unequal implementation Be careful to create a better ownership on the tasks Avoid too much top down communication Financial constraints
Risk management General risks Demotivation of partners
medium high
Coordinator will do his utmost to stimulate the partners regularly and ensure they are doing their tasks. In case of real problem the coordinator can refer to the bilateral contract signed with the partner
low
Necessity to withdraw from subcontracting research and looking for other solution was a bit time-consuming, so the research started later than it was planned, however the staff is doing their best to meet the deadlines
The inadequate experts involvement
medium high
To increase time flexibility, to expand the number of experts. To use tools for meetings
Misunderstanding of the goals and objectives of the project because of changing staff or the fact that some partners remain silent during the meetings or language issues
High
Partners should be briefed at each meeting about the expectations and deliverables. We shall make sure all partners can express their questions with sufficient follow-up.
No risks so far (5)
High
Lack of commitment of some partners WP6: Non participation of each partner
LLL-HUB
Project manager to ensure they deliver quality inputs and try to encourage them to participate more actively. high
high
Dissemination has an important concern with communication and we feel some frailty on this point
Evaluation Report (Final Version –May 2016)
11
Risk linked to a specific WP (tell which one) WP2: difficulty to coordinate and compare the research made by the different partners
Medium
Medium The coordinator and FREREF have elaborated a methodology enabling partners both to get ownership of the research topics and to enable a comparative analysis.
WP3: low participation in local forums
medium
Medium To use the EUCIS-LLL network to find people to assist
Unequal quality of the inputs of the different partners (LLL-LABS, WP2): contents, capacity to outreach other actors, etc.
medium
High
No risks so far (5)
National reports (LLL-LAB WP2) that focus on completely different issues: no coherence
Use examples of other partners in order to show what is expected, direct coaching for the partners who have problems understanding the instructions.
Make sure partners follow the methodology
Risk for your institution (or yourself) Difficulty to hand over the low concept of the project to the new project officer
high
Audrey (EUCIS-LLL) will be there to brief the new project coordinator and a transition period will be ensured.
No real impact of the project on the state of civil dialogue at EU level in education
high
high
EUCIS-LLL has enough levers of action to promote and exploit the project results and will try to do so to the various EU institutions, especially if a LLL intergroup is created in the Parliament.
low
medium
To inform project manager and to train the new members
No risks so far (5) Changes in the project team
LLL-HUB
Evaluation Report (Final Version –May 2016)
12
Local and Regional elections in Spain in 2015
low
low
No action taken
Change of staff: making sure the transition is smooth and we do not loose knowledge
medium
medium
We will recruit a person who is specialised in project management with a specialisation in education and training issues to avoid any problems.
Insufficiency of people to perform tasks
medium
medium
In this crisis APG is a risk position of having their personnel to act by contingency and a fragile position of liquidity (treasury).
As usual in many European projects, partners are not used to do a risk analysis, not to mention a risk management policy. Anyway, the generic risks are here.
Impacts management General impacts Political
LR
Better civil dialogue at EU level
N/A
Focus on exploitation of results
Political
LR
Better communication between actors at national/local level
N/A
Implementation of Forums – choice of strategic participants
SRLR
Raising awareness on ET2020
Involvement of stakeholders, dissemination activities, commitment from the partners
SRLR
Raising awareness on European affairs
Through a broad participation in the LLLFORUMS
SRLR
Bringing actors from different sectors together. In the long run possibility to set up cooperation mechanisms
EUCIS-LLL could help partners identify potential partners in the countries
SRLR
Involvement of stakeholders on E&T issues
Capacity building from the partners building on the results of the project
LLL-HUB
Evaluation Report (Final Version –May 2016)
13
Dissemination LR
HIGH
minimum Communication in action and partners involvement and commitment
Impacts on your Institution/Region Political
LR
Better position of EUCIS-LLL in dialogue with EU institutions Better knowledge on how to manage an EU project New project management skills
N/A
Technical
LR
Technical
LR
Research
MR- Better LR understanding on the obstacles/challeng es related to LLL strategies implementation MR- Evidence to show LR the importance of LLL at EU level as part of our advocacy actions MR- Building up the LR network with national activities taking place within this project MR Medium minimum
High level promotion of the project
Higher expertise from staff Knowledg Apply those competences in other context e of new (next job) communic ation tools, strategic planning‌ Active participation in national events
Exploitable results are to be sought
Capacity to involve them, using the European organisations member of the platform
Insufficiency of data from national authorities requiring more time consuming for APG team.
Impacts for yourself MRLR
LLL-HUB
Learning about actors, strategies and systems related to LLL
Further search on the topics to enrich what is learnt during the project
Evaluation Report (Final Version –May 2016)
14
Building relationships in the different countries for further initiatives Time HR management
High
medium
More time consuming for me considering the insufficiencies of APG team.
At this time of the project, only very general impacts are anticipated.
Intermediary vision (from the 2nd questionnaire)
LLL-HUB Project – Intermediary vision of the partners The project concerns are much richer than in the first questionnaires. The idea of "change" has made its way. Also there is a focus on the whole range of actors (who were in fact implied in the LLLForums): stakeholders, experts, etc.
LLL-HUB
Evaluation Report (Final Version –May 2016)
15
Keywords
LLL-HUB Keywords as proposed by partners Partners have well integrated the key issues of the Project.
What are you most proud of?
the FORUMS the experts
At this moment, all partners are putting a lot of efforts in recruiting exerts to participate in the FORUMS. Though the FORUMS, which are "only" meetings, have eventually preceded the field work of the LABS (as observatory, think tanks and relays), they evidence the need for such meeting and sharing activities, and their fruitfulness. The ability to "recruit" experts and bring them "out of their comfort zone" to discuss LLL policies and implementation is considered as a great success from the part of all partners.
Risks management General risks A too general AGORA
Medium/high
No communication by partners can lead to different ‘islands’ and no overall view Inability to deliver a product or a deliverable on schedule (finalised version of State of Play - people involved in it moved or soon will move to a different organisation) No risks so far Political reasons-(elections)
Very likely to happen
No real Extra meeting products for the exploitation Big Mention this during our next meeting
low
high
LLL-HUB
Appoint new personnel before the former leaves, pass comprehensive information about the work done so far to them
Evaluation Report (Final Version –May 2016)
16
Risk linked to a specific WP(tell which one) Dissemination
High
WP4 LLL-AGORA - inability to mobilise the local experts to participate in the LLL-AGORA
low
Many reports on a shelf and not used properly
medium
Partners are demotivated to Medium produce more work to answer the above-mentioned need WP1 management & WP6 High dissemination: financial and activity reporting being in line with the promised deliverables WP5 exploitation Medium
No extra partners in other countries, to have for the AGORA high
Better agreements to make in Krakow
Keep the contact with the group of experts that participated in LLLFORUM, keep them aware of the project progress, share information through e-mails, organise additional meetings etc. Low Partners share ideas to solve this disseminati issue + allocate work to identify on & impact elements to highlight “ “
Low Some elements are still not online disseminati such as the map (special feature of on & impact the website) and the contributions for news from the partners for our social media are limited up to now. Partners have to think about the next steps for instance how to develop further the Lab but also how to share knowledge and practices to improve their own.
Risk for your institution (or yourself) No real interest in international developments in LLL Difficult to come up with dissemination efforts towards stakeholders (not applicable for Forum and Agora) Spending too much time on the project without being supported by the project money
High
Very likely
No LLLplatform here Big
Strong preparation of the LLLAGORA Brainstorm during our next meeting
High
Some risks are well listed (lack of transversal communication between partners leading to "islands"; dissemination efforts are not seen as rewarding; the issue of exploitation
LLL-HUB
Evaluation Report (Final Version –May 2016)
17
Impacts management General impacts Positive
LR
European Network of national platforms
Exploitation in the right way, starting a.s.a.p.
Wide range of actors has MR been reached during the project Through sharing the outcomes from our research and the State of Play many target groups has been made aware of the implementation of EU LLL strategies in Malopolska region and in Poland The LLL-FORUMS – gathered MR 55 actors: representatives of various market, educational and training institutions, policy-makers, public authorities in order to ensure high quality debate, it enabled a cross-sectorial, multi-stakeholders cooperation, discussions based on a genuine research and dialogue at regional/national level on EU LLL strategies helped to formulate comprehensive recommendations
LR
Bringing Do a follow-up of the Forum, organise stakeholders webinars in touch with each other MR Keep the contact with the group of experts that participated in LLLFORUM, keep them aware of the project progress, share information through e-mails, organise additional meetings etc.
LR
MR
Dissemination
High
Medium
Policy
LLL-HUB
LR
continuous update of the project website Raising Presentation More videos, policy awareness at the KA3 recommendations, to be drawn by on LLL infoday as the data we have collected in the labs good example for policy change (350
Evaluation Report (Final Version –May 2016)
18
participants)
Experts involvement(lack of educational centres support)
LR
.
Difficulties to meet expert group
Many experts expressed in the forum the lack of support or their companies or institutions Aware about the project importance
Impacts on your Institution/Region Positive
LR
European Network of national platforms
Exploitation in the right way, starting a.s.a.p.
Positive
LR
LLL strategy
Contacts with the Ministry
MR
Fulfilling its mission: “competent, qualified and employed residents of the Malopolska region. Establish a Database of LLL Experts Reinforce our knowledge of LLL and its implementation by policy makers and educational stakeholders Regional Elections
A better Complete Labs understan ding of the most important players within lifelong learning
LR
LR
High
MR
Administr ation stands still
MR
Setting up the network
Medium
Discussions, communication and joint work (collaboration)
Impacts for yourself Positive
LLL-HUB
Writing reports, organising meetings, starting a small working group
Evaluation Report (Final Version –May 2016)
19
A better I’m starting to go through some understan literature to get to know more about ding of this issue. the most important players within lifelong learning Gaining knowledge about LR LLL issues, developing social and language skills, gaining experience in implementation of EU projects, developing creative thinking and problemsolving skills Gain experience in MR participating in project activities Understanding what is going on in the countries by talking to experts Work MR overload
High
Medium
Time Delay in availability National reduction Report
Learn from the others.
Improve time management
Partners are starting to collect the positive impacts of the Experts gathering and Forums.
Project Advancement and project Management Works as forecasted (apart from the main shift at beginning) Everybody is pleased with PM NB: There was a good evaluation from the Executive Agency for the intermediary report.
What was successful so far?
General understanding of the project objectives, agreement and commitment Good will A certain habit to work in network A shared concern about o producing concrete elements (through concrete thinking) o and putting things and people in the move The idea of Forum Observatory, Relay and Think Tank o the most developed so far? o how to further develop the others? From the interviews (as many slides as needed)
NB: Most of the pitfalls from 1st questionnaire have been overcome LLL-HUB
Evaluation Report (Final Version –May 2016)
20
What could be improved?
A common understanding about what "research" means? How to put the LABS into practice? How to ensure the "trans-" dimensions (transnational, trans-regional, trans-dimensional)? Who is in charge of capitalisation?
Advanced vision from the 3rd questionnaire
LLL-HUB Project – Advanced vision of the partners
Evolution of the vision
LLL-HUB Project – Evolution of partners' vision throughout the project LLL-HUB
Evaluation Report (Final Version –May 2016)
21
the first vision reflected mostly the concerns about the work at "national level"; the second one was more embedded in project processes and management issues; the last one is clearly showing the concerns of the partners regarding the outcomes of the project.
What people are especially proud of or enjoyed most
the building of teams of experts and their involvement into the project; the comparative report; the ability to achieve the project goals all; the commitment of partners; the positive feedback from the Agora; the innovative practices; the moments when all people were brought together to discuss LLL; the structure of the LLL-Hub and the methodology have worked; contacts with an learning from the other partners; high quality and usable outputs; the proactivity of partners.
What could have been done better and/or project flaws
a better agreement with local authorities to be able to influence educational and labour legislations; a larger use of social media; more transnational exchanges; developing a European vision; more involvement into exploitation; more time devoted during the meetings to go into deeper discussions about the topics and content; more face-to-face meetings; the difficult question of project sustainability.
The main lessons learnt
there is a large gap between education and labour market; though there are lots of organisations involved in LLL, they are not always working together at local level, and at European level there is a lack of interest to bring together a common approach; there are "organisational issues: for example, education is a national concern, and Europe does not want to interfere too much; we, as an organisation, could do more to enhance LLL pathways for learners, and that there are not enough mechanisms in place to facilitate this process; it has given us a visibility and contributed to our organisational development; we have to continue the fight to get more equity in Europe; a need for LLL platforms in every country; it enabled us to organise an attractive final conference with the participation of EU representatives.
The most significant outcomes
the comparative report; the FORUMS; the AGORA; the attention to LLL;
LLL-HUB
Evaluation Report (Final Version –May 2016)
22
bringing together the experts to collaborate and reflect upon LLL policies.
The contribution to the convergence of LLL policies in Europe
EU is trying to unify strategies between Education and Employment, but Country standards and traditions are hampering that; there is no common European Policy and Strategy, Education and LLL are considered national issues; there is a different level of awareness regarding LLL [in the different countries]; we were able to provide EU representatives with our point of view and show them "in real" what was going on in LLL; the project did not contribute to reduce the discrepancies between countries because there was not enough exchanges on this topic; the project did not manage to influence certain political levels; each partner and each country should continue working on the dissemination of results achieved so far; it was frustrating that our project was not given the right to a "true" high level meeting; a way could be to develop more national LLL Platforms, which could also be places to take into account the productions of the different projects in order to capitalise on them.
Impacts analysis (end of project) Impact type
Range
General impacts Legislative
MR
Legislative
LR
MR
Use the outcomes of the Agora Regional LLLcentres
SR
Courses for Poverty and LLL Collaboration of partners and experts
SR
Implementation of LLL-HUB Methodology
MR
LLL-HUB
SR
SR/MR/ LR
forecasted impact
Standstill Spanish Educational Law New Spanish Educational Law LLL-P (EUCIS-LLL) actions and moves to be forwarded LLL-Platform national 32 Regional centres
Training programmes Partners and experts are very likely to collaborate in the future
impact already recorded
possible action to maximise the impact
1st step to repeal it. Search suggestions for improvement and updating website and social media First meeting has taken place First meeting has taken place Start in August Some partnerships have already been formed
Presentation of LLLHHUB National Report and proposals To follow up for an ongoing activities plan within the partnership Involve municipalities and also the ministry Talking with the association of municipalities Involvement of the ministry for social affaire Find a way to involve partners involved in the project in one or the other follow up initiative or project
Use of the methodology in other contexts since it has been successful. A real possibility of it is promoted in the right way to the right people.
Evaluation Report (Final Version –May 2016)
23
Impact type
Range
forecasted impact
Impacts on your Institution/Region Investment MR Dual Training
Volunteerism
impact already recorded
possible action to maximize the impact
Recognition of DT as educationemployment shaft
MR
Certification
Recognition of importance of volunteering in professional profiles
MR
Some involvement of stakeholders and decision makers.
Recognition and social media activity
Network for LLL
MR
Network for experts
Thematic working groups
MR
Better understanding of the process of lifelong learning in Europe and Flanders Image building: putting CVO Antwerpen on the lifelong learning map in Flanders Use of (similar) Methodology in other project application Capacity to coordinate a major EU project Awareness raising through dissemination of Comparative Report
LR
Partner for the government, employers’ organisations and unions Small
Using the AGORAgroup Conference in January 2017
MR
Hard to tell (small?)
Impact type
SR/MR /LR
SR
SR/MR
Considerable awareness of one of the main project products by reaching key policy makers and around 8000 subscribers of our mailing list
Rang e
forecasted impact
LR
To know more and
Search more VET and enterprises involved and more and more investment.. Search new practices and projects to implementation of volunteering certification Better understanding of the importance of Eu projects for the mission and vision Involvement of other LLL-organisations Annual conferences and other events – to inform the target groups
/
Have these insights disseminated to the entire organisation in the form of a workshop for example
/
Participate frequently in lifelong learning events
This already influences our applications in March 2015 (rejected) X
Managing to successfully use the methodology in another context
impact already recorded
possible action to maximize the impact
Further projects
Impacts for yourself
LLL-HUB
Recognition
and
Influence
Evaluation Report (Final Version –May 2016)
decision
24
have better acknowledgments
social media activity
Networking
SR and MR
Involving more colleagues
More contacts
Better project management skills Use of (similar) Methodology in other project application Capacity to coordinate a major EU project Awareness raising through dissemination of Comparative Report
LR
Medium
Processing of financial claims is more fluent. This already influences our applications in March 2015 (rejected) X
SR/MR/ LR
SR
SR/MR
makers. To document, to publish, to discuss, to dialogue, to cooperate. Using a new foundation for new activities (for LLL) List the lessons learned once the project has finished Managing to successfully use the methodology in another context Further projects
Considerable awareness of one of the main project products by reaching key policy makers and around 8000 subscribers of our mailing list
Life after the project
we will organise another Forum before the end of 2016; we will derive a plan of activities for dissemination and promotion and try to reach other regions build our national LLL platform; disseminate LLL-Hub reports to our stakeholders; have another meeting with partners in September 2016; keep in contact with project partners and search on new projects where we could put in practice the results of the work done; try to find other resources to continue our engagement; look into the option of having EPALE as possible communication platform to maintain the virtual network of experts; continue to advocate for LLL whenever possible; all partners see the role of EUCIS-LLL (LLLP) as a key to maintain the link with European Institutions regarding the further developments of LLL.
Interviews of coordination and WP leaders For this specific part, see external evaluation report in Appendix 4
LLL-HUB
Evaluation Report (Final Version –May 2016)
25
Appreciation of the methodology The overall methodology proposed and used in the project was based upon a spiralling process to intertwine top down processes (e.g. field research leading to the writing of the National State-ofPlay) and bottom-up processes starting from the field research (Practices, learners' stories, case studies, etc.) to build up a whole set of lessons learnt and recommendations:..
Overall methodological process for the LLL-HHUB project Such a methodology was well suited to the overall "LLL-HUB "device" architecture: Labs, Forums, Agora:
Architecture of the constitutive elements of LLL-HUB And in accordance with the information collection and management processes:
LLL-HUB
Evaluation Report (Final Version –May 2016)
26
Methodology of data collection, interpretation and capitalisation in LLL-HUB
Appreciation of the functioning of the Actors Networks There were several networks of actors intertwining in the LLL-HUB project the network of consortium partners the local networks in the LABS and FORUMS the European Network during the AGORA The local networks were considered as a great success for the project. They enabled to have plenty of different people from different institutions and with different viewpoints to interact and share, especially during the FORUMS. The Partners' network within the project worked very well. The face-to-face meetings were friendly and fruitful. These meetings were really important to support the horizontal communication between partners, especially during the first two years of the projects when the work to be done was mostly nationally/regionally focused. The Agora was effectively a time of European openness. There were a lot of discussions and sharing about the practices and field examples. Within the project there was an interest for what the others were doing, but no cross participation in Forums (except from coordination); there were two issues here: the travels that it would require and the question of language, the Forums being held in local language; lack of time to discuss the forums outcomes in Cracow beyond the presentation of partners' interviews; lack of "space" to build the roots of a "European Labs network" within partners; it was not forecasted in the project, but the concept could have been envisaged in Sofia, for example; such a network would be the real support for a common reflexion about policies convergence in the mid and long term; EUCIS-LLL is seen as the main "vector of Europeanisation"; the remaining issue is how will EUCIS-LLL Network re-appropriate the LLL-HUB work, and more specifically the HUB structure for sustainability? LLL-HUB
Evaluation Report (Final Version –May 2016)
27
Appreciation of the core WPs: LABS, FORUMS, AGORA At the beginning, the work of the partners within LABS was mostly devoted to a desk research activity, in order to produce the National States-of-Play. The field research part was more intended to have a confirmation, from the part of professionals, of the situation pictured by the texts and figures from the desk research. We (FREREF) thought that it could be a great opportunity to not only collect facts and figures, but also to get examples of the "real life" of LLL on the field. This is why we proposed the general framework for the LLL-Hub structure and for the LABS (as Observatory, ThinkTanks and Relays) and a methodology for collecting innovative practices and learners' stories. At the beginning it seemed a bit abstract for partners, but then they started gathering experts for their forums and it became more concrete. As it was designed, the Labs should have worked during some weeks before presenting their work to a panel of experts and asking for their advice. The main effort was put on gathering experts and most of the" real work" of the Lab was done during the experts meeting(s). The other important activity realised by partners was to collect practices and learners' stories, which were discussed and capitalised later in the AGORA. The main impact recognised by partners was in putting together, in the experts' meetings, people who were not used to work together and having them share and exchange ideas. The LAB eventually turned into a "local network of actors in LLL", which is a very interesting outcome. During three quarters of the project time, partners were more "Country centred" or "Region centred". The real moment when they started to look at the European point of view was when preparing and participating in the AGORA. The organisation of the AGORA activities, which devoted enough time to formal (the workshops) and informal (the breaks, lunches, dinners, etc.) networking activities, was a true European experience for the project participants and their guests. The link with European institutions was taken care of through two activities: the participation of Sophia Eriksson Waterschoot from DG EAC and Joao Santos from DG Empl in workshops and panels, and the High level meeting with Joao Santos (only for project partners). The reaction from the commission level was a bit disappointing in the sense that they saw LLL as only country relevant and said that they could not "interfere" with countries policies. The reaction of Joao Santos was also disappointing for some partners, because he stated that there would not be any kind of "special money" to pursue the LLL-HUB activities; but, really this was what we could expect unfortunately. Both FORUMS and AGORA were considered as great successes for the project by all participants and external stakeholders.
Appreciation of the transversal WPs: Communication – Valorisation – Exploitation - Sustainability Communication and dissemination Within the project The "starred" communication process between EUCIS-LLL and all the partners was efficiently dealt with. As usual, the electronic platform was mostly used as a document repository, the communication taking place rather by e-mail. The transversal or horizontal communication process was less successful, mostly because during three quarters of the project partners were mostly focused on "national" issues and "national" LLL-HUB
Evaluation Report (Final Version –May 2016)
28
dimensions (or even regional ones). It was only a propos the AGORA that the European (or at least trans-national or trans-regional) dimension was put in front as the key process.
At National level The communication and dissemination activities at Country or Region level were mostly dealt with through or in relation with the FORUMS and experts' meetings. All partners report about the innovative process at work during this phase and the very fruitful outcomes regarding the ability of people from different institutions and with different interests to share and work together. They also pinpoint the possibility of a sustained process after the end of the project.
At European level The EUCIS-LLL team is always well experienced and efficient regarding the general communication about what they are producing; the valorisation of what is happening in their networks; the exchange with high level people in the different commissions and with MPs. They did apply their skills within the project, which means that a good work was done from their part at European level. Nevertheless, there was not a real appropriation among partners of the European dimension; even the horizontal communication between partners (i.e. not going through the Project coordination) was not very developed. It was only the evaluation who took time to cross read all the practices and learners' stories. This is why we presented the findings from theses readings in the Sofia meeting under the synthetic concept of LLL Pathways. It was well listened to. What was especially satisfying is the place given to this content – which, for us, is the real innovative production of the project during the Agora in the workshops.
On social networks
Facebook: the project Facebook page was managed by APG. One can see that there was no real "Facebook concern" or "Facebook culture" among the partners. Facebook is primarily a personal tool, and if people do not use it for their own communication, it is very difficult to have them use it as a group/community tool. Twitter: the LLL-Hub twitter accounts @LLLHUB - https://twitter.com/LLLHUB Website: there was a nice effort done to issue a web site with a good graphic charter (http://www.lll-hub.eu/ ). It gathers all relevant information about the experts, the field data collected (practices, learner stories), the methodology, the outcomes (National reports, final comparative report, etc.).
Valorisation and Exploitation The issue of "exploitation" was a bit disturbing because no one really saw what to "exploit". There were some discrepancies between the partners about that. We think it would have been more profitable to shift from "exploitation" to "valorisation", even at the level of the proposal, as most of the results of the LLL-HUB project are immaterial, and there are no real concrete elements that could be "exploited" in a commercial or even non-commercial way. Valorisation took effectively place within the project, mainly at the Forum level in the territories, and through communication and dissemination in different events: locally (e.g. Flanders, Caceres, Mardin), nationally (e.g. Netherlands and Portugal, and at European level (EUCIS-LLL and FREREF) What comes out of the project at the end is that: partners are able and willing to disseminate and valorise the outcomes of the project at their regional/county level by sustaining the experts' meetings activity; at European level, EUCIS-LLL – together with the contribution of FREREF - seems to be able to valorise the European dimensions of the project; the comparative report is a great tool for that, but the value of the databases of experts and of practices should not be underestimated. LLL-HUB
Evaluation Report (Final Version –May 2016)
29
Sustainability The issue sustainability is a constant concern of the evaluation process. It was dealt with in the different questionnaires and during the meetings, especially in Sofia. The concerns there were: is there a possible life after the project? is it only a question of money? what about a "LLL-HUB state-of-mind"? how could each partner, and especially EUCIS-LLL appropriate the project outcomes and productions? Partners are pretty confident in their ability to sustain the reflexion at local level. Some local outcomes already happened; for example, there will be a national platform for LLL developed in the Netherlands. There is clearly also a concern about how the European Institutions are regarding the ""LLL Issue". Their position is not clear about that. It is considered as a kind of "transversal" thematic between DGEAC and DG Empl, but nobody is really in charge, and even the term LLL itself is less present in the communications and plans. This is naturally questioning for the project partners.
Appreciation of the main deliverables The main deliverables regarding the spirit of the project and the dissemination and valorisation processes are the National States of Play and the Comparative Report Writing the National States of Play was a painful exercise for most of the partners. People involved in the project are more on the practitioners' side; they are not researchers and not used to compile documentation and take the best of it to rewrite a sense making document. It was also difficult for them to articulate the examples from the field research with the information and data collected during the desk research. The third difficulty was to write a long document in English, which required from most of them a first translation from the original documents in national languages. So there were some flaws, like the occasional compilation of existing reports, in English, with a copy-paste strategy; at least it was well written, but it did not necessarily match the table of content and added little novelty (e.g. for the Turkish partner). Nevertheless, in the end, all partners were able to produce their National State of Play, with a level of homogeneity and quality suitable for issuing the comparative report. One very important outcome of the exercise was for the partners themselves, who were able to have a much more comprehensive vision of the LLL area in their country/region and should now be able to implement a full observatory of the progresses and evolution of the policies and regulations. The Comparative Report was put together by EUCIS-LLL who are experts in doing these kinds of compilations and synthetic publications. They made the best from the National Reports and issued significant tables and arguments about the situation of LLL in Europe (at least for the 8 countries/regions concerned by the project. The document is sound and easily readable, with high quality regarding the format and rendering. The "theoretical issues" are very interestingly illustrated with examples taken from the innovative practices and learners' stories, which give a very lively report.
LLL-HUB
Evaluation Report (Final Version –May 2016)
30
Appreciation of the AGORA from the part of external Stakeholders (see also Appendix 3 for detailed results of the questionnaire) All participants appreciated the general organisation, the venue, the keynote speeches (especially M. Baert), the workshops and panels. They enjoyed the networking, the debates within the workshop, the practical and concrete examples, and the experience from the field together with the comparative report. They felt like belonging to a community. They think they could contribute to maintaining the LAB and FORUM activity and to disseminating outcomes at national level. They would like to include intergenerational learning in their activities, or promote LLL guidance and dual education. They just regret that there was not more time for discussion, debates and exchanges!
LLL-HUB
Evaluation Report (Final Version –May 2016)
31
LLL-HUB Findings The main findings and outcomes of the project are: the LLL-HUB architecture (the LABS, FORUMS and AGORA), and its role in sustaining an open dialogue between a diversity of experts and stakeholders in the LLL domain; the diversity of standpoints and interpretations regarding the Lifelong learning issue itself; thus the importance of anchoring the possible convergence of policies into the sharing of innovative practices, thus building a common understanding of the key concerns of LLL; the diversity of the LLL policies and practices in Europe, needing to foster and sustain ongoing efforts at every level (local, national, European) to make further steps towards convergence.
An innovative architecture to explore the LLL policies in different regions and countries in Europe The LLL- LABS (see also report on Methodology) Three roles were imagined for the LLL-LABS: Observatory, Think tank and Relay.
The role as Relay, means the ability to relay the European policies and initiatives to the field and to issue recommendations to the local actors in order to implement those directions.
The LABS as relays of the European policies and initiatives The diversity of situations in the different countries / regions, as reported in the National State of Play, show that this part of the job is very important and not trivial.
The role as Observatory was the one that took place when issuing the National State of Play and collecting the innovative practices and learners’ stories.
The role as Think-Tank took mostly place during the forums, which were a privileged time and place to share and build new knowledge. It was then associated to a "Do-Tank role" as some possible common actions emerged from the meetings.
LLL-HUB
Evaluation Report (Final Version –May 2016)
32
The LABS as Think tanks for issuing recommendations towards European Institutions The think tank level is also the level at which a cross re-appropriation could be made between the different countries.
The LLL-FORUMS They enabled a radically new dialogue and evidence the necessity: to articulate regional/national levels throughout Europe and with the European level to foster the articulation o between civil society and national policy makers o between levels national regional/local European The FORUM level is where the innovations either happen or are acknowledged and capitalised.
The LLL-AGORA The LLL-AGORA was the time by excellence for building the LLL-HUB European Network of experts.
Regarding "Lifelong Learning" There is a lot of implicit meaning behind these three letters, and we could hope that the desk research would have made them more explicit. One of the important element is that many people continue to think "training" instead of "learning". Thus, the focus is more about the systems, organisations, tools and processes of continuing education, or adult education, rather than about the complex processes and actors of the less formal learning along life. Most of the LLL-HUB partners belonged to education and/or training institutions; two of them were more related to the working area (Caceres chamber of Commerce and Malopolska WUP), and one was more dealing with global concerns regarding refugees and migrants (Province of Mardin). This diversity of situations and standpoints was the main reason why we suggested to collect innovative practices in LLL learning and LLL learners' story telling; it was a way to reintroduce aspects from the less formal learning experience. In the end we think that we have some very interesting results. We will present them in the part about LLL Pathways (lessons learnt). See also the report from external evaluation in Appendix 4). Currently, the interest in LLL is also boosted by: Employment problems Migration issues Relationship with companies LLL-HUB
Evaluation Report (Final Version –May 2016)
33
Social disintegration Etc.
The comparative report The comparative report starts with a very good reminder of definitions, and "historical" processes regarding the evolution of LLL concepts. It enables to replace the issues of the different national/regional situations within a larger framework. The comparative reports the presents a set of comparative tables that enlighten the diversity of situations- and some common items - of LLL in the different countries, according to the different criteria commonly used in the communications from the commission.
About recognising LLL as a specific domain needing specific structures and resources The LLL concept in itself is a fluctuating one. It is supposed to cover all aspects of learning and knowledge building, and all moments in life when we learn and build knowledge. There is then a large open space for different interpretations, discussions, evaluations, implementations, etc. The main issue here is to recognise that there is a specific LLL domain, which is different from Education, initial, continuing or adults, different from training at the workplace, different from self-learning, and probably englobing parts of all of them. It is at crossroads between education, employment and citizenship, and certainly a key issue when regarding inclusion and wellbeing. The LLL-HUB research processes and reports show the importance that all concerned actors attach to all the diverse forms of learning, and the processes which foster, support and enable them. It is thus very important that the European Institutions also recognise LLL as a distinctive area and devote distinctive actions, initiatives and resources to it.
Diversity of the LLL Policies and practices in Europe and how to help them converge The desk research provided with the National State-of-Play (Belgium, Bulgaria, Netherlands, Poland (Malopolska), Portugal, Spain, Turkey (Province of Mardin). As we saw, the concept of LLL in itself is rich and diverse, so it is not surprising that the implementation of policies, strategies and actions in countries where historical, cultural educational and economic conditions are so varied led to a whole fan of concrete situations. As we see through the reports (national and comparative), some efforts have to be deployed to have a convergence of LLL policies between countries and with the European concerns. Nevertheless, the reports show that every country: has a concern for evidencing specificities of LLL within their education and employment policies; is willing to acknowledge the fact that LLL is a domain requiring to work in networks between all actors of the different components of the area; is more than willing to share experiences, practices, knowledge, people and resources with other partners in Europe; European projects are naturally a privileged way for doing this, but bilateral and multilateral cooperations, exchanges and mobilities could also be developed for that; is anxious to stay at level with European directives and evolutions.
LLL-HUB
Evaluation Report (Final Version –May 2016)
34
Outcomes of evaluation The evaluation participatory processes, both internal and external produces a large quantity of information. According to the evaluation methodology, the outcomes of the evaluation processes (not to be confused with the outcomes of the project that were presented above) are discussed within a four level framework:
The 4 levels of presentations of the outcomes of the evaluation process
Satisfaction The satisfaction level deals with the immediate appreciation of what is happening, especially at the end of events and productions. Satisfaction is the level which is at stake when dealing with meetings evaluation questionnaires. This questionnaires aim at assessing the level of satisfaction of participants regarding the organisation, the venue, the agenda; the meeting management; the issues on project management and finances; the possibility for partners to express their concerns and present their works; the specific issues at the agenda. Detailed content of the answers are available in Appendix 2. Overall the satisfaction is at a high level for all partners.
Results The result level deals with the quality of productions and outcomes of the project. First of all, all deliverables were delivered, rather on time. There were some issues regarding one or the other partner at some moments, but, in general there are no major flaws regarding this part of the project.
The methodology The methodology as such is a full outcome of the project. A specific report was issued for methodology. The specificity of this methodology is the articulation of a top-down process (desk research, leading to the national reports) and a bottom-up process (field research, leading to the database of innovative practices and learners' stories, and to the lessons learnt about LLL-Pathways.
LLL-HUB
Evaluation Report (Final Version –May 2016)
35
The LLL-HUB architecture The LLL-HUB architecture is a unique way and – together with an innovative structure- of putting in place a European network (at least the start of it with already seven Countries or Regions) in the area of LLL. It conjugates both a sound basis at the local level with the experts and experts' meetings (as far as they may continue after the project), and an organised web of interactions, within the Forums at local level and through the AGORA at European level. It is rather expectable to have a degree of sustainability for the network of experts at local level, because it can be intertwined with other activities. On the other hand, it is more difficult to expect a new opportunity for an Agora in the future years (should be every other year or so), without any specific financing. A solution could be to integrate a new Agora within a recurrent event organised by one or the other partner, or by an external stakeholder.
The National State of Play and the Comparative Report The national State of Play are based on a desk research process, which was gathered all data about the situation of LLL in the country, the forming of the LLL policy, and the relation between this national policy and the European policies and initiatives. All these data are more or less already available in one or other publication of the European Commission, but it was useful and formative to try to have them gathered in a common document. The comparative report was able to give a picture that puts in perspective the current and possible evolution of the policies, with the idea of a kind of convergence between them and with the European policy level.
The innovative practices and learners' stories To our opinion, the collection of innovative practices and learners' stories, and the methodology developed to capitalise on them is clearly a key innovation from LLL-HUB.
Impact The Impact levels deals with the short-term, mid-term and long-term impacts that may be already experienced during the project and at its very end, or forecasted for the" life after the project". It is a very difficult exercise to reflect at this level on what is produced and implemented during a project. What is especially difficult is: to evidence "non-trivial" possible impacts, i.e. impacts that will really change something on the field, other than simply the interest of working together in a European project (which is not sufficient anymore; to imagine what could happen after the end of the project; once the project is over, all people have to start looking at something else, because they are not funded anymore for what the project was dealing with; the only reasonable impacts that could be expected are about making synergies between the concerns developed during the project and some recurrent concerns for the structures; to explicitly refer to the project productions in further works; this is not too difficult generally for the project coordinator because it is their "on-going business"; for partners it is more difficult because they go back to their usual activities (chamber of commerce, ministry organism, training institution, etc.), and the project becomes a fruitful parenthesis and a rich experience that one would like to experience again soon.
Local level The main experienced impacts and forecasted at short term at local level are : the federative role of LLL-LABS and LLL-FORUMS; LLL-HUB
Evaluation Report (Final Version –May 2016)
36
a real innovation regarding who is speaking with whom now; the work about collecting innovative practices and storytelling ; the value given to a bottom-up methodology, starting from field experience to issue recommendations for policy makers.
National level Education is sometimes a national concern, sometimes more a regional concern, but always with some national components. Continuing education and adult education are more diverse, including responsibilities from universities, schools, but also from companies and specific organisations, rather private than public. LLL, on the other hand, is something related to personal learning pathways. The learning itself is on the responsibility of individuals. The systems enabling and supporting learning, the accompaniment professionals, the recruiters, the HR people in companies, the employment agencies, the organisations working for the recognition and valuation of less formal learning, are the responsibility of the authorities who have the power to sustain and encourage them, most of the time local authorities: proximity is a key concept here - proximity of people, of the job market, of the professionals. This does not mean that mobility is not to be considered, rather the opposite. However, mobility can also be envisaged from a local context towards another, using the privileged links that a Region, or a City, may have with another, for example.
European level Naturally the role of Europe is crucial. Europe is the place where policies are decided and initiatives are taken, hence the role of Relay for the LLL-LABS. Europe is also the place where innovation might be gathered in order to influence the new policies and initiatives, hence the role of Observatory and think-tank for the LLL-LABS. Europe is also the provider of a whole set of resources without which nothing could be possible at European level, but also at local level. We could expect that such a project would trigger some initiatives at least to reconsider the issue of LLL a full concern in itself, beside Education and Employment. European networks such as EUCIS-LLL and FREREF are advocating for recognising that and supporting specific actions and projects in this area. We truly hope that the Commission and the Parliament could rely on the outcomes of projects like LLL-HUB to understand and value the key importance of LLL.
Lessons learnt The Lessons Learnt level deals with the issuing of what was learnt during the project that could give rise to recommendations for practitioners, decision makers and policy makers at local, national and European level. We will focus here on two main elements the Life Long Learning Pathways, as a concept to summarise the main ideas at work within the LLL-HUB project the LLL-HUB architecture as a key tool to achieve (another step towards) convergence of LLL policies in Europe See also the report from external Evaluation in Appendix 4 for more detailed lessons learnt from the different activities and actors.
LLL-HUB
Evaluation Report (Final Version –May 2016)
37
The idea of Life Long Learning Pathways One of the most important lesson learnt from the LLL-HUB project, is the idea of Life Long Learning Pathway, and how to work with it to understand, develop and support LLL policies, in the countries and across Europe. Reading from practices, storytelling, case studies and interviews Our concerns (as partner and evaluators), is to contribute to the objective about "convergence in LLL policies in Europe" and to contribute to sustainability of LLL-Hub structure. This may be done by fostering cross fertilisation between practices in the different territories and thus help capitalising at project level. After going through all reports from the partners regarding the innovative practices, LL Learners' stories and interviews and case studies, we found out that we have gathered a lot of constructive and innovative information about what could be called "LLL pathways; i.e., the way people (learners, institutions, companies, accompanying structures, employment agencies, etc.) build together the processes and tools of Life Long Learning, and the way LL Learners travel along these pathways in order to develop their learning throughout their professional life. Here are some key findings resulting of the search of commonalities between all examples, beyond their diversity in form, content and context. Each idea is illustrated with references to examples given by the different partners. It would be too long to annex all documents here, so we only give some reference. The text of the different documents should be available from the project web-site. LLL is "territorialised" LLL is strongly related to the dimensions of territories: historical, cultural, economic, societal, educational, etc. Territories – Regions, sub-Regions, Megalopolis, etc. - are the places where LLL innovation can happen at a "practical" level. Furthermore, territories have enough power and responsibilities to make things happen in LLL. Most of the time LLL is in their attributions, even if it is not always explicitly expressed in formal LLL policies statements. Nevertheless, the "territorial evidence" may cross boundaries and foster trans-border cooperation in LLL.
Examples from LLL-Hub partners
LLL-HUB
Evaluation Report (Final Version –May 2016)
38
People have to "enrol" in their LLL Pathways People are not always well aware of what they are learning and how they are learning it. If it is clear when they follow a formal education program, it is less obvious when they enrol in accompaniment processes or when it comes to less formal or informal learning, peer learning, self-learning etc. How to "make people want to learn" during their whole life? If they are not formally engaged in learning, we have to work with them about the benefits they could collect from gaining knew knowledge. And also we have to help them recognise that nonetheless they are able to do it, but that they already do it all time. If they dropped out of school, if they were not very successful in their studies, or if life was tough with them and it was not possible for them to complete their studies and get a diploma, or if they arrive in a new country where there previous studies are not validated, they need a strong accompaniment to hang back on their learning life. If they are already at work and want to progress or evolve or change their job tracks, even if they are not following further education courses, we must help people recognise for themselves that they are engaged in a learning journey and following what we call here a learning pathway. As a starting point, how to help people recognise what they already know, how they have learned it, what they learn in their day-to-day life; and also to express what they would like to learn to go further. All along the process, it is important to show them and help them recognise the value created by the journey along the learning pathway, for themselves as people, for their family, for their workplace, for their company, for their value on the job market, etc.
Examples from LLL-HUB partners What is a LLL Pathway and how to build one? Naturally, each and every LLL Pathway is different from the other. Nevertheless, it is possible to observe some typical processes and patterns: alternate sequences: formal / less formal, individual / collective, in education / at work (or outside both); multi-modes – continuity in outbreaks – prepare to un-predictability. Support and accompaniment are a key success factor: by professional (teachers, trainers, employers, etc.) by third parties (unions, organisms, volunteers, tec.) by peers (family, friends, etc.)
LLL-HUB
Evaluation Report (Final Version –May 2016)
39
Examples from LLL-HUB partners Roles of actors and stakeholders A lot of persons, situations, contexts, influence people's LLL pathways: Individuals Peers and family Accompaniment structures Job market and companies Institutional bodies Territories Europe
Examples from LLL-HUB partners Mechanisms and tools LLL Pathways building and travelling are enabled and supported by a whole system of mechanisms and tools, more or less formal, more or less "institutional, more or less "improvised", but always fruitful and helpful. Such a process "has to" be flexible, ad-hoc, stretchable in time and place. People might be convinced by examples, stories, testimonies, meeting other people, etc. But some elements are inevitable in order to recognise the learning itself: reflexivity, and a "given amount" of evaluation; evaluation must be suited to the non-formality of the learning, and thus one may think about peer evaluation, self-evaluation, appreciative evaluation, etc. rather than usual assessments. Perseverance is required and might be efficiently sustained by a continuous appreciative recognition of the efforts made and progress achieved
LLL-HUB
Evaluation Report (Final Version –May 2016)
40
Examples from LLL-HUB partners Recognising and valuing LLL Pathways Recognition and valuation may start with self-recognition and peer recognition. Then most of the time, people look for field recognition (people around them who accompany them or help them) and workplace recognition (how what they have accomplished help them do a better job or find a better job). Institutional recognition is not always necessary, in the sense that people are not always engaged in a program that will end with any kind of diploma. Nevertheless, "paper recognition" might be a great tool for motivation (award, plaque, prize, etc.) Societal recognition is also important. It means that those people that did not "fit" well in their societal environment at the beginning may gain by taking their place in the society. It has also an influence and impact on citizenship. In the end, financial recognition should also take place, by means of a better salary, of a job at last, or the capability to create one's own employment, etc.
Examples from LLL-HUB partners We need systems to foster, support and sustain LLL Pathways In the same way that we build educational systems, we need to build LLL support systems if we want to be able to accompany every person successfully. At individual level, it is important that everybody could "have an idea" of what is possible for them, mostly by identifying the possible places to find information and accompaniment ("LLL single stop shop"). At company's level, it is important that HR people and recruiters work closely with all actors implied in the management of LLL Pathways supporting system. It might be a key thread of the societal responsibility of companies. LLL-HUB
Evaluation Report (Final Version –May 2016)
41
At territories level, it is important to ensure that all components and constituents of the LLL system are "embedded" in a well identified proposal of value creation. It is important to ensure continuity beyond the different sequences, breaks, etc. The territory is also the place to experiment, capitalise and issue recommendation for the policy makers, at national and at European levels.
Examples from LLL-HUB partners Territories gave their own learning pathways Territories learn from their own practices and history, but they also learn from other territories. Most of the time it is not possible to transfer experiences and practices from one context to another, the process is more one of learning from the experience of others, and try to re-appropriate for yourself what can be successful re-adapted. The role of Europe there could be to foster, support and sustain the inter-territorial learning about LLL Pathways and implanting of international LLL Pathways, to provide with frameworks, guidance, and finance and to create a dynamics at European level. The LLL-HUB project has been a step further in the LL Learning Pathways of the partners regarding the diversity of the LLL policies and the possible actions that could be taken to foster their convergence.
How to achieve (another step towards) convergence of LLL policies in Europe The LLL-Hub project showed that there is a strong need to sustain a structure which would enable to have a permanent knowledge of the state of play of the implementation of LLL policies in the different countries, not only at a macroscopic and statistic level, but also through field practices and life stories.
LLL-HUB
Evaluation Report (Final Version –May 2016)
42
LLL-HUB architectures and functions for sustaining the convergence of LLL policies in Europe
The activities of the LLL-HUB structure, as Observatory, Think Tank and Relay, fulfilled the needs of information and experience sharing at all levels, also in informing the policy and decision makers about the successful existing practices, how to find ways to propagate and disseminate them among actors, and how to use them to influence the necessary evolutions of the global policies of the Commission.
LLL-HUB
Evaluation Report (Final Version –May 2016)
43
Appendices
Appendix 1 – General Questionnaires Only the questionnaires are presented here. The results are commented in the text of the document (Vision of the consortium partners)
Early vision 1. Your general vision of the project 1.1 How would you describe in a few words the vision that you have currently of the LLLHUB project? Which are, for you, the main objectives and characteristics of this project? How would you phrase the Key Success Factors of the project? What could be the main pitfalls the project could encounter? 1.2. What is your role in the project? Did you propose it? Does it meet your expectations? What are your personal objectives and concerns in the project? What do you expect from your participation? What are your main responsibilities? (See question 3. for the risks associated)
2. Project Advancement 2.1. What are your contributions up to now? How do you appreciate them (over expectations, as forecasted, less than forecasted, different from what was planned…)? Give details. 2.2. How do you appreciate the work done within the WPs which you belong to (over expectations, as forecasted, different from what was expected…)? Give details. 2.3. How do you appreciate the general unrolling of the project? 2.4 How do you appreciate the project management?
3. Risk Analysis Table of risks as you perceive them Thank you for filling the table below in the most thorough and accurate way Risk
probability
General risks Example : Inability to deliver a product or a deliverable on schedule
LLL-HUB
low
Impact high
Action (1) The project management is in charge of piloting the project in a way that ensures a smooth delivery of products and deliverables by each partners, together with the partners. In case of problem, a solution has to be found first with the partner, then at project level, then, if necessary in
Evaluation Report (Final Version –May 2016)
44
accordance with the Agency Risk linked to a specific WP(tell which one) Example for WP2: ability to medium mobilise the local experts to participate in the lab in a long term perspective
Risk for your institution (or yourself) Example: difficulty to mobilise medium enough time from our part to achieve the project works
high
Each partner is committed in this task. If necessary they may find some good ideas from other partners' action or rely on lists of possible experts from the project head
medium
You may envisage to delegate part of your activity (either your activity in the project or your "other" activities", which requires both anticipation and proper resources management all along the project
You may add as many lines as you wish. (1) action to take to minimise risk, or what to do if the risk occurs
4. Impact Analysis (early stage) Impact table as you expect them or as you already recorded them NB: an impact is something that inflicts a change in what is happening. One generally tries to minimise the "negative" impacts (that produces changes that are not wished, or not in the right direction), and to maximise the "positive" impacts (changes that are in the right direction according to the expected results of the action). Thank you for filling the table below in the most thorough and accurate way Impact type
Range (2)
forecasted impact
impact already recorded
possible action to maximize the impact
General impact Impacts on your Institution/Region Impacts for yourself
(2) Short range (SR), medium range (MR), long range (LR) You may add as many lines as you wish.
5. Orientation of the project for the coming year 5.1. What are for you the most crucial steps for the project during the next year? How do you intend to contribute to them? 5.2 Do you think there should be some reorientations for the project (even minor ones)? If yes, which ones? LLL-HUB
Evaluation Report (Final Version –May 2016)
45
5.1 Are there some intermediary planned results of which you think that they will probably not be achieved or not according to schedule, or not completely? How will you manage this situation?
6 Role of evaluation 6.1. How do you appreciate the overall evaluation of the project so far? What would you suggest to improve it? 6.2 What kind of information are you expecting from the evaluation to help you improve your own position and your work in the project?
7. Further remarks: thank you for sharing your other remarks and/or suggestions
Intermediary vision 1. Your general vision of the project 1.2. How would you describe in a few words the vision that you have currently of the LLL-HUB project? What has been the most successful so far? Do you perceive some significant changes since the beginning (apart from the general shift of dates for the whole project)? What could be improved for the last project year? Why? How? 1.3. What about your role in the project? Does it meet your expectations? How are you and your institution currently implied in the project? Is it different from what you expected? To what extent? What are you especially proud of in your participation in the project? What could have been done better from your part? Is there something you plan to change during the last year? Why? How? 1.4. Which kind of keywords do you think best represent the project (outside of "LLL")? Please list below (5 keywords at least)
2. General Project Advancement 2.1. How do you appreciate the work done within the WPs which you belong to (over expectations, as forecasted, different from what was expected…)? Give details. 2.2. How do you appreciate the overall unrolling of the project? (Too slow, too fast, not enough accompaniment, etc.). Give details. 2.3 How do you appreciate the project management? 2.4 What are your main concerns about the last year of the project? 2.5. Do you think there should be changes
in the task repartition? in the planning? in the responsibilities?
LLL-HUB
Evaluation Report (Final Version –May 2016)
46
other?
3. LABS and FORUMS 3.1. We defined 3 roles for the LABS: Observatory, Relay, and Think Tank So far, in your Lab, which one(s) took the largest place? Why? Please explain. 3.2. How do you intend to develop the other roles in the future? Please give some examples of activities that could be undertaken during the project last year. 3.3. How do you appreciate the idea of Forum (whether you already held it or not)? (For example: "it is really fruitful, once people are aware, it is not so difficult to organise, we should do it again" – or, on the opposite "it was really too complicated and too costly, we will not be able to do it by ourselves after the end of the project" – or….?) 3.4. How would you describe the main roles of your Forum (whether you already held it or not)? (e.g. collect information, share between experts, disseminate experts' point od views to a larger audience, express recommendations for policy makers, precise the state of play, collect new practices, impact on innovation, ….)
4. Risk Analysis Table of risks as you perceive them Thank you for filling the table below in the most thorough and accurate way Risk
probability
Impact
Action (1)
General risks Risk linked to a specific WP(tell which one) Risk for your institution (or yourself)
You may add as many lines as you wish. (1) action to take to minimize risk, or what to do if the risk occurs
5. Impact Analysis (intermediary stage) Impact table as you expect them or as you already recorded them NB: an impact is something that inflicts a change in what is happening. One generally tries to minimise the "negative" impacts (that produces changes that are not wished, or not in the right direction), and to maximise the "positive" impacts (changes that are in the right direction according to the expected results of the action). Thank you for filling the table below in the most thorough and accurate way Impact type Range forecasted impact impact already possible action to (2) recorded maximise the impact General impact Impacts on your Institution/Region Impacts for yourself LLL-HUB
Evaluation Report (Final Version –May 2016)
47
(2) Short range (SR), medium range (MR), long range (LR) You may add as many lines as you wish.
6. Orientation of the project for the coming year 6.1. What are for you the most crucial steps for the project during the next year (which is the last one)? How do you intend to contribute to them? 6.2. Do you plan to have another round of Forums next year? If yes: when? Before the end of the project or after? If no: why? 6.3 Are there some results for which you think that they will probably not be reached according to schedule, or not completely? How will you manage this situation? What should be done at project level?
7. Sustainability of the LLL-Hub structure after the end of the project 7.1. What is your opinion regarding the sustainability of the LLL-Hub structure after the project? (For example, is it realistic or utopic?) Give details. 7.2 At your level, what could be done to try to extend the life of your Lab after the end of the project? What do you intend to do? Did you ask your experts what they think about it? What were their answers? Which resources would you need? Are you able to mobilise these resources?
8. Further remarks: thank you for sharing your other remarks and/or suggestions
Advanced vision 1. Your general vision of the whole project 1.2 How would you describe in a few words your synthetic vision of the LLL-HUB project? (how would you describe the project in just a few sentences?)
1.2. What has been the most successful? 1.3. What about your role in the project? What did you enjoy most? What was the most difficult? What are you especially proud of in your participation in the project? What could have been done better from your part? 1.4. What were the main flaws of the project?
2. What would be the main "lessons learnt" for you, from the work done in the LLL-HUB Project? 2.1. At personal level? LLL-HUB
Evaluation Report (Final Version –May 2016)
48
2.2. At institutional level? 2.3. At European level?
3. Your appreciation of the Project outcomes The project produced several significant outcomes, at national and European level. 3.1. What are the most significant outcomes for you at national level? Explain why. 3.2. What are the most significant outcomes for you at European level? Explain why.
4. Your appreciation of the project contributions to the convergence of LLL policies in Europe 4.1. Now that the project is almost over, what are the main discrepancies that you may still see between the different LLL policies in Europe? 4.2. To what extent did the project contribute to reduce some of them? Give as many points as possible which show that, to your opinion, this contribution was effective 4.3. What remains to be done? To your opinion, how could we make a step further to continue enhance and foster this convergence?
5. Impact Analysis (end of project) Impact table as you still expect them or as you already recorded them NB: an impact is something that inflicts a change in what is happening. One generally tries to minimise the "negative" impacts (that produces changes that are not wished, or not in the right direction), and to maximise the "positive" impacts (changes that are in the right direction according to the expected results of the action). Thank you for filling the table below in the most thorough and accurate way Impact type Range forecasted impact impact already possible action to (2) recorded maximise the impact General impact Impacts on your Institution/Region Impacts for yourself
(2) Short range (SR), medium range (MR), long range (LR) You may add as many lines as you wish.
6. The life after the project 6.1. What do you intend to do in the next year to continue the work started during the project at national level? 6.2. What do you intend to do in the next year to continue the work started during the project at European level? LLL-HUB
Evaluation Report (Final Version –May 2016)
49
6.3. How would you see the possible role of LLL-P (EUCIS-LLL) in helping you sustain the work after the end of the project?
7. Further remarks: thank you for sharing your other remarks and/or suggestions
LLL-HUB
Evaluation Report (Final Version –May 2016)
50
Appendix 2 – Meeting evaluation Here are presented the questionnaires including the synthesis of the results
Kick-Off Meaning of the scales, depending on questions; 1 "really not good" … to 5 "very good" Or 1 "not agree at all" … to 5" totally agree" 18 answers (some answers might be partial, hence the sum of answers might differ from the total) General organisation of the meeting Location Rooms Duration Food Organisation of activities The agenda was well dealt with Other (please specify) Welcome package Comments : 2 days could be enough Better presentation of all partners Meeting room too small Good organisation from FREREF Clear, useful and good articulation between all topics Meetings started late
1 0 0 0 0 0 0 0
2 1 5 2 0 0 0 0
3 0 4 1 1 0 1
4 4 3 10 10 4 7 0
5 13 6 5 7 14 11 0
1
Roles and expression of partners 1 2 3 4 5 Everyone could express themselves as they wished to 0 0 2 7 9 I fully understand everyone's work 0 2 1 8 7 The role of each partner seems clear to me 1 1 1 7 8 Difficulties were clearly stated 0 0 1 13 4 Others (specify) 0 0 0 0 0 Comments If you do not agree with one or the other question, please list your reasons here. The role of every partner is very different: small session to discuss in small groups could have been helpful. 3 persons: not a clear vision of everyone’s background. It was difficult for every partner to take part in the discussion and to participate (maybe a problem of English fluency for some partners?) Project advancement The objectives are clear for me I think that the organisation is well adapted to the project I understood well the financial issues and the tool for the financial reporting LLL-HUB
Evaluation Report (Final Version –May 2016)
1 0 0 0
2 1 0 0
3 0 1 1
4 9 9 4
5 9 8 13
51
There are some points left that I did not understand regarding financial 7 0 0 1 aspects. Please, list them below. Why not following the ceilings of the commission for subsistence costs Not so clear about the repartition of task between CVO and EUCIS-LLL (not answered : 8) For me, the articulation of the different WPs is not clear. Here are the 3 2 1 3 pending questions for me. Clear guidelines would have been nice More exchange about the articulation between lab/forum/agora What about Labs after the end of the dedicated WP? (Not answered: 9) I have a clear vision of what I have to do till the next meeting 1 0 2 4 (please specify the main tasks that you have to realise) Starting the labs Finding experts Dissemination in the network I fully understand the importance and challenges of dissemination and 0 0 1 1 valorisation. Here are the first three actions that I intend to do in this area Presentation of the project to the networks, ministries, on website... Comments Good presentations and personal contacts very nice. For the next time : more inputs from all partners (each of them with a specific task during the meeting) Does everyone understand the project in the same way?
2
0
7
10
Lisbon Meaning of the scales, depending on questions; 1 "really not good" … to 5 "very good" Or 1 "not agree at all" … to 5" totally agree" 19 answers (some answers might be partial, hence the sum of answers might differ from the total) General organisation of the meeting 1 Location Rooms Duration Food Organisation of activities The agenda was well dealt with Other (please specify) Comments 2 days might be enough for the agenda, activities were good, but lack of free time during the days, bothered by the fact not everybody attend the dinner excellent organisation 2 the agenda was a bit confused on the role and the presentation of each partner
LLL-HUB
Evaluation Report (Final Version –May 2016)
2 1 1
3 1 3 5 1 1 1
4 4 9 6 8 6 9
5 13 5 7 10 12 9
52
Roles and expression of partners 1 2 Everyone could express themselves as they wished to 1 I fully understand everyone's work The role of each partner seems clear to me Difficulties were clearly stated Others (specify) Comments If you do not agree with one or the other question, please list your reasons here. languages issue for some participants all partners well prepared, more involved good communication process more debating time could be useful some partners are really silent partner
3 1 4 2 3
4 6 10 10 10
5 11 5 7 6
Presentation of the situation in Portugal I found that this initiative is very interesting 1 2 16 The presentation helped me better understand the situation in Portugal 1 5 13 I still miss some key elements and I would like to get more information 1 2 1 2 about them Please precise which ones Other comments Interesting start session concerned only the first 4 levels of education, not LLL. Lots of initiatives are missing Project advancement Project administration and financial issues (WP1) I am rather clear about all the financial issues If not please precise what you are missing I am clear with the reporting issues. I know what to do and when If not please precise what you are missing Still some doubts concerning the calculation of the allocated budget, might be clarified when doing the reporting Labs (WP2) We are up to date in the recruiting process for the Labs members If not, please precise which are the difficulties encountered should we not have all already contacted our experts ? Invitation to other partners from other countries, could be interesting to have a EU-network for this (under EUCIS-LLL ?) We agree that the Lab structure should last after the project Here are some first insights on how we plan to sustain it We think that implementing the Labs requires a multilevel collaboration between local stakeholders Here are some elements on how we have implemented it (or still plan to implement it) We think that implementing the Lab requires a multiregional/multinational cooperation in order to be more successful Please give some elements to justify your position multiregional yes, multinational no : Labs are aimed to be national "requires" is too strong, but it would help indeed FORUMS (WP3) LLL-HUB
Evaluation Report (Final Version –May 2016)
1
2
3
4
5
1
11 7 11 8
2 1
1
1
9
8
2
7
9
1
8
8
2
5
9
3
53
I understand well how to implement the Forum in my territory If not, please precise what you are missing Forum issues have not been clarified at all during the meeting Evaluation (WP5) I understand how the evaluation process is conducted in LLL-Hub and why If not, please precise what you are missing Dissemination (WP6) I understand the key issues in disseminating the LLL-HUB project outcomes and value creation Here are three key actions that I will do when I come back to promote the project changing email signature, newsletter inform colleagues (2) website, create pages related to the project (2) managing social media promote communication between partners General I have a clear vision of what I have to do till the next meeting Please specify the main tasks that you have to realise organising Forums (4) develop Labs (3) financial reports working on dissemination (3) avoid lack of communication between partners writing state of play financial report results on the common platform Comments
1
1
6
8
3
10 9
1
1
7
10
8
11
Krakow Meaning of the scales, depending on questions; 1 "really not good" … to 5 "very good" Or 1 "not agree at all" … to 5" totally agree" 15 answers (some answers might be partial, hence the sum of answers might differ from the total) General organisation of the meeting Location Rooms Duration Food Organisation of activities The agenda was well dealt with Other (please specify) Comments Lack of time (need more meeting time on Monday) (2) Too much time on looking back instead of looking on what’s next One day should have been spend on the organisation of the AGORA LLL-HUB
Evaluation Report (Final Version –May 2016)
1 0 0 0 0 0 0
2 0 1 1 0 1 1
3 1 2 1 0 1 3
4 3 2 7 7 6 4
5 11 6 5 7 6 7
54
Too much time on the Malopolskian meeting Bad wifi connection
Roles and expression of partners 1 2 I appreciated the “tour de table” about the situation in each partner 0 0 country I fully understand where everyone stand in the project 0 2 Others (specify) More easy to distinguish similarity between partner situation (ex. VNFIL) Comments If you do not agree with one or the other question, please list your reasons here. Please, send short report before meetings Not a good “tour de table” from every partners Participation to the event in Malopolska I found that this initiative is very interesting The presentation helped me better understand the local situation I still miss some key elements and I would like to get more information about them Please precise which ones Start-ups : more practical tips to get involved Concrete examples/initiatives to illustrate the debate Good but took too much time (2) Too much focus on politics Not sure what is expected in the project Other comments
3 3
4 5
5 7
1
8
3
0 0
1 1
4 2
8 8
2 4
Project advancement Project administration and financial issues (WP1) I am rather clear about all the financial issues If not please precise what you are missing Date for report 2 ? Not in charge of this task from my organisation I am clear with the reporting issues. I know what to do and when If not please precise what you are missing Missing the overall control (what to do, when, deadlines, delivering documents...) Having the needed information in time is important LABS (WP2) We are up to date in the functioning of our LAB
1
2
3
4
5
0
0
4
8
3
1
0
2
7
4
0
0
3
4
6
We agree that the Lab structure should continue to work during the project (and even after) Here are some activities we are undertaking to have the Lab continue to work after the Forum Keep contact with experts (5) Update national reports (3) Interviews (2)
0
0
2
4
4
LLL-HUB
Evaluation Report (Final Version –May 2016)
55
State of Play Strengthen level 5 in Netherlands and connection between level 4 and 5 Not clear for the moment Forums (WP3) I think we have done a good job with our Forum 0 0 0 6 7 Organising the Forum was quite a job and it took us a lot of resources 0 3 4 2 4 Other comments about the Forum Good event, helped to bring together various actors for exchange views, experience, comments our labs results better than expected. Agora (WP4) I see clearly what we will have to do to prepare the AGORA 1 3 2 7 0 If yes, please precise the first three actions you will implement Selection of 3 good practices – posters (3) Proposition for experts participation Not sure how to proceed Video of learners Communication to be articulate with own communication at local, regional, national and European level. If not, please precise what you are missing The structure: how to make it attractive for our high level experts? Need a draft program Agora image/logo Agora is too vague : need more time to discuss this issue Waiting for support documents Dissemination (WP6) I understand the key issues in disseminating the LLL-HUB project outcomes 0 0 3 8 2 and value creation Here are three key actions that I will do when I come back to promote the project Publish info on Facebook and Twitter (4) Dissemination about project result (3) Promote the AGORA (3) Informing experts (4) Insist to cover what was forecasted General I have a clear vision of what I have to do till the end of the project 0 1 3 8 1 Please specify the main tasks that you have to realise State of Play according to the template (4) Contact with experts National exploitation plan (2) Organising Agora (4) Finalising Lab (3) Forum (3) Dissemination (3) Expected to have to management update for reporting in time Transnational cooperation I agree that we have to make an effort to develop more transnational 0 0 0 5 8 collaboration in the project (not only during meetings) When I come back, I will think about some actions and activities to develop more transnational activities with other partners Here are the next five actions/activities that I will undertake LLL-HUB
Evaluation Report (Final Version –May 2016)
56
More in between updates on what we are doing Should be done in the frame of the project Shared responsibility from partners, especially from management and evaluation Participation in other events from partners of the project Ok for the idea but nothing to implement for the next months Need more opportunities to connect with other partners I think we could use more support or tools to better develop transnational 1 1 3 1 5 cooperation Here are my suggestions Facebook Newsletter Website of the project? Yammer platform? Should have been discussed during meeting time Common platform Not tool-time and participation is problematic EPALE/ADAM – EU social media and dedicated news services Other comments Great potential in the partnership : people are involved and it i a chance to use this cooperation to communicate recommendations
Sofia Meaning of the scales, depending on questions; 1 "really not good" … to 5 "very good" Or 1 "not agree at all" … to 5" totally agree" 14 answers (some answers might be partial, hence the sum of answers might differ from the total) General organisation of the meeting Location Rooms Duration Food Organisation of activities The agenda was well dealt with Other (please specify) Comments Too short to discuss everything on the agenda
1
1
Preparation of the AGORA I see clearly how to motivate my experts:
1
Evaluation Report (Final Version –May 2016)
3
2
2
4 5 4 5 5 7 11
2 1 1 1 1
1 1
Roles and expression of partners I appreciated that every partner could present what they have done so far and what they intend to do next I fully understand where everyone stands in the project Comments
LLL-HUB
2
5 9 7 6 7 6 2
3 3
4 7
5 5
2
11 1
3 1
4 7
5 4 57
For coming …………………………………………………….….. for taking an active part in the workshops…………..
If not, please precise what you are still missing I fully understand which are the goals, key issues and main concerns of the Agora If not, please specify what you are still missing Things need to be discussed in more detail, as the program development I am aware of the work I have to do to participate in the innovative practices contest Here are the three tasks I will start with when I come back Select good practices (6) Look for visual/ posters...(4) Preparation of awards Talk with experts about selected practices (3) Logistic preparation Other comments Project sustainability I am aware of the importance of sustainability at my local level Here are three-four-five actions that I will undertake to ensure a life after the project in my Region Build a local/regional platform Support good examples in Mardin Next Forum (2) Inform experts (2) Dissemination (2) Follow up I have some good ideas on how I/my institution could contribute to the overall sustainability of the project ideas and outcomes together with Eucis-LLL Please list these ideas below Build a platform Combine VET beneficiaries and labour market together Local network on the supporting basis of an analysis of urgent needs Follow up projects (2) newsletter I have some suggestions for EUCIS-LLL, on how to integrate the LLL-HUB structure and work into the other works of the LLL Platform Please list below Seminar after the Agora Labs and Forums
1
Evaluation Report (Final Version –May 2016)
9
4
1
1
5
3
2 1
3 1
4 9
5 2
5
1
3
3
2
3
1
2 3
3 1
4 6
5
1
2
3
4
5
I think the Lab/Forums in my Region could continue to work Here are the critical conditions to fulfil to be able to continue the work of the Lab/Forums in my region Not to lose people with such wide project, we need some tangible results. Working more on a local basis Network Depends of human and financial resources (2) Dissemination
LLL-HUB
1
58
I understand the key issues in disseminating the LLL-HUB project outcomes and value creation Here are three key actions that I will do when I come back to promote the project Monitoring tool Update fb page with best practices Material for Agora Dissemination (4) Contact new organisations Exploitation I have a clear vision of what I have to do to participate in exploitation as presented in Sofia Please list below what you will do when coming back Check option for follow up projects (3) More regional based Think about LLL platform Create national network Transnational cooperation
1
8
2
1 1
2 1
3 4
4 5
5 2
1
2
3 1
4 9
5 3
2
5
4
4
1
5
2
1
3
7
1
1
9
3
4
4
2
I really appreciated the presentation of the "most memorable things" for each partner I learned interesting things from the practices and stories of other 1 partners. I can cite at least one or two examples that could be re-appropriated at my local level: (nothing) I think there are common issues that emerge from the work done during this last meeting before the AGORA. Here are a few of them, to my opinion synergies regarding best practice examples individual offer = vouchers experts and partners role Other comments Ending the project I see rather clearly what is left to do for me before the end of the project Here are the main tasks that are to be done for me: Agora organisation (3) Finalising deliverables Reporting (2) Agenda of the Agora (2) Check future project options Contact experts European network Regarding the project deliverables I know what I have to do and how to do it in order to finish the deliverables I am participating in If not, please precise what you are missing Regarding the reporting I have all information
2
to fill the next reporting ……………………………………………..
LLL-HUB
Evaluation Report (Final Version –May 2016)
59
to fill the final reporting at the end of the project………… If not, please specify what you are missing final reporting need to collect and correct contribution from local partners Other comments
LLL-HUB
Evaluation Report (Final Version –May 2016)
60
Appendix 3 – LABS, FORUMS and AGORA evaluation LABS There was no specific evaluation for Labs; only the questions in the general questionnaires and the interviews after the Forums.
FORUMS The evaluation of FORUMS was done by each partner for their own purpose. The only information that was collected was through #the general questionnaire 2 (Intermediary Vision – specific question about Forums) and the interviews made by Joël Bonamy INTERVIEW AFTER the-FORUM 1. FORUM : One day ? place, enrolling, … 1.1.
What was a success? What did function best? What was best appreciated : + by the participants + by the experts + by you 1.2. What could be improved for a future FORUM? 2. What was new and innovative? + What was new / State of play… + did you organize an evaluation questionnaire? Interviews with participants? + What was innovative: where does innovation lay? 3. Will the Forum pave the way to a permanent LAB? + What would feed a long-lasting group of reflexion? + What should be its mission: observatory – relay – think-tank? + What should be done to make it a reality? 4. How will you share this experience with the other LLL-HUB partners? 5. What are the impacts of the FORUM? + impact on participants + impact on politics + What should be done in order for the impact to be multiplied?
AGORA A questionnaire was distributed at the end of the Agora, gathering the viewpoints of all participants, which were mostly external stakeholders. Here is the synthesis of answers: Meaning of the scales, depending on the questions: 1 "very poor" … to 5 "excellent" or 1 "totally disagree" …to 5 "totally agree" LLL-HUB
Evaluation Report (Final Version –May 2016)
61
54 answers (some answers might be partial, hence the sum of answers might differ from the total)
Organisation of the conference
1
2
3
4
5
Location Rooms Duration Catering Agenda Speakers Debates Contest Thank you for specifying, to your opinion: What was the most successful Good speakers (2) Workshops (13) Keynote by Baert (6) Recommendations and panel General organisation (3) Networking (5) Experts experience (4) Results of the reports Experience from the fields Sharing experience (5) Debate (2) Information about some courses Feeling that I belong to a community
0 0 0 1 0 0 0 0
0 0 0 2 0 0 4 1
1 1 2 8 6 5 7 8
14 15 24 19 19 27 24 18
39 35 28 21 21 19 18 16
What could have been improved More practical examples More interactive communication between participants Lack of sense of responsibility for the future Need more time for workshops and for best practices (2) Last panel (2) Some speakers (2) Seats during breaks Presentation supports available for the participants (2) Overall organisation One more day for networking and cultural activities Workshop and presentation of projects in it (3) Signalisation of workshops Giving the idea that all countries are in similar situation Presentation of best practices and real success stories of LLL More info about education system in other countries More opportunities for non EU members Interaction on Social network Comments I’d like to have a follow up about assessing non formal and informal learning in a formal education program system LLL-HUB
Evaluation Report (Final Version –May 2016)
62
Main room good but do not ease interaction (2) Congratulations and thank you (9) Excellent (2) Best practices were too fragmented in Workshops, it could have been interesting to have a Best practices market People who don’t have a good command of English should not be a part of such event Content of the conference, interest regarding the LLL-Hub project issues and the LLL policies in Europe
1
2
3
The conference was inspiring and opened new roads in LLL I now have a better view on some interesting practices regarding the implementation of LLL policies in different European Countries I think that the conference has opened innovative roads to enhance the convergence off LLL policies in Europe Fostering the convergence of LLL policies requires sustaining the LABS’ life in each Country. I think the conference helped me better understand the LLL-Hub issues, outputs and findings How to go further? I could envisage the following actions in my Country/Region/ to take into account/propagate the outcomes of the AGORA and of the LLL-HUB project: Join forces! EPALE Maintain labs and forums (3) Develop local LLL initiatives Dissemination at national level (2) Include intergenerational learning activities in our career guidance program Propose activities in family training for learning proactivity Promote LLL guidance from cradle to grave Putting more effort on the recommendations and real life practices No concrete idea but inspired to speak about this theme with stakeholder on my territory. Dual education I would like to participate in a further project on the same kind of concerns If yes, please give us your e-mail! e.babuleva@abv.bg carine.gyeryak@insp.cfwb.be Thomas.Ischer@menon.org.be cko.mihailova@abv.bg i_stoytcher@yahoo.com celiomangurs@ipt.pt vkisima@gmail.com nina@spithost.nl belska@uppk.pl Amelia.marques@esce.ips.pt mbenero@unex.es fe.martinezsainz@gmail.com gerencia@cenavanza.com zozo.komarova@gmail.com alexsis1987@abv.bg heeringe@euronet.nl pilar@esotex.com r.koclega@op.pl christa@heeringconsultancy.nl d.cicovic@rep.srpska.eu martinaneedham@donegaletb.ie roger.henssen@zuyd.nl Here is what for me was the most interesting in this conference Workshops (2) Networking (6) Recommendations (2) View of the top floor H.Baert (2)
0 0
1 0
10 23 14 8 25 17
0
3
14 20 13
0
1
9
0
1
11 19 16
LLL-HUB
Evaluation Report (Final Version –May 2016)
4
5
20 19
63
Test a methodology (labs, forum, agora) that ensure good tools to reply similar project The presentation of the discuss-community.eu platform Time of debate Sharing experience (5) Focus on learner needs Learning bring value Here is what I missed most in this conference Lack of interaction Focus on higher education and the people delivering not only workers but innovation (?) Using European tools (europass, ecvet...) Young people experience Lack of real proposals More partners More visual tools on real situation Other comments More time for group discussion Philosophy of the project to be developed in a common core and specific tools Make LLL more fun Hope that the recommendation will be followed Thank you
LLL-HUB
Evaluation Report (Final Version –May 2016)
64
Appendix 4 – Report from the external evaluation REPORT FROM THE EXTERNAL EVALUATON Role of the external evaluation and main actions of the external evaluation For assuring the quality of a complex project as LLL-Hub, evaluation is a necessary component. The internal evaluation and quality plan has been conducted by FREREF and it was complemented by an external vision. The following report presents the external evaluation that has been conducted through: - Participation to the general meetings: observation and partial report - Reading of the reports - Interviews with the management of the project : at the first phase of the project and after the Agora - Interviews of the partners after the forums - Interviews during the Agora
Content of the external evaluation report The report deals first (part I) with how the forums unrolled and the method that was adopted; then (part II) it questions the process that began and the impacts expected. The third part describes the lessons and recommendations and the follow-up of the forums and the methodology adopted to collect innovative practices. Part IV discusses the lessons from the Agora: the unrolling and the peer dialogue and questions the sustainability of the outcomes and the perspective for acting for LLL at the European level.
I. Method and unrolling of the FORUMS I.1. Unrolling: a common methodology that worked I.2. Main success: exchange and dialogue between diversified experts and stakeholders II. Process and Impact of the FORUM II.1. What was set in motion II.2. Which Impact III. From the FORUM to the AGORA III.1. Outcomes and Lessons from the FORUMS III.2. Innovations and Recommendations III.3. From the States of Play to the collection of Practices IV. Lessons from the AGORA IV.1. Unrolling and the success of peer dialogue IV.2 Sustainability and Acting for Lifelong Learning
LLL-HUB
Evaluation Report (Final Version –May 2016)
65
I. Form and method of the FORUMS Seven forums were organised during the period March – May 2015 + one in November 2015: o 27 March 2015, LLL-HUB FORUM – Bulgaria o 13 April 2015, LLL-HUB FORUM – Poland o 23 April 2015, LLL-HUB FORUM – Portugal o 11 May 2015, LLL-HUB FORUM – Belgium o 13 May 2015, LLL-HUB FORUM – Spain o 28 May 2015, LLL-HUB FORUM - Netherlands o 11 November 2015, LLL-HUB FORUM– Turkey-Mardin Interviews by the external evaluator: 10 April 2015) 15th of April2015
Bulgaria: Ludmil Kovachev Poland-Malopolska : Malgorzata Sobolewska
11 May 2015
Portugal: Etelberto Costa
20 May 2015
Belgium-Flanders: Hannelore Audenaert
15 May 2015
Spain-Cacerès: Raúl Iglesias Durán
4 June 2015
Netherland: Hans Daale
16 December 2015
Turkey-Mardin: Murat Kaya
I.1. Unrolling: a common methodology that worked The interviews show that the Forums went well, following a common methodology that the partners felt well adapted. Moreover it went out so well that as the participants and the organizers said they lack of time to go through the debates. Let examine the elements of the unrolling of the FORUMS
Timing
The meeting was organised on the basis of one day (Bulgaria, Malopolska, Turkey) or one half-day (one morning: CVO, Leido, Spain). It t was a one morning session: the risk was too high that the experts leave if it lasts in the afternoon. Actually the discussion had been so intense that it finished only at 3 p.m.! The methodology is good, but participants need more time to talk. If another Forum was organized, the participants would like a duration of two days, in order to have more time for reflexion and formulation of recommendations.
Participants : from 12 to 60 participants
As it was recommended, the participants were chosen to give a good representation of LLL stakeholders: official institutions, formal and non-formal education and training organisations, and representatives of the economic world and labour representatives. One day with 32 participants. National stakeholders (national authorities: national Education and Sciences Ministry, Employment and Labour Ministry), Regional authorities, local authorities (municipalities), LLL-HUB
Evaluation Report (Final Version –May 2016)
66
Labour and employers organisations (One of the biggest union and one of the 6 employer organisations), Education and VET organisations (universities, schools, education providers).

Plenary and workshops
The common standard methodology, each LLL HUB Forum was organised in one-day event centred on the 4 (four) main sessions: Session 1
Session 2
National Frameworks for lifelong learning: towards flexible pathways and comprehensive education and training systems Lifelong learning actors: taking the jump towards learner-centred systems
Session 3
Lifelong Learning communities: partnerships and shared responsibility
Session 4
Policy recommendations to improve the implementation of Lifelong Learning
Central question: Are education and training enabling flexible pathways for learning? Where are we heading to? Learning focused on the trainer or learning focused on the trainee? Have partnerships and networks helped/added any value to LLL? What is their role in what concerns shared responsibility? What should be suggested in what concerns policies about LLL?
Around three quarters of an hour were deserved to a general discussion in each of the sessions. Discussions of the FORUM took place according to a Methodology which had been developed specifically by our project partner CVO Antwerpen with the help of the Vrije Universiteit Brussel. The forums followed a general pattern and were divided in three parts: general introduction; thematic discussion groups; conclusion and recommendations. After a general introduction experts discussed three broad topics in separate sessions: 1) National Frameworks for lifelong learning: towards flexible pathways and comprehensive education and training systems, 2) Lifelong learning actors: taking the jump towards learner-centred systems 3) Lifelong learning communities: partnerships and shared responsibility. At the end of the day all experts came together to present their conclusions and policy recommendations. Four discussion groups in parallel along the four topics: Validation of non-formal and informal learning Adult learning education and labour market barriers between actors and sectors Themes : Theme 1: How to build a learning outcome approach through more flexible learning pathways Theme 2: How to make citizens acquire skills for their personal fulfilment, social inclusion, democratic participation and employability LLL-HUB
Evaluation Report (Final Version –May 2016)
67
Theme 3: How to engage citizens, including the most disadvantaged groups, in lifelong learning by providing quality and attractive learning opportunities at all ages Theme 4: How to pool the expertise and resources of the different sectors concerned for coherent and comprehensive lifelong learning strategies
Moderators
Some partner indicated the importance of moderators in order to facilitate the debate and to produce a report of them. In each group: + an expert was selected to start the discussion, the other could add their observations. + One reporter for the final document (these reporters play an important role to collect and proceed the information and debates). In the end we had the conclusions, the state of play, the synthesis.
Conclusions and questionnaire The formulation of policy recommendations by the three parallel groups. They validated 15 recommendations of the national report and added 10 more. The 25 recommendations will be integrated in the national report and be communicated to the policy makers. We can foresee that one out of three of these recommendations will be implemented.
I.2. Main success: exchange and dialogue between diversified experts and stakeholders On what main aspects lies the partners’ common feeling that the FORUMS were a success?
The sense of a BEGINNING of a process
The first aspect of the success of the forums lies in the intensity and quality of exchanges. The most innovative is for us the methodology and the structure of the forum: the way to make the experts think and share opinions; the way to put on the table the topics and make them think and share. It is often the first time that experts from so different fields discuss together and involve themselves It is the 1st time that representatives of non-formal and informal learning are speaking on the same level with representatives of formal education. They were all really involved and they consider themselves as “experts of LLL”. I did not expect such an involvement of the experts. It gave to the organisers the sense of a beginning of a process. The best point of satisfaction: the experts wanted to cooperate, share opinions between them.
The quality of the exchanges
Exchanges between participants from different fields that were not used to discuss on a common level.
LLL-HUB
Evaluation Report (Final Version –May 2016)
68
This FORUM gave opportunities for communication, discussions, joint work and consolidated common findings in the form of working groups which led to results at different levels of stakeholders representation – National authorities (Ministry of Labour and Social Policy, Employment Agency), Regional Authorities (Regional Inspectorate of Education, Regional Employment Agency, District Administration), local authorities – Municipality, social partners, schools, VET centres, carrier guidance providers, Universities. Participants actually represented almost all “actors” in the field of LLL as a “horizontal” dimension, on the one hand, and a “vertical” dimension covering different levels of policy making and policy implementation, on the other hand. - to be able to meet together people from different environments : those on the ground (training institutions) and the deciders (policy-makers from education, labour and social ministries). - they could show their views, thoughts, opinions, ask questions, … What is felt as very innovative is the fact that exchanges could mix bottom up and top down approaches This is an innovative approach of exchange of information and does not happen very often. Such an arrangement of different levels of representation and joint interaction during the working groups is very useful for evidence based policy making as a combination and synchronization of bottom up and top down approaches: - the bottom up policy making was supported through the input made by attendees acting on a local and regional level - and the top down approach was performed by the provision of most recent information on national developments by attendees representing national authorities. The feelings of the organizers are confirmed by the answers to an evaluation questionnaire that the participants filled at the end of each forum. A feed-back questionnaire has been distributed and the data will be processed: In general they thanked for the organisation of the Forum and for the opportunity to discuss together. Even if they knew each other they appreciated that a boost was given to a general discussion. They are most interested in the AGORA: they think it will give a new dimension to the discussion. What was best appreciated: a questionnaire was circulated and will be processed. They express the possibility of exchanges in experiences. I’ll try to have a whole day instead of a half day: people had to leave by 1 p.m. and the timing was tight and we had to rush. Even if the discussions worked well and we had two very good moderators.
A reinforced validity of the States of Play
The interviewees feel that one result is a reinforced validity of the States of play The possibility to combine two approaches: a top-down approach (from the national and regional authorities) and a bottom-up approach (from experiences and ideas of local authorities and education providers). The possibility to share experiences: how to finance LLL and how to integrate LLL objectives in the regional strategies (5-Years planning).
LLL-HUB
Evaluation Report (Final Version –May 2016)
69
II. Process and Impacts of the FORUMS II.1. What was set in motion The forum is not viewed as a one-shot event. It is one moment of a whole process.
Before: meetings with experts
A forum is not only a meeting with people invited. It requires to work before with the participants. It was the idea of a LLL-Lab regrouping experts able to discuss and evaluate how the European initiatives and directives about lifelong learning are designed, implemented and evaluated at local, regional and national levels. Hence before the forum, each partner had meetings with experts, stakeholders, responsible of institutions. we worked with the stakeholders. This helped to make a realistic national report and it completed the information that we collected mainly on internet. There were many meetings before the Forum. They were done with experts to discuss the themes. The State of Play is a sort of summary of these meetings. Since 2012 we tried to have the possibility to make work together the responsible of the formal Education (Ministry of Education) and organisations for non-formal and informal learning. That was the idea of the project LLL-HUB. It is not easy to mobilize experts on a voluntary basis but we had experts of high quality that are linked with us on collaborative works or through friendship. The meetings organised aimed at preparing: - the state of play: Lab meetings: Before the forum, we had 4-5 meetings in order to prepare the state of play. a) kick off with presentation of the project and its goals and schedules b) discussion on contents, methodology and tools to be used c) actual state of each case, for control concretion and clarifying doubts still remaining. - and the forum: There were two meetings three months before the forum: one about the themes of the national report and another one on the organisation, methodology and wishes of the forum.
During: dynamics of the exchanges, new unexpected aspects
The preparatory meetings allowed new contacts, and helped to precise some aspects of the implementation of LLL policy and to adjust the State of play. They were useful to prepare the possibility of a LLL-Lab. Hence the forums were moments when people met before and others can confront their views. These people are from very different fields. Experts are from different fields and it is not easy for them to discuss. The same topic is viewed very differently in higher education and in schools, for example. They break the habit of discussions between the same people, other categories take part to the debate.
LLL-HUB
Evaluation Report (Final Version –May 2016)
70
The idea of meeting and discussing and finding solutions was not new but it was new to include people that are not usually part of these discussions.
The discussion is deepened, it goes beyond conventional quantitative data; it compels to discuss qualitative aspects, i.e. concrete implementation of LLL. It was new also to discuss qualitative aspects instead of the usual quantitative results. These people introduce new fields that were never taken into consideration before, because they are outside of institutions’ competences. It is the case of socio-cultural organisations that act informally on public as migrants. The role of socio-cultural organisations: - they were more present than in the State of Play. - they are organisations that are not directly on education, but for example on work with migrants: they have a library where migrants come, they organise a “skating for migrants”; it is during these moments that things happen: it gives them the opportunity to listen to their questions and address them. They play an important role for non-formal and formal education: they help to reach the target groups and they help to pass thresholds or barriers to more formal education.
After: a new forum, experts’ meeting, platform …
The dynamics that was launched by the forums should not stop. The basis for maintaining exchanges between the people that participated are already present. It is in coherence with the idea of LLL-Labs that “are intended to reflect on how to improve transversal dialogue and cooperation, to break down barriers between sectors, to foster communities of values and practices based on a shared meaning of lifelong learning”. After the forum, the participants agreed to meet every 3 months in the LAB. There is a will to continue discussions on practical solutions. Some of the departments of the regional government use some of our findings. There is a desire to maintain the contact with the providers, get their opinion. But nothing is already decided. It could be an opportunity to organize meetings where participants can share what they have learnt. The Forum has created partnerships and it could be an effective network. Other actors should still be convinced to join the debate. It was one of the recommendations of the debate: to include in the discussion responsible of the labour market: Ministry of Employment and employers’ organisations. Hence a second Forum will be organised next November. The theme will be: what can we do in non-formal and informal learning for the labour market. Timing and modalities will be different in accordance with the context. The next forums will be different: In one region, it will be organised by 2 enterprises and the Mayor In another town: it’ll be organised by LLL researchers; the question will be: what cases in order to improve your research. Each one will have a different methodology. LLL-HUB
Evaluation Report (Final Version –May 2016)
71
The notion of LLL-Labs is important, but its different functions will not develop linearly, one after the other. It should be contemplated a permanent process where the state of play is not a diagnostic but a basis to be discussed continually in order to register the progress of LLL implementation. There is no linear progression from Observatory (the elements for the State of Play), to Relay (the discussions with stakeholders and experts) and Think-tank (Forum and permanent Lab). Instead we must keep the results of the state of play open to the public and it could be criticized or completed all the time. It is not obvious. Means should be found in order to mix authorities and representatives of the civil society. It is necessary that the authority recognize the legitimacy of the process. It is difficult for the Civil Society to make recommendations; but the National Agency for Qualifications was inside the project, they know what we are doing, the President was at the Forum. We could discuss the conclusions. What we have done is a process step by step allowing propositions and suggestions to be formulated. After the forum, no new meeting or FORUM are foreseen, because this activity and the conclusions are not taken into account by the administration authority. For instance, there has been intense discussion on supporting education professionals, but this activity is not recognised by education directors or by administration authority. The forum was a first step but it is a long way to be recognised by authority. This process should be understood in the long term, far beyond a European project; but what will be its funding? The LLL-HUB project is entering a phase of Exploitation. We want a platform. The real question is to know if we can proceed in the future: we’ll have more forums until the end of the year and in 2016. we would like to open a permanent discussion on LLL, but how to do that without money!
II.2. Which Impacts At this stage, the interviews could not measure more than immediate perception of future impacts.
Communication
The partners mention the presence of media and the dissemination through their own means of diffusion The presence of Media: radio, press, TV Dissemination : - internal: the report, the newsletter for teachers, the Staff meetings - external : a newsletter and a “recommendation Box”: it is a nice way to present the recommendations. We are able to put the discussion in the general public: a good platform of dissemination through our magazine for learning. The context of learning is changing: it is important that other organizations and institutions can look at what we have achieved and make their propositions.
On participants
LLL-HUB
Evaluation Report (Final Version –May 2016)
72
The possibility to meet in a network the participants seems to be realistic. But a LLL-LAB will need efforts to bring together those who could not participate to the forum, and to open a platform of permanent discussion. impact on participants: they were involved and interested. They appreciated to show what they think and that their experience is considered as important. it was a good opportunity for each participant to know the others. They can join in a network. the question is to widen the network : 50 persons were invited to participate to the forum. 32 came. The others are willing to participate. The Lab meetings will be the opportunity for them to participate. organize the LLL platform the Forum was a good signal for the adult Training providers of the necessity to incorporate multiple players and multiple sectors. It will need change of behaviour of the main actors. Public sector: entering the process with the National Institute for Training. They provide the dissemination in their newsletter and web. Researchers-academics: they realize that they should be working together with us. I am trying to have some of them to write papers for the platform. Enterprises: they appreciate the network, the contacts with public institutions; but they don’t see the results and how it will make them able to make money from the results. It is the concretisation of long run efforts for one partner or for another a new phase. The Lab as a structure : it concretises our efforts of these last 5 years. The theme of LLL is new in our region. Most of the people don’t have a clear idea of what is meant by LLL. Hence the project is a beginning for us.
On politics
The impact on decision makers depends on how they are organised. It’s forthcoming: the recommendations will be presented to policy makers at the national level at the regional level, the impact is more concrete because the representatives (Governors, deputy mayors) were among the participants. in the national report, it was underlined the shortcomings on Adult Education and on Learning on the job. This was commented during the Forum. Some authority wants to apply the recommendations and to develop it in their strategy. However the immediate impact is the introduction of new subjects: the concrete implementation of LLL policy and the question of informal learning. impact on politics: the question of implementation the impacts depend on the Ministry : will it accept to be part of the framework, and to introduce non formal and informal learning. Of course it should be proved in medium and long run. If the forum were more frequent, it could have an impact on politics. It will be the Agora because it will commit a wider audience. We should not overestimate our results and the most important now is to disseminate them.
On partners
For each partner, the question is the impact of the forum on the organisation and the durability of the reflexion on LLL and its implementation. LLL-HUB
Evaluation Report (Final Version –May 2016)
73
How much the forum and its preparatory works will allow the implication of the actors? It is still uncertain even if one partner is launching a LLL platform. the impacts will come if we do not count only on politicians. It will take time. -People are representative of their institutions or companies. For example the impact on the regional public institution will be very short: two experts came and the impact will be more on an individual level than on an institutional level. The institutions are not much involved. But progress was made: - we got the involvement of experts : they understood our aim and they discussed and shared ideas and conclusions. - the Chamber of Commerce is supporting at a high level the project. - the conclusions of the Forum will be included in the national report. A LLL Platform will be set up: it is a real outcome of the forum and insurance of the prolongation of the debate and coordination of LLL perspectives.
LLL-HUB
Evaluation Report (Final Version –May 2016)
74
III. From the Forum to the AGORA III.1. Outcomes and Lessons of the FORUMS As the external evaluation interviews took place one week or so after the organisation of each forum, it could not benefit from the report that each partner made out of their observation and out of the evaluation sheets filled by the participants. Hence these reports are reproduced here. They confirm that the forums were successful in achieving their objectives and they draw for each country some perspectives for organising and maintaining the LLL-LABS. Bulgaria – The evaluation shows that the FORUM has reached the objectives defined within the project. The proposals made for organizing regular forums at national and regional level is additional evidence that the format of discussions implemented was adequate. Information about the FORUM was published in printed and electronic media resulting in numerous inquiries and requests to send the final version of the National State of Play to many institutions, organizations and experts in the field of LLL. This is additional evidence about usefulness of the work done in the frame of the project and the quality of the event. The composition of participants was also assessed well. The proposals made about broadening the scope by involvement of employers and cultural institutions3 will be taken into consideration in the future work of the Bulgarian partners. Concerning the duration, the participants agreed that the time was not sufficient, so this should be discussed by project partners when finalising the Forum Methodology, as a project product which could be exploited in the future. Belgium - CVO, as organiser, is very pleased with what they have achieved by organising the Flemish Forum. A fresh wind now goes through the Flemish LLL landscape. It is now up to CVO and other stakeholders to keep the topic ‘warm’ so we feel there should be some kind of follow-up sessions. Poland - The public part – „Strategic panel” was a very good idea as it gave a possibility for participants (training institutions) to ask questions directly to the representatives of authorities/policymakers (including EU Commission), however more time should be allowed for it. Regarding the workshops, the experts were divided in two groups, each discussing two topics. It would be better to have 4 groups discussing 1 topic each, and additionally make the experts switch places during workshops so as to let them respond to every topic. Portugal – The overall work is assessed as successful due to the good team work, and excellent results of groups’ interaction of the groups. It was well recognized by all that different perspectives like items such as “intergenerational” and “networking to other projects” are very important and justifies the continuity of a project like LLL-HUB. It was recommended to ensure the sustainability of the strategy to implement in the country and strengthen the local actions of proximity, providing answers that permit people in the territories, involving policy makers and leading them to share new ways about the LLL issues. The Education and Training pathways/programs shall be adjusted to the needs of labour market and the adjustments should have the involvement of political actors and stakeholders especially employers, higher education and especially in vocational education and training. Recognition of prior learning and vocational guidance are key. Public policies should envisage the needs of two types of unemployed workers: graduates entering the labour market and long term LLL-HUB
Evaluation Report (Final Version –May 2016)
75
unemployed workers that are less educated and qualified. Access to guidance should be provided at the public level. The age limit should also be reconsidered in today’s socio-economic contexts. Netherlands - Leido is satisfied with the Forum, overall, the outcomes and recommendations – and with all critical comments – and the suggestions, most of them with the focus on involving the employers. The outcomes are the basis for starting-up in 2016 the National Platform on LLL. The Ministry of Education is willing to have this initiative incorporated in the next strategy on LLL. So, the most important conclusion is that this project is helping the stakeholders to reach some important goals. They will, however, need to involve the world of work in coming activities. The National Platform on LLL will foster individual approach as well as promote the use of NLQF. For the next Forum, efforts will be made for greater general publicity and involving wider audience, making it possible for others – maybe not a real expert yet – to participate and to learn about the project and Leido’s goals. Spain - The time was a bit short to discuss all the topics in a morning session. In Spain, to make a full journey session is not a solution but it could be done organizing the forum in two morning sessions. The experts’ contribution was the most valuable aspect in the forum. Recommendations are valuables, but must be integrated into different fields
III.2. Innovation and Recommendations In November 2015, the partners were invited in a meeting in Sofia to express what were for them the most innovative aspects of the process of Labs and forums. These reflexions are reproduced and followed by the key recommendations expressed in their forums. It should be underlined that the main innovative aspect is the involvement of experts coming from diversified fields; it is considered as a start for long term cooperation and it gives sense to the recommendations. Bulgaria The most innovative aspects: An opportunity to bring together experts and stakeholders from different sectors and it produced an efficient dialogue. - All participants gained an overall systematized picture about LLL state of play in the country. - The Forum created an opportunity for experts to exchange information, good practices, and experience and establishes contacts for future cooperation. Recommendations: - The FORUM confirmed that the content of the National Report on LLL State of Play adequately presents the situation in the country and that the Recommendations defined in it correspond to the main challenges faced in the process of LLL policy development and implementation. - The recommendations defined at the Forum enrich those included in the National State of Play. Belgium The most innovative aspects: LLL is very fragmented between ministries of Education, Social affairs, and Labour. This is running against cooperation between formal and in-company vocational training. There is a need for flexibility.
LLL-HUB
Evaluation Report (Final Version –May 2016)
76
Recommendations - Definition of LLL: lifelong and life wide learning focus on employability; - Goal: 15% participation in LLL and 6,8% (Flanders Outlook 2013), but a lot of learning activities are not in statistics and are invisible: learning in socio- cultural organisations, training organised by economic sectors, statistics Eurostat: for 25-64 year 25% of second chance learners in adult education < 25; - Fragmentation in sectors for LLL in Flanders: departments of education, culture and employment. This plays against a coordinated vision policy on LLL and cooperation between different policy departments: a more proactive policy on LLL is advisable; - Policy on EVC/EVQ is inadequate. The national and European framework is advisable; - Validation of competences: HRM culture still depending on diploma and certificates; - Obstacles for a learner centred orientation: legislation and financing mechanisms in adult education. More financing necessary for programs and group learning, flexible programs for learners, custom-made programs in cooperation in with economic sectors; - Programs in socio-cultural organisations can be more flexible but competences are not recognized - Provisions for disadvantaged groups are sufficient, but need for: flexible custom-made programs and initiatives for activation, more practice-based programs, in-service training, dual learning programs, services for learning advisory for adults, and activation of learning in social services, health services, employment services; - Reform of existing incentives for learners (see "time to learn!") - Partnerships are necessary. But the fragmentation of LLL sector in Flanders makes that the cooperation and partnerships that exist are hindered by policy on subsidies and legislative rules of different departments (education – culture – employment). For best practices: bottom up initiatives based on regional needs, and partnership can be stimulated by co-location; - Several initiatives in Flanders on LLL and building partnerships but all of them disappeared too quickly! Flanders needs: a new vision on learning based on an analysis of what kind of learning our society really needs, subsidies should be allocated according to needs of these environmental analyses, a sustained communication plan on LLL ( in the past: Life learning week), and a LLL-focus in every policy department (employment, social welfare, education,…). Netherlands The most innovative aspects: Holland is considered as a country of high level LLL. Actually there is only one civil servant dealing with LLL in the Ministry of Education. There is no platform on non-formal and informal learning and their link with formal learning. The questions of LLL for aged and unemployed people are not dealt with (for instance there is no subvention for those who are 10 years before retirement). The situation is also confused in the case of fighting poverty: the different ministries have no contact between themselves. Recommendations The recommendations made to address the Ministry of Education are highly important as this organisation is the most crucial stakeholders when it comes to the improvements in the addressed area of lifelong learning. Portugal Recommendations The main outcomes are the key recommendations that include the following: - Increase the discussion at the political and “operational”/organizational level in order to promote more involvement of the community; - Promote the sharing of knowledge, practices and meanings by using platforms and fostering communities of practice; - Reinforce the recognition of prior learning and extend the age of access to training; LLL-HUB
Evaluation Report (Final Version –May 2016)
77
- Provide the needs of unemployed young graduates who cannot find work, the low-skilled unemployed and the graduates long-term unemployed age up 35 and to do the adjustment of training to market needs for this people; - Provide mixed solutions to more adults in middle age to improve education, otherwise in two decades people are still in market without qualifications; - Continue to improve LLL with flexible pathways with the involvement of political actors and stakeholders especially employers, higher education and vocational education and training and also in providing the training of trainers and teachers constant adaptation of the new realities; - Introduce partnership between regional authorities (CIM) and national authorities - focus in reduce the deficit qualification and certification of local people and promoting their employability. - Recognize the need for increased social skills on young people and promote LLL in formal, informal and non-formal settings, from childhood; - Assure the recognition of policy makers and businesses of the importance of promoting training in the workplace more closely, so that young people or adult learners can have a proper training and a tutor to assist them in their training. - Open up education through new technologies and promote the acquisition and development of skills throughout life recognizing the benefits for the employability, competitiveness and innovation in Portugal. Spain The most innovative aspects This project was a complete challenge for us because we are not a training institution, even if we do some VET. We realised that the methodology for Labs (with three dimensions: observatory, think tank, relay) works well and gives complete results. Hence we’ll apply it to other fields: - labour integration: as we don’t know what companies need, we applied the methodology and we can now do training. - introduce in formal learning some aspects. Recommendations The main outcomes of the Spanish held Forum includes the policy recommendations and experts’ involvement in the discussions and shaping these recommendations. - Recognize the need for increased social skills on young people and promote LLL in formal, informal and non-formal settings, from Turkey The most innovative aspects 1) The refugee issue: they are 60 000 in Mardin (for a total population of 800 000). They live in three camps. In one tent live 6 families with a common place. LLL is a very important issue: the necessity to learn Turkish language in order to understand each other. 12 cultural education centres have been set up. They learn out of daily problems (example of how to make soup out of almond). For those who are not living in camps, a great survey on 40 000 homes has been organised in order to know what they need (go to school, how to live without assistance, …) and what NGOs could provide in terms of vocational training. 2) Out of a “romantic” view: if you want to teach people, they are expecting something. As you don’t give money, you should prove that they will get something after the training. If not, it will appear “romantic” to this people who need to eat and live immediately with 6 children or more. We experienced to reveal the skills (women as woven artisans that produced and made marketing of the product in an Italian fair). But the problem is sustainability: there has been no follow-up of the marketing by the governorship.
LLL-HUB
Evaluation Report (Final Version –May 2016)
78
III.3. From the States of Play to the collection of Practices The view on the perspectives after the FORUMS Just after the organisation of the forum, it was difficult for the partners to have a concrete idea of how concretely they will proceed in next steps. There is a common perception that the exchanges should go on and deepen in each country. Between the FORUM and the AGORA, it could be fruitful to organise new discussions or a lab. It is a challenge, because the participants say that they appreciated to meet in the Forum but they did not express a desire to meet again. The experts will discuss these documents -in two months for the next step. We will give them the support for organizing their meetings, but they have the responsibility of their meetings. Four meetings are foreseen in different regions. In the end an event will be organized. There is some hesitation on how useful is the launching of an LLL-platform in order to give to participants the opportunity to maintain discussions. Now the most important thing is not to launch a new platform of discussions, it is risky when other platforms exist. We will try to use the platform (a working group) of the Ministry of Education or the one of the National Agency which has a platform for Adult Education and Training. The experts were interested to hear about it and wanted to know how to register. The message of the forum is: go on and start a network and a platform. Before launching a platform we want to have the opinion of other organisations during the second Forum. The objective for 2016 is to propose a think-tank at the disposal of the government: the National Civil Society Platform for LLL. But the feeling is that what has been done up to now should be followed in the future and should not be stopped by the end of the project. In that sense most expectations are on the Agora: will it boost the possibilities? Will it launch a new sustainable dynamic? What has been done this year should be reproduced over the coming years. It could be fruitful to have new meetings and discussions after the AGORA, but no participant express that desire. The LLL-HUB project should push and motivate people for a permanent LAB. But after the project, when the research comes to an end, where will we find the money, how will we mobilise experts? The project is a beginning.
The view from the reading of the innovative practices and learners' stories The field research part was intended to add "real field experiences" to the desk research work that was merely collecting data and figures from reports. Each partner provided with a lot of innovative practices, learners' stories, case studies, etc., but not much time was devoted to a cross reading of them. This is why the evaluation team decided to present work done from a comparative and cumulative reading of all the documents that was synthesised in the concept of LLL Pathways at the Sofia meeting. This information is given in Appendix 5.
LLL-HUB
Evaluation Report (Final Version â&#x20AC;&#x201C;May 2016)
79
IV. Lessons from the AGORA IV.1. Unrolling and the success of peer dialogue Unrolling The Agora in Breda was the moment when the whole work of LLL-HUB has been presented and discussed. With the active participation of national teams composed of partners of each country with their experts, plenary and workshops were designed in order to facilitate the exchange of practices and policies. The States of play, the comparative report, and a presentation of 21 practices were distributed and were a basis for the discussions. The plenary has mixed a keynote (Herman Baert), interventions of partners and interventions of reprsenetatives of the European Commission. It gave value to the project and agora has well replaced the European dimension of the project and its recommendations. Compared to the unrolling of the Forums, the most innovative part of the AGORA was the presentation and discussion of innovative practices and case studies of learners. 21 practices were chosen by the national teams among 54 innovative practices presented and 27 case studies of learners. They were the basis of the three main themes discussed in the three workshops. For each of them, a presentation of three case studies was followed by a discussion inviting the participants to share their own experiences and views of the theme; then they were invited to contribute to the development of a set of policy recommendations. - “Implementing lifelong learning strategies on the territories”: Efficient implementation of lifelong learning strategies at territorial level Cooperation between formal education and business - “Enrolling people in their lifelong learning pathway”: Raising awareness on lifelong learning and reaching out strategies Mentoring and lifelong learning guidance - “Recognising and valuing LLL Pathways”: Supporting learners in their life transitions Recognising prior learning and taking individuals a step further. The last day, the recommendations of the workshops were presented in the plenary and a general synthesis was discussed. Peer dialogue The presentation of the experiences was well appreciated and gave floor to intense discussions: They received positive feedback from the experts of all the countries partners Maybe there should have been a little more time for the presentation of best practices during the workshops. Nevertheless the fact is that discussions were indeed ignited after the presentations and there was room for participation of the audience. Participants would have liked to have more time for discussions:
LLL-HUB
Evaluation Report (Final Version –May 2016)
80
The workshops effect should be higher and the presenters of best practices should have more time to present all the different objectives, aims and challenges of each project they were presenting. The innovative experiences are not mainly seen as transferable but as a means to enlarge the perspective of LLL: The workshops had many participants ready to contribute Hard to know about the transferability of what have been heard during the event, but it widens the scope of our perception of LLL in order to impact more people by doing our own project. It was suggested that life stories should be on the website. The way learners describe their experiences is alive and very interesting, hence they don't need to be corrected in terms of English, The presentation of innovative practices gave place to a Contest. The winners were chosen by the votes of the partners. This initiative motivated the partners and it was so successful that it could be reproduced in future projects: The LLL-Hub Awards really motivated the winners. It was suggested that because during the workshops participants got to know only 3 out of 21 total best practices, it would be better maybe for the future to present 21 posters for 21 best practices. Stating the fact of how successful the awards were and that the winners seemed really happy, the LLL-P will organise other LLL Award Contest.
What came out of the exchanges on practices: The participants’ points of view insist on the fact that getting knowledge of other countries experience enlarges the conception of LLL. It is important to get out of one’s comfort zone Go beyond what each partner has done Go beyond thinking that what we did was better than what has been realised by others It is the core challenge of this research because it is very large and it includes all the problematics of LLL. The methodology followed helped: from one aspect, especially one that is highlighted by an innovative practice, synergies are derived through the analysis and they are discussed in the agora. Initiate and share more practices to get more evidence from the field that LLL works This way of conducting projects should be valorised: establish together the state of play of LLL, discuss it and find the synergies to develop concretely LLL. It widens the scope of our perception of LLL in order to impact more people by doing our own project. Some participants realised that this methodology based on bottom up is the practice of Labs: observation, proposition, exchange of practices, implementation and evaluation. The agora helped the participants to re-appropriate the methodology: The workshops are the concretisation of the relay function of the Labs I understood the methodology of the LABS: relay-observatory-think tank.
LLL-HUB
Evaluation Report (Final Version –May 2016)
81
The participants underline also that the innovative practices presented were often the outcome of European projects. And this gave to them the perspective that practices are not only local but may be part of a larger reflection on LLL strategies. The interviews realised during the Agora highlights the following main aspects:
Strategies linked to the territory : Business implication Boost business education partnership Working for the industry as a link for the trainees
Empowerment and capacity building
Partnerships : I am interested in building new partnership for my organisation Partnership should be maintained by network and website
And a main remark is the need to strengthen the link between national and European levels. This can be reached by reinforcing links with the European Commission but it needs also to maintain the links between partners and exchange views on LLL policies: Need more to see the European dimension on the sharing of practices. There are links between our countries: sharing views and definitions. Need to strengthen the link project-National Agency-European Commission because too often the results of project are neglected by European institutions.
IV.2. Sustainability and Acting for Lifelong Learning The Agora had not been the place where the conditions of sustainability are defined. But some main aspects concerning the follow-up give clues for the sustainability.
Sustainability at the local level: methodology as a key aspect The importance of taking into account the local level has been well underlined. The methodology is a key aspect of the sustainability: It arises the interest of the national Erasmus+ agencies (that participated to the AGORA). It promotes a light and easily sustainable organisation around LAB + FORUM. It makes possible to value any opportunity to speak about LLL strategy. It should be a main aspect of the dissemination: the partners and the participants have appreciated the presentation of the methodology. They think it is really interesting and valuable and it would be definitely beneficial to be shared for promoting LLL: Something could be done to involve and motivate experts to be part of the circle the project. The involvement of the experts is very important for the exploitation of the project. It is suggested that all partners send a joint letter to each national agency pointing out our recommendations. Already LLL-P (EUCIS-LLL) has thought of organising an LLL Award. It takes into account the success of the LLL-HUB contest and the capacity of dissemination of innovative practices. LLL-HUB
Evaluation Report (Final Version –May 2016)
82
It could be organised in October 2016 during the LLL-Week. The criteria could be the creative and inclusive practice in LLL (creativity, transferability, inclusion, learner’s needs, partnership and transversality). However the partners and organizers are aware of the difficulty to analyse common elements (“invariants”) from much diversified experiences and practices. This requires a research work. It should be necessary to add a researcher in order to help the national teams to an analyse practices and to present commonalities.
Acting for Lifelong Learning at the European level What has been demonstrated by the collection of practices in each country is that the key elements of LLL dynamism are to be found at the local level. Contrary to the national states of play, the unrolling of Lab-Forum showed the local capacity to formulate policies around innovative practices. The main commonalities (“invariants”) have been used to formulate recommendations. The question is how to design a method for Europe to take into account such propositions. In order to sustain the outcomes of the project and to act at the European level, two contradictions should be overcome: - the national level is not of the competence of the European commission The Unit "Europe 2020" of DG EAC (Sophia Eriksson Watershoot) has read the report but cannot act for national platforms. - there is no official institution responsible for LLL in the European commission. Hence strategies in ESF, skill agenda are not well integrated into a LLL strategy. The involvement of the wide network of LLL-P (EUCIS-LLL) is one possible direction We want to federate the members of the LLL Platform the way is to give them by giving them specific roles. There is the need to find out specific goals in order to motivate members to be involved. It is suggested that we find out from the EC what we can do and which is interesting for them and also for the partners. We could try the same methodology with a different topic that suits all the partners. LLL-P will use the Policy recommendations of the project and will prepare elements to propose for the next framework (2020-30). The aim is to build arguments for a national and European dialogue and to recommend the creation of a LLL group that links DG EAC and Employment directorate. In the end the creation of a LLL Agency should be advocated.
LLL-HUB
Evaluation Report (Final Version –May 2016)
83
Items of the semi-directive interviews A. Interviews of partners after the FORUM 6. FORUM : One day ? place, enrolling, … 6.1. What was a success? What did function best? What was best appreciated : + by the participants + by the experts + by you 1.2. What could be improved for a future Forum? 7. What was new and innovative? + What was new / State of play… + did you organize an evaluation questionnaire? Interviews with participants? + What was innovative: where does innovation lay? 8. Will the Forum pave the way to a permanent Lab? + What would feed a long-lasting group of reflexion? + What should be its mission: observatory – relay – think-tank? + What should be done to make it a reality? 9. How will you share this experience with the other LL-hub partners? 10. What are the impacts of the forum? + impact on participants + impact on politics + What should be done to multiply the impacts?
B. Interview of Management – Coordination during the project 1. What is LLL-HUB for you? 2. Your vision of the project 2.1. Did your vision change since the beginning of the project? Do you see it differently? 2.2. What is the thematic agenda in relation with Europe: where and when will be organised encounters with the European Commission? Will they be concomitant with EUCIS-LLL actions? 3. The strong points of the project 3.1. What is well going on? 3.2. What should be improved? 3.3. What will the partners gain from the project? 3.4. What value added : For each member for EUCIS-LLL LLL-HUB
Evaluation Report (Final Version –May 2016)
84
for Europe 3.5. Next key steps : Which are they? Do you foresee changes in the agenda or will it be the same as the initial one?
C. Interview of Management – Coordination after the AGORA 1. Your personal vision of the project after the Agora? - What are for you the main points of the project? - What you have best appreciated - What has functioned best - Did you expect such outcomes? 2. What will LLL-P do with LLL-HUB? - What do LLL-P retain from the project? - How will or should EUCIS-LLL use the LLL-HUB (working groups, conferences, … )? 3. How do you see that the European Commission and Parliament will be able to use the LLL-Hub’s outcomes? - Are there already feedback? - Which tools of dissemination: a leaflet, a manifesto? - Is there already convergence: (like in the case of the recommendation of Erasmus+ France for the building of a Erasmus observatory in France).
LLL-HUB
Evaluation Report (Final Version –May 2016)
85
Appendix 5 - List of interesting practices collected by the LABS and used do illustrate the different documents Bulgaria Interesting Successful practices: 10 1) Evaluation of the impact on the National Lifelong Strategy (NLLLS) of the Republic of Bulgaria for the period 2008 – 2013 2) Project No.BG051PO001-4.3.03-0001 “New Opportunity for My Future” 3) Project No. BG051PO001-4.3.02-0001 “Career Guidance System in School Education”, funded under OP HRD 4) Project No.BG051PO001-2.2.02 “Career Development of employed persons”, funded under OP HRD 5) Project No.BG051PO001-3.1.10-0001 “Improvement of Quality of Adult Vocational Training Services Provided by the Vocational Training Centres”, funded under OP HRD 6) Project No. BG051PO001/3.3.07-0001Project “Students’ and Pupils’ Practices”, funded under OP HRD 7) Project No. № BG051PO001-2.1.06 “Development and introduction of an information system for labour force competences’ assessment by branches and regions”, funded under OP HRD 8) Project No. 519195-LLP-1-2011-BG-KA3-KA3MP “Learning Augmented Reality Global Environment” (LARGE) 9) “Regional Strategies of Learning Validation and Education” (ReSOLVE), No. 521401-LLP-12011-1-BG-KA1-KA1ECETA2, funded by EC Programe “Lifelong Learning”, Horizontal programme, Key activity 1, Measure 1 “Support for European Cooperation in Education and Training” 10) Network for Effective Lifelong Learning Initiatives and Information (NELLII) Project under LLP, No. BG 521362 – UP–1-2011-1–BG – KA1KA1ECETA1 Case studies (collected Stories from Learners): 5
Belgium Interesting Successful practices: 3 1) B-post 2) Leren werkt! 3) Oscar Case studies (collected Stories from Learners): Life Stories : 7 Interviews of learners : 16
Netherlands Interesting Successful practices: 10 1) Creative employability, The Hague University of Applied Sciences 2) Intergenerational Learning, Learning for Learning 3) Associate Degree in the Netherlands, Ministry of Education, Culture and Science, department for HE 4) Non Formal Qualification at level 5 of the NLQF, Heering Consultancy LLL-HUB
Evaluation Report (Final Version –May 2016)
86
5) 6) 7) 8) 9) 10)
Rotterdam Academy, The Rotterdam University of Applied Sciences ISLA (cooperation with the USA), University of Toledo Flexibility and Adult Learning, Ministry of education-LLL Programme Permeability between VET and HE, LICA Life Management, EBC*L Education and the regional Labour Market, Municipality of Almere
Poland Interesting Successful practices: 4 1) Project „Festival of Professions in Malopolska” 2) Project: 50+ experienced, needed, qualified 3) Labour Market and Education Observatory of Małopolska 4) Malopolska Partnership for Continuting Education 5 other are in Polish :
Portugal Interesting Successful practices: 9 1) SITESE -CITEFORMA 2) INOVA+ 3) FLY HIGHER PROJECT 4) AIRVET PROJECT 5) IN2SAI PROJECT 6) ISCTE/Universidade Europeia 7) School of Abrigada 8) Universidade Aberta 9) Universidade de Évora Case studies (collected Stories from Learners): 3
Spain Interesting Successful practices: 8 1) Route 2.0, Extremadura 2) Lecturer Training and Support Unit (LTSU), Plasencia, Extremadura 3) Studium Platform, Salamanca 4) DUAL TRAINING, Extremadura 5) Soufflearning, Extremadura 6) Tripartite Foundation, Spain 7) JORNADAS DE EMPRENDEDORES/AS LOCALES……PASA A LA ACCIÓN… Case studies (collected Stories from Learners): 5
Turkey Interesting Successful practices: 1) COP(M)ADAM, Ayvalik – Balikesir 2) DREAMS ACADEMY, Istanbul 3) 100% Ecological Farmers Markets, Istanbul, Izmir, Kayseri 4) Bazar Virtual Market of Mature Entrepreneurs, Eskisehir LLL-HUB
Evaluation Report (Final Version –May 2016)
87
5) 6) 7) 8) 9) 10)
EDUC-ARTS, Ankara Library of my Dreams, Kocaeli Entrepreneurship Foundation, Istanbul Education without Labels, Adana, Ankara, Bursa, Gaziantep, Istanbul, Izmir, Kayseri, Samsun TEAM Game / inter-disciplinary Art Workshop, Ankara, Istanbul Prevention of domestic Violence against Women, Turkey
Case studies (collected Stories from Learners): 7
LLL-HUB
Evaluation Report (Final Version â&#x20AC;&#x201C;May 2016)
88