Page 1


www.leonardo.org.uk

www.grundtvig.org.uk

www.transversal.org.uk


TABLE OF CONTENTS 1.

About this handbook

Page 5

2.

Evaluation explained

Page 6

3.

Evaluation resources and requirements

Page 8

4.

An easy guide to evaluation for individual participants

Page 12

5.

Evaluation examples for projects

Page 15

6.

Further reading

Page 23

7.

Glossary of useful European project management terms

Page 25

8.

Contact us

Back page

Understanding Achievements – ECOTEC Research & Consulting Ltd

Page 4


1. ABOUT THIS HANDBOOK What is it? This handbook provides an introduction to evaluation activities for participants and projects funded under the Lifelong Learning Programme (LLP). It covers a range of activities from individual study visits to large scale cooperation projects. The guide has been tailored in particular to the Leonardo, Grundtvig and Transversal programmes. However it includes information that is useful for anyone involved in European projects. Who is it for? The handbook is for everyone involved in the Leonardo, Grundtvig and Transversal programmes. It is also aimed at anyone interested in applying for funding whether for an individual course or study visit or for a large scale project. Participants, Project Managers and Evaluators involved in other European projects and activities may also find it a useful introduction to evaluation. What does it contain? Section 1 provides an easy guide to evaluation and explains why it is an important element of your project or activity. Section 2 highlights evaluation resources as well as the requirements both before and after your project. Section 3 provides some case studies and ideas for participants in activities such as courses, study visits and placements. Section 4 presents examples of evaluation activities for projects. Section 5 offers links to further reading. Section six

Understanding Achievements – ECOTEC Research & Consulting Ltd

gives contact details to find out more information. Finally, section seven provides a glossary of useful European project management terms. What happens next? You can find case studies as well as practical tips from the National Agency and other projects in this guide. These should give you some good ideas as well as encouragement to get going with your own evaluation activities. If you need more details about any of the projects or topics covered in this booklet then please get in touch. We’re here to help so call 0845 199 2929 or email llp@ecotec.com. You can also find more information on our programme websites: Leonardo – for skills and training www.leonardo.org.uk Grundtvig – for adult education www.grundtvig.org.uk Transversal – for education and vocational training study visits www.transversal.org.uk

Page 5


2. EVALUATION EXPLAINED What is evaluation? Evaluation involves learning from experience. It requires you to reflect, understand, adapt and make changes in order to meet your aims and to improve your performance. Evaluation makes an assessment of your success. It helps you to understand how and why outputs and outcomes have been achieved as well as identifying any weaknesses or lessons to be learned for next time. Evaluation can therefore help you to make the most out of your LLP activities both during and after the funding period. What does evaluation entail? Evaluation involves asking lots of questions. For example, whether you have met your original objectives? Have you achieved what you set out to achieve? Whether you could do anything differently next time? You need to answer these questions, analyse and judge the results and then do something to take action or make changes. You also need to think about how you plan to use the results of your evaluation in order to get the most out of your findings.

There are 4 key stages to evaluation:

1. Define your performance indicators (signs of progress) 2. Gather information 3. Analyse your data 4. Report your findings

How does evaluation relate to monitoring? Evaluation is often associated with monitoring but it is much wider in its scope. Monitoring requires the tracking and recording of information and data and is of course a fundamental part of your project management. Evaluation involves interpreting these facts and figures and therefore involves a qualitative as well as quantitative assessment. You can use the information to find out what, how and why something happened and whether you achieved your original aims. Having good monitoring systems for your project is therefore an important starting point for gathering the evidence to be evaluated. For individuals, keeping track of what you are doing, is also essential for a record of your activities.

Understanding Achievements – ECOTEC Research & Consulting Ltd

Page 6


Who is evaluation for? The main beneficiary of an evaluation is of course you! If you are participating in a larger project then your project partners will also profit from an evaluation. However, evaluation of your activities or project also provides helpful information for several different groups. Evaluation activities and findings can provide a valuable insight for the National Agency and European Commission into the processes and results of your activity or project.

For projects, evaluation can be used to measure the impact of your activities and can help you to build on your successes and best practice. Reviewing your work and getting feedback from others also adds weight to your activities and outcomes. Evaluation can therefore help to improve the quality of your work. In addition, evaluation can help you to shape your dissemination and exploitation strategy as well as to identify new opportunities for the future.

A good evaluation can also validate your conclusions and lend credibility to your activities and results. It can therefore help to impact on a wider audience as well as add weight to any programme or policy recommendations that you have for the future.

By incorporating evaluation into your activities you are promoting a learning culture that makes a valuable contribution to the quality of Europe’s education and training systems. It can also provide some accountability for your activities as you have benefited from public funding through the LLP.

Other people that can benefit from your activities will also be interested in your findings. These can include colleagues and learners; beneficiaries and user groups; stakeholders and policy-makers; as well as groups in your field and other LLP projects. You should identify your key stakeholders and involve them in the design of your evaluation so that your findings will be relevant and useful to a wider audience.

Did you know? You will need to tell us about your plans for evaluation in your application form and if you are successful, you will be required to carry out evaluation throughout your project.

Why is evaluation important? At a personal level there are real benefits to carrying out evaluation activities. Spending time reflecting on your performance and reviewing what you have got out of an experience can help you to both celebrate your achievements and identify areas for development.

Understanding Achievements – ECOTEC Research & Consulting Ltd

Page 7


3. EVALUATION RESOURCES AND REQUIREMENTS Are there any resources to help me with my evaluation activities? For smaller activities and projects your grant will not include specific funding for evaluation activities. However conducting an evaluation may be easier than you think. As a minimum it is something that you will be doing anyway as part of your final report. If you are involved in a small project on a tight budget then peer review could also be a way to integrate evaluation into your activities. You could team up with another project or group in your field and develop an arrangement to review and provide feedback on each others activities. In addition, disseminating information about your activities can spark interest and generate feedback on your progress and products from other individuals and groups. Larger projects often set aside specific funds for evaluation activities. For example, this could be allocated under your subcontracting budget if you plan to involve an external evaluator. There are other resources that can help with your evaluation: 

Europass – This is an important mobility resource for individual participants in the LLP and is also valuable as an evaluation tool. It helps individuals to present themselves effectively and is also a means for employers and recruiters to gain information

Understanding Achievements – ECOTEC Research & Consulting Ltd

about your skills, competencies and qualifications. More information is available at: www.uknec.org.uk. Europass consists of five documents: -

-

-

-

Europass CV – this is a useful record of your skills and qualifications; Europass Language Passport – this document includes a self-assessment grid to help you identify your language levels; Europass Mobility Passport – this document provides recognition of your skills and knowledge gained during any learning or training period abroad such as a placement or exchange. Your mobility experience is monitored by an organisation in the country of origin together with the host organisation of the visit. They complete the document in a language agreed together with you. It provides an assessment as well as verification of what you have done during your period away. Europass Certificate Supplement – this provides information about the type of education and training you have completed and is issued by the awarding body. Europass Diploma Supplement - this document includes details about the content and the type of your higher education qualifications. It is issued by the university registry. The QAS project – Project Support for Quality and Sustainability (the QAS project) is a joint project between the National Agencies of the Czech

Page 8


Republic, France, Sweden and the UK. It supports partnerships within Comenius and Grundtvig by providing useful resources for project management and sustainability. Resources include a guide to Project Management for Beginners and also an Evaluation Quiz. More information is at: http://weaver.programkontoret.se/Default____5836.aspx.

Evalsed – This is an online resource from the European Commission’s Inforegio for Regional Policy. It provides guidance on the evaluation of socio-economic development with a specific focus on evaluation in EU cohesion policy. However, it provides a wide range of detailed information and evaluation tools that you may find useful for your project. Evalsed is available at: http://ec.europa.eu/regional_policy/sources/docgen er/evaluation/evalsed/index_en.htm. ECOTEC, the UK National Agency for the Leonardo, Grundtvig and Transversal programmes – We can help to disseminate your findings or any lessons learned from your evaluation activities. We have monthly e-flashes from each programme which are sent out to a mailing list of subscribers. Our quarterly newsletter, edUKation, is sent electronically to over 10,000 subscribers. We can also include interesting case studies, news and events on our programme websites. If you send us some brief details and a photo we may be able include your information so

Understanding Achievements – ECOTEC Research & Consulting Ltd

you can share your experiences with other interested parties and projects. 

We also have a national collaboration forum, our called Thematic Networking Groups, which work on different subject areas important to our programmes. You may be able to use them to share your experience and knowledge on particular themes relating to your evaluation activities and findings. Further information is available at: www.transversal.org.uk/thematicnetworking.

Does everyone involved in the LLP need to undertake evaluation activities? Everyone applying for LLP funding should give some consideration to evaluation before, during and after their activity. The following table provides an overview of what you will need to consider at application stage and at the final report stage for the Grundtvig, Leonardo and Transversal programmes: 1. Transversal Study Visits 2. Grundtvig In-Service Training 3. Grundtvig and Leonardo Preparatory Visits 4. Grundtvig and Leonardo Partnerships 5. Leonardo Mobility Projects 6. Leonardo Transfer of Innovation Projects

Page 9


Evaluation requirements before and after your activities Type of LLP Activity Transversal Study Visits

Requirements at Project Application Stage

Requirements in Project Completion Report

You will need to provide details about your motivation for the visit

The end of visit report is in essence an evaluation of your

as well as the expected impact.

experience. Specific information is also required about the

You need to answer questions about how your participation in the

impact of your visit.

study visit will contribute to your professional development. Other questions relate to the impact of your visit as well as how your organisation will benefit.

Grundtvig In-service Training

Grundtvig and Leonardo Preparatory Visits

You will need to explain your aims for the visit as well as the

The final report is also an evaluation of your experience. As

expected impact on your personal and professional development.

well as completing an evaluation of the training activity you

You will also need to provide information about how you intend to

will also need to complete questions about the impact of your

evaluate your participation in the short and long term.

visit at a personal and professional level.

You need to provide information about the aims of your project and

You will need to provide information about how you assess

an explanation about how your future project will be linked to the

the activities accomplished during the visit.

activities of your organisation.

You will also need to describe the extent to which you met your objectives as well as the impact of your visit.

Grundtvig & Leonardo Partnerships

You will need to provide details about your motivation and

You will need to describe the extent to which you achieved

objectives for the project.

your aims and objectives.

You need to describe how you will evaluate whether the aims of

Specific information is also required about how you monitored

Understanding Achievements – ECOTEC Research & Consulting Ltd

Page 10


the partnership have been met and whether the expected impact

and evaluated the progress and impact of the Partnership.

has been achieved.

You will also need to provide details about the main

You will also need to provide details about learner and staff

conclusions and consequences of your monitoring and

involvement in the evaluation of project activities.

evaluation activities.

Leonardo Mobility

You will need to provide details about your evaluation processes at

You will need to explain the methodology used for the

participant and project level as well as how the results will be used.

evaluation of the project as well as the conclusions drawn.

Leonardo Transfer of Innovation (ToI)

Detailed information is required about your project’s work

You will need to indicate the type of evaluation and testing of

programme. You will need to include a work package dedicated to

results/products/and processes as well as when, where and

project management which may include evaluation activities. This

how it was carried out. This should cover the scope, method,

may also be part of a separate work package.

tools and sample. Information is also required about the findings, conclusions and lessons of your evaluation and testing. This includes details about whether the results/product/or processes were modified and adapted after the evaluation and testing.

Understanding Achievements – ECOTEC Research & Consulting Ltd

Page 11


4. AN EASY GUIDE TO EVALUATION FOR INDIVIDUAL PARTICIPANTS This section is aimed at individuals taking part in activities such as contact seminars, courses, conferences, placements and study visits funded under the LLP. It provides information about how to carry out your own evaluation activities and also includes case studies from other participants. It should help you to make the most of your LLP experience. What can I evaluate? Self-evaluation is important even if you are taking part in smaller activities or individual visits funded under the LLP. There are many areas that you can consider during your self-assessment:  Before your trip you can think about your expectations of the visit and set yourself learning objectives. For example: improved knowledge and skills; increased networks and new contacts; the achievement of qualifications and certificates; enhanced career prospects; or new comparisons to benchmark your working practices and systems. When you return you can evaluate whether you met your aims and objectives.  All participants will need to complete an LLP final report after their visit. You can use this time to critically reflect on all aspects of your visit and think about your next steps;  You may need to undertake an evaluation exercise for your own organisation, colleagues or learners. You can also assess how you can use knowledge gained during your visit to evaluate working practices and systems in your organisation.

Understanding Achievements – ECOTEC Research & Consulting Ltd

 You will also need to provide feedback for the organisation that hosted your visit, course or placement. This can help to improve the quality of the visit for future participants.  You can evaluate the impact the visit has made on other areas such as your motivation, confidence, inter-personal skills, language learning, and cultural development.  Finally, it is important to assess what you can do next for yourself, your organisation, colleagues or learners.

Yvette Farhad participated in a work placement at Antwerp Zoo in Belgium. Her visit was organised by Lancaster and Morecambe College and was funded under the Leonardo Mobility programme. After her visit, Yvette completed a report and an evaluation questionnaire for her college prepared a presentation to inform others about her experience. From her evaluation activities she recognised that she achieved more than expected in terms of both personal and professional development. She says “I have gained a lot of confidence from my work experience at Antwerp Zoo. I have also gained skills living and working abroad which was not possible at home”. From her evaluation of the visit Yvette now has different aims for the future.

Page 12


When should I carry out my evaluation activities? You can carry out evaluation activities before, during and after your visit:  Before your trip set yourself goals and make an action plan on how to achieve them.  During your stay assess your success and whether you are on target. You can then make any necessary modifications in order to achieve your aims.  Following your visit you can review what you have gained from your experience and work out what to do next for your personal, professional or organisation’s development. Alex Burkill took part in a Leonardo Mobility trip to Germany. His visit was organised by SW Durham Training Ltd and funded under the Leonardo Mobility programme. He explained that during the visit participants contributed to a short talk about their views of the trip. They completed a diary of events and made presentations on their experience to make others aware of their visit. Alex’s trip helped increase his confidence and highlighted some of his key assets. Having reflected on the visit he said “Training and working in a company has helped me to develop my skills a lot faster, as the knowledge and opportunities offered on the trip are ones that wouldn’t have been available to me had I not taken part”. Since his return Alex has asked his organisation for more opportunities in an area of work introduced to him during the visit.

Understanding Achievements – ECOTEC Research & Consulting Ltd

Who can be involved in my evaluation activities? There are two types of evaluation activity: 

Self-evaluation is an assessment of your own performance. For individuals and small projects selfassessment may be the only form of evaluation connected to your LLP activity, so it is important to do it as well as possible.

External evaluation involves an independent view of your activities or project by a third party. External evaluation is more associated with larger LLP projects but it can be relevant to smaller activities. This could be through peer review on a formal or informal basis by colleagues or by other participants a course or placement. They could provide a valuable independent view of your achievements and constructive criticism for the future.

Mara Linnett went on a Transversal Study Visit to Sweden on the theme of gender equality in education and training. To prepare, she read the visit materials, researched the topic and considered her own expectations. Mara wrote the group report and took a key role in assessing what the group had learned. Participants divided into small groups to ensure that different opinions were included. Mara explained that “it was the best approach – it meant that everyone had a voice”. After the visit, Mara also wrote an individual report and sent it to her colleagues. Page 13


How can I go about it? As a minimum you will need to carry out self-evaluation as part of the completion of your LLP final report. Conducting your evaluation is therefore perhaps quicker and easier than you think as it is something you are doing anyway. You can also do the following: 

Set yourself realistic goals and develop a list of actions Make an action plan either by setting goals through a personal development plan at work or by giving yourself targets on a more informal basis at home. You could also use Europass to record your skills and knowledge as well as to include evidence from your organisation and the host of your placement or visit. More information about Europass is included in the previous section of this handbook. Review and make improvements Taking time to review your performance during an activity such as a placement or study visit can give you a chance to really get the best out of the opportunity.

Report your findings Make a note of your findings. You can use them for your own benefit as well as to disseminate to colleagues and learners as well as other interested parties.

Assess what to do next You should think about how to make use of your findings. For example are you able to enhance your career prospects, improve your working practices, use new teaching methods or materials, encourage further networking and future collaborations, motivate your

Understanding Achievements – ECOTEC Research & Consulting Ltd

colleagues or learners, or identify new development opportunities for the future?

Gillian Stewart joined seventeen participants from fourteen European countries on a Transversal Study Visit to find out more about teacher education in Finland. The visit was hosted by Joensuu University Practice School. When choosing her Study Visit, Gillian measured her own objectives against the strategic aims of her organisation, the Belfast Education and Library Board, so that the experience and knowledge gained could be fed directly back into priority work areas such as teacher quality and efficiency and school improvement. In particular, she was keen to investigate and analyse the reflective practice processes of undergraduate and newly qualified teachers. After the visit, Gillian used her individual report as a tool for her own evaluation of the visit. Her report was disseminated to colleagues working on the Beginning Teachers Programme and to the Senior Management Team at the BELB. Gillian has since found that the comparison of systems and techniques observed during the visit have been useful to evaluate and develop working practices back in the UK. She says “Being able to gain first hand experience of another teacher education system while finding out more about the teacher training and graduate teacher practices of the other delegate countries was truly enriching. I very much look forward to staying in touch with my new friends and colleagues and to continuing this debate and exchange of ideas.”

Page 14


5. EVALUATION EXAMPLES FOR PROJECTS This section is tailored towards people working on projects funded under the LLP. It provides some suggestions about incorporating evaluation into your project’s activities. It also includes case study examples from other projects. When planning your evaluation it is important to think about the following: What can we evaluate? The evaluation of your project should be considered as an essential activity and integrated into your quality management plan. It is important to think about the purpose and intention of your evaluation. You should also make sure that your evaluation covers the processes as well as the results of your project. So you should assess both what was done and also how you went about it. You will need to include quantitative and qualitative data in your evaluation. First of all you will need to decide exactly what you want to evaluate and think of key questions at the planning stage of your evaluation. For example, questions could cover whether you met your aims and objectives; value for money; efficiency; effectiveness; and the impact of your activities, as well as identifying any unintended consequences from your actions. Your evaluation questions are crucial so you need to make sure that they will be effective. It is therefore also essential during the design of your evaluation plan to think about how you will use the results and findings. You should also incorporate areas relevant to the LLP and the particular type of activity you are undertaking. For example, your evaluation could consider topics such as Understanding Achievements – ECOTEC Research & Consulting Ltd

transnationality and partnership-working; project management; dissemination and exploitation; innovation; equal opportunities; and contribution to LLP strategic objectives. You should also evaluate the user-friendliness and relevance of any products with their intended target groups as well as the distance travelled on soft outcomes such as increased confidence and improved skills of learners.

John Perry is involved in Lancaster and Morecambe College’s Zoo Work project, helping students gain work experience in Belgium with Leonardo Mobility funding. In addition to the programme’s requirements, John asked students to complete an evaluation form after their placements. Feedback helped evaluate the efficiency of the visit organisation and the relevance to the students’ own learning and employment aims. They are also required to make a presentation to the European Committee at the college. The host organisation of the placement was also asked to report on the activities and attitude of the students. The results were used to measure the experience gained, increased confidence, maturity and skills. John explained, “As a publicly-funded organisation, we have to show that the funding added real value to the learning experience and that it improved the college’s delivery of vocational education and training”. More details about the visit are available at: www.lmc.ac.uk/home/european Page 15


Who will be involved in our evaluation? When you have decided the purpose of your evaluation you should identify your main users and stakeholders and involve them in the design of your evaluation. Your evaluation can then be more influential as:   

you can ask better questions; obtain higher quality information from these groups during the evaluation; and improve ownership and use of its findings. You also need to decide who in your project will be in charge of the evaluation so that your evaluator has a main point of contact. Those responsible for the evaluation should make sure that actions are followed-up so that improvements are made during the project. You may also choose to involve a steering group with representatives from your users and stakeholders as well as your partnership to share responsibility for the evaluation. You should ensure that you have clear channels for your evaluator to be able to communicate findings to key decision-makers and stakeholders in your project in time for them to act on recommendations and opportunities.

Understanding Achievements – ECOTEC Research & Consulting Ltd

Ajay Desai and Jane Fleming are involved in the InTeLS+ and YENTELS projects managed by Coventry and Warwickshire Chamber of Commerce. They highlight that “evaluation is of foremost importance, as it highlights areas where improvements can be made”. Jane and Ajay explained that comprehensive evaluation activities will be carried out throughout the project relating to both internal and external processes as well as of products being developed. This has involved discussions with the project manager, one-to-one talks with the project team, discussions with key national stakeholders, structured telephone interviews with SMEs and end-users as well as a desk analysis of a range of documents. The work package leader for quality was made responsible for this area of work. Any areas identified by the evaluation are highlighted as tasks to be reviewed at partner meetings and focussed on in interim reports. They explain that “evaluation acts as an information process that provides a systematic determination of merit, worth, and significance against a set of predetermined standards”. InTeLS+ and YENTELS are Leonardo da Vinci Transfer of Innovation projects, more information is available at: www.intels.biz.

Page 16


Who should carry out our evaluation activities? There are two types of evaluation activity: self-evaluation and external evaluation:  Self-evaluation This refers to an assessment of your own performance or by those involved at the centre of your project such as the promoter or partners. For small projects self-assessment may be the only form of evaluation connected to your LLP activity. Therefore it is important to do it as well and as objectively as possible. An in-house evaluation can also be a valuable active learning process for those involved. For larger projects, self-evaluation is also valuable and is often conducted in conjunction with an external evaluation. For this type of internal evaluation the person involved should be as independent from its management as possible and be provided with the time and resources to do it properly. Some organisations may also have separate teams or departments that can undertake a more independent evaluation of your activities.  External evaluation This involves an independent view of your activities or project by someone else or other groups. External evaluation is often carried out by a subcontracted independent reviewer of your project. It can also involve assessment of outputs or processes by participants, beneficiaries and experts as well as peer review by other similar projects or groups in your field. This type of evaluation is therefore essential to ensure that what you are doing is relevant to your target audiences, as well as to add credibility and transparency to your actions. An independent reviewer’s opinions can also carry extra weight when working to meet project deadlines. Understanding Achievements – ECOTEC Research & Consulting Ltd

Information about commissioning and managing external evaluators is available in the Evaluation Guidance Note for Leonardo Projects. To download a copy visit www.leonardo.org.uk, select ‘useful documents and information’ and then ‘disseminate and exploit results’.

Be All you Can Be! is a Grundtvig Learning Partnerships project managed by Red2Green. The project examines the role of education and work opportunities for young people and adults with learning difficulties. More information is available at: www.red2green.org. Susan Owen from the project explained that staff and learners from all the partner organisations are involved in their evaluation activities. Forms have been designed so that partners can evaluate their discussions after meetings and learners are able to evaluate visits and cultural activities. Weekly meetings for core staff working on the project also incorporate informal evaluation. Susan explains “You learn as you go along, so evaluation changes slightly”. Feedback from their learners has been positive. The staff evaluation highlighted the importance using clear terminology to prevent misunderstanding. As a result the partnership now takes more time to discuss this.

Page 17


Interim evaluation Interim evaluation is an opportunity to assess your progress at the mid-way point and whether you are on track to achieve your aims by the end of your funding period. Your project should also be at an appropriate stage to give the evaluator a good idea of how things are progressing. As with ongoing evaluation it can help to identify potential problems as well as improvements for the next phase of your project.

Final evaluation The purpose of a final evaluation is to assess the overall lessons learned from your project. It is therefore also known as a summative evaluation. Conducting evaluation at the end, or after your activities, can help you to draw conclusions on your success, measure impact, inform others of your results, as well as inspire new ideas and opportunities for the future. It can also assist with identifying what worked as well as what didn’t work so well for next time.

When can we undertake our evaluation activities? It’s important you do your evaluation before and during your project as well as after it has been completed. This will enable you to make improvements as you go along and will leave you with a valuable record of what was achieved at every stage of your involvement in the LLP. To begin with you need to develop your evaluation strategy. Information about evaluation is required at the application stage so you should aim to have this in place from the beginning. There are three stages when you could undertake an evaluation: ongoing evaluation; at an interim stage; and at the end of your activities. 

Ongoing evaluation Ongoing evaluation is a review of performance during your activities. It aims not only to track your progress but also to give you a chance to adjust and make improvements along the way. Ongoing evaluation can also help to identify early warning signs of problems as well as helping to find solutions. This is also known as formative evaluation as it helps you to shape what you are doing so that you can meet your targets. It can help you to critically reflect on whether the processes, such as project management and monitoring systems, are effective and how they might be enhanced as well as identifying improvements to any products being developed. You should therefore align stages in your evaluation with phases of your project to make the most of the findings.

Understanding Achievements – ECOTEC Research & Consulting Ltd

Page 18


Liz Buchanan at VisitScotland is involved in the Vocational Management Training for the European Tourism Industry 2 (VocMat 2) project, funded through Leonardo Transfer of Innovation. Liz explained that evaluation activities will be undertaken at different stages of the project. A milestones chart is reviewed throughout the project. Monitoring reports from partners are checked every quarter and include evaluation forms to establish how well the project is progressing. Partners expectations were identified at the start of the project and will be evaluated again mid-way through and at the end.

What resources are available for our evaluation? You will need to allocate time and a budget for your evaluation activities. You should also think about the resources you have available to inform your evaluation and put together a strategy for the data collection. You will need to decide how you will measure the impact of your activities. For example, you should consider how to collect baseline data to compare the situation before and after your activities. You may consider using control groups to measure the difference between the effects of participation in the project to what may happen without any involvement. It is also important to measure the distance travelled in relation to soft outcomes and improvements made by learners. Evaluation could include assessments at the start and then at the end of your learners’ involvement in the project. More information about measuring soft outcomes is available at www.london03.ecotec.co.uk/resources

Feedback is collected from partners, participants, external evaluators, peers and experts from both the education and tourism sectors as well as from SMEs. The partnership has also used different methods to collect feedback from external audiences including workshops, seminars and online questionnaires. Evaluation activities have proved to be vital in assessing expectations and impact. They also identify weaknesses, that need correcting during the project, and strengths that can be built on. Find out more at: www.vocmat.com/index.asp. Understanding Achievements – ECOTEC Research & Consulting Ltd

Page 19


 Khawar Iqbal is involved in two Grundtvig Partnership projects managed by Teaching and Learning Challenges Ltd. The projects are called Developing EU Culture in the Suburb Areas of the City and An Intercultural Approach to Learning Foreign and Second Languages in Europe. Khawar says “you need to think about evaluation at the beginning and think about how you are going to do it”. Khawar is also involved in a Multilateral project called INTERTOOL, which aims to provide European project managers and teams in adult education with the basic specific intercultural competences necessary for successful transnational co-operation. They have used online questionnaires to gather data for their basic needs survey. The partners have also conducted interviews during the second phase of their needs analysis. INTERTOOL will include a Virtual Intercultural Team Tool (VITT) with a specific section on evaluation. The tool will be available next year at: www.intercultural.ro/intertool.

What are the best methods for our evaluation? There are lots of tools to help you during an evaluation. You should think about the best and most appropriate methods to collect information for your project. You could consider incorporating the following methods into your evaluation:

Understanding Achievements – ECOTEC Research & Consulting Ltd

 

A management information review can be helpful in providing easily accessible data to be analysed during an evaluation. Consultations can provide an early opportunity to share knowledge and explore options; Self-administered questionnaires and feedback forms can be useful in providing quantitative as well as qualitative information. Online evaluation can give access to a wide audience but some people may be put off using them due to technical difficulties whilst others may not be able to access the technology. The use of structured questions in interviews can also be helpful in providing both quantitative and qualitative data. Face-to-face interviews may be more expensive than telephone contact but sometimes provide more of an insight. Focus groups require facilitation and recording skills but can be useful in bringing out views and feelings. However they may be taken over by more dominant participants. Depending on the group dynamics other members may be more reluctant to share their views. Observation of participants during activities and piloting of products are valuable forms of evaluation. They can provide an important insight from your target group of the relevance and appropriateness of your project as well as providing critical feedback on your project.

More information about advantages and disadvantages of different evaluation methods is also available in the Evaluation Guidance Note for Leonardo projects. To Page 20


download a copy visit www.leonardo.org.uk, select ‘useful documents and information’ and then ‘disseminate and exploit results’.

Exchange of Mentoring Methodologies and Approaches (EMMA) is a Grundtvig Partnerships project managed by Dundee College. The project brings together a range of organisations and agencies supporting learners in their learning pathway towards or through employment. Lorna Reith from the project explains that the partnership use questionnaires to evaluate each partnership meeting. They assess everything from the information received beforehand, to the accommodation used as well as the content of the meeting itself. Dundee College also uses evaluation questionnaires, feedback forms and one-to-one meetings with learners to evaluate the support that they receive.

How can we make use of our evaluation findings? The timing of your evaluation is important in order to make best use of the findings. As mentioned previously, stages in your evaluation should fit with your project cycle. You should also be informed of recommendations in time to make changes and inform key groups of the findings. It is important to use your evaluation report as part of your dissemination materials. You may need to target it to your different audiences. A comprehensive summary with key conclusions is also useful. However, to make the most out of your evaluation you should also think about how your project fits with the wider picture. For example, do the subjects and questions being explored in your project fit in with a larger set of issues? If so perhaps the findings from your evaluation could be more influential. You could use them to back up your project conclusions, contribute to issues being debated, or to inform decisions to be made in the near future. You should therefore use your evaluation as part of your evaluation strategy.

Thanks to their evaluation activities the Dundee College team realised they were focussing on one kind of learner. It made them question how they could broaden their approach. Lorna says “You learn a lot about how your organisation is delivering. Evaluation makes you reflect, it is a healthy thing. You want to certify that you are doing the work well”.

Understanding Achievements – ECOTEC Research & Consulting Ltd

Page 21


Clive Billingham was responsible for hosting a Study Visit at Leicester City Council In June 2008. Delegates from across Europe met to find out more about how the city’s education department and schools are dealing with diversity, race equality and community cohesion.

importance of this interaction between delegates for learning and recapturing what you have done. He explains “Evaluation is important to see what participants get out of the visit. It is critical that participants de-brief each day as it helps them to reflect on their own learning experience”. Participants then collaborated on the final report, to be submitted to the National Agency. They also completed individual questionnaires to capture the variety of opinions within the group. An evaluation session took place on the final day. Feedback from the participants was used to inform the planning of future visits. After the trip, the group report was disseminated to senior managers in the organisation. The visit was used to highlight opportunities for international work as well as for professional development. Clive learned that not only had they achieved their own aims and objectives but that it was also a good experience for everyone involved.

Clive Billingham (centre) with the Study Visit group

Clive and his team prepared a programme for the visit beforehand, which briefed delegates on the host’s objectives for the visit. These were discussed at an introductory session on the first day, giving participants the opportunity to add their own objectives. This helped to set the scene and enabled participants to revisit their learning objectives at the end of their stay. De-briefing sessions were held daily. Clive highlighted the Understanding Achievements – ECOTEC Research & Consulting Ltd

Page 22


6. FURTHER READING DISSEMINATION In addition to the information in this guide you can find further details about dissemination in the ‘Dissemination Guidance Note for Leonardo projects’. To download a copy visit www.leonardo.org.uk, select ‘useful documents and information’ and then ‘disseminate and exploit results’. DISSEMINATION AND EXPLOITATION ‘Sharing Success’ is a new dissemination and exploitation handbook for everyone involved in the LLP. To download a copy visit www.leonardo.org.uk, select ‘useful documents and information’ and then ‘disseminate and exploit results’. EUROPASS You can find more information about Europass at: www.uknec.org.uk EVALSED This is an online evaluation resource from the European Commission’s Inforegio for Regional Policy. More information is available at: http://ec.europa.eu/regional_policy/sources/docgener/evalua tion/evalsed. EVALUATION More information about Evaluation is available in the ‘Evaluation guidance note for Leonardo projects’. To download a copy visit www.leonardo.org.uk, select ‘useful documents and information’ and then ‘disseminate and exploit results’. Understanding Achievements – ECOTEC Research & Consulting Ltd

Further information about Evaluation is available in the LSC Pan London ESF Programme Evaluation and Dissemination Guidance Note: www.london03.ecotec.co.uk/download/logos/evaluation_gui dance.pdf The EQUAL Support Unit also has a Guidance Note on Evaluation for Development Partnerships’ available at: www.equal.ecotec.co.uk/resources/gpg/ EVENTS To download your copy of the Event Management Guidance Note for Leonardo Projects visit www.leonardo.org.uk, select ‘useful documents and information’ and then ‘disseminate and exploit results’. EXPLOITATION More information about exploitation is available on the European Commission’s Dissemination and Exploitation web-pages at: http://ec.europa.eu/dgs/education_culture/valorisation/index _en.html MAINSTREAMING More information about mainstreaming is detailed in the EQUAL guide ‘Mainstreaming and creating impact – a guide for Development Partnerships’ at: www.equal.ecotec.co.uk/resources/gpg/ PARTNERSHIP WORK ‘Learning Together’ is a new Guidance Note focusing on how you can work effectively with partners and get the best Page 23


out of your European project. The Transnational Partnership Guidance Note also provides useful information about working with your partners: To download a copy of either document visit www.leonardo.org.uk, select ‘useful documents and information’ and then ‘disseminate and exploit results’. PROJECT MANAGEMENT The "Survival Kit for European Project Management" provides information about managing your European partnership. To download a copy visit www.leonardo.org.uk, select ‘useful documents and information’ and then ‘disseminate and exploit results’. Project Support for Quality and Sustainability (the QAS project) provides useful resources for project management and sustainability. More information is at: http://weaver.programkontoret.se. VALORISATION The Valorisation Guide provides useful information about exploiting the results of your project and building on your projects achievements. To download a copy visit www.leonardo.org.uk, select ‘useful documents and information’ and then ‘disseminate and exploit results’.

Understanding Achievements – ECOTEC Research & Consulting Ltd

Page 24


7. GLOSSARY OF USEFUL EUROPEAN PROJECT MANAGEMENT TERMS Eligible Countries – The 27 EU Member States (Austria, Belgium, Bulgaria, Cyprus, Czech Republic, Denmark, Estonia, Finland, France, Germany, Greece, Hungary, Ireland, Italy, Latvia, Lithuania, Luxembourg, Malta, Netherlands, Poland, Portugal, Romania, Slovakia, Slovenia, Spain, Sweden, United Kingdom), plus the three European Economic Area countries (Norway, Iceland & Liechtenstein) and the candidate country Turkey. Lifelong Learning Programme – Parent programme designed to promote exchange, co-operation and mobility between education and training systems within eligible countries. Grundtvig Programme – Sub-programme of the Lifelong Learning Programme that has a specific focus on the development of teaching and learning for adults and young people not involved in tertiary education. Leonardo da Vinci Programme – Sub-programme of the Lifelong Learning Programme that has a specific focus on the development of vocational education and training for employees, students and education and training staff. Comenius Programme – Sub-programme of the Lifelong Learning Programme that has a specific focus on the development of school education for children, young people and staff.

Understanding Achievements – ECOTEC Research & Consulting Ltd

Erasmus Programme – Sub-programme of the Lifelong Learning Programme that has a specific focus on the development of teaching and learning experiences for students and staff in Higher Education. Transversal Programme – Sub-programme of the Lifelong Learning Programme that has a specific focus on Policy Development, Information and Communications Technologies, Languages and Dissemination. Transnational Partnerships – Groups of organisations from eligible countries who agree to work together on a specific piece of work, research, project or exchange. Mobility Activities – Activities which involve staff, students or employees travelling to a host organisation within an eligible country to participate in an exchange, visit or project meeting. Centralised Actions – Those activities funded by one or more of the various sub-programmes, where project organisers both apply and report back directly to the Executive Agency in Brussels. Decentralised Actions - Those activities funded by one or more of the various sub-programmes, where project organisers both apply and report back to their appropriate National Agency. Dissemination and Exploitation of Results – The manner in which the results of an exchange, partnership or project are exploited or continued after the funding period ends.

Page 25


National Agency – The organisation(s) responsible for administering the Lifelong Learning Programme in each eligible country. In the UK, ECOTEC manages the Leonardo, Grundtvig and Transversal programmes, and the British Council manages Comenius and Erasmus.

Project Outcomes – Qualitative indicators of a project’s progress, success or impact (e.g. people having become more employable as a result of being able to converse in a second language).

Executive Agency – The Education, Audiovisual & Culture Executive Agency who are responsible for administering the centralised actions of the Lifelong Learning Programme at European level Project Co-ordinator – The organisation responsible for submitting an application on behalf of a transnational partnership. In some cases (Leonardo Mobility and transfer of Innovation projects) this also means holding the contract and budget for the whole project, managing the project and reporting on progress. Project Partner – Organisation responsible for working collaboratively with other organisations in eligible countries. Project Compendium/Directory – A directory of projects funded via the Lifelong Learning Programme programmes, published each year and containing an overview of the aims, objectives, results and partner organisations involved in each project. Project Outputs – Quantitative indicators of a project’s progress, success or impact (e.g. the number of people participating in language skills training as a result of the activity).

Understanding Achievements – ECOTEC Research & Consulting Ltd

Page 26


Understanding Achievements – ECOTEC Research & Consulting Ltd

Page 27


CONTACT US If you have any queries about the material in this handbook, or any suggestions you would like to see included, please let us know. You can also contact us if you need more advice on any of the topics discussed or assistance with any problems you may encounter during the lifetime of your project. Our contact details are: Leonardo UK National Agency Helpline: Email: Website:

0845 199 2929 leonardo@ecotec.com www.leonardo.org.uk

Grundtvig UK National Agency Helpline: Email: Website:

0845 199 1919 grundtvig@ecotec.com www.grundtvig.org.uk

Transversal UK National Agency Helpline: Email: Website:

0845 199 3939 studyvisits@ecotec.com www.transversal.org.uk

You’ll find other guides and case studies on our websites!

If you would like to subscribe to any of our newsletters please use the subscribe function on the websites or email llp@ecotec.com.

Understanding Achievements – ECOTEC Research & Consulting Ltd

Page 28

Understanding Achievements - Guide to Evaluation  

Guide to evaluation for Leonardo, Grundtvig and Transversal projects

Read more
Read more
Similar to
Popular now
Just for you