Eu unawe evaluation summary report gk

EU-UNAWE Evaluation Summary Report

Report compiled by Grace Kimble, December 2013

2

Summary This document presents the summative impact of EU UNAWE workshops for teachers and pupils. It explains how project co-ordinators collaboratively agreed a framework for conceptualising domains of learning about astronomy for early years children. It synthesises different evaluation approaches in different countries, and presents evidence within the UNAWE evaluation framework.

It explains an initial phase of methodology agreement between the six EU UNAWE partner countries; South Africa, Spain, The Netherlands, Germany, the United Kingdom and Italy. It includes defining shared impact objectives (March 2012), constructing a draft framework and modifying it collaboratively (October 2012). The framework allows conceptualisation about learning in astronomy in the domains of (1) Motivation, (2) Scientific Skills, (3) Knowledge, (4) Intercultural attitudes, and (5) Legacy. The framework was used in planning evaluation and collecting data during spring and summer 2013.

There is evidence that the programme delivered key goals of advancing children’s learning about astronomy and culture, through delivering teacher professional development workshops and resources, which were subsequently used by teachers to provide learning activities for children in class.

The website www.unawe.org demonstrates that a range of resources have been produced and translated, for use by teachers and astronomy education professionals. One outcome related to the programme is the development of a peer review platform for assessing the quality of education resources.

The project has built capacity for future evaluation within project partners and in the wider EU-UNAWE network through workshops at two International meetings, as a response to identifying a need for shared understanding of evaluation approaches.

3

Contents 1. Background

5

2. Process

6

3. UNAWE Summative Evaluation Framework

9

Stage I: Collaborative agreement of impact objectives Stage II: Concept agreement and trialling Stage III: Collaborative Modification Stage IV: Resource production 4. National impact

19

I. Motivation II. Scientific Skills III. Knowledge IV. Intercultural attitudes V. Legacy: Capacity building workshop 5. International impact

44

6. Recommendations

50

7. Conclusions

52

Appendices

A: UNAWE evaluation framework booklet B: Italian evaluation report C: UK Teacher CPD and classroom observation interviews D: The Netherlands Teacher CPD observation E: Evaluation workshop case study booklet

4

Evaluation Framework contributors

Germany: Natalie Fischer and Dr Cecilia Scorza, Anita Mancini

Italy: Alessandra Zanazzi and Lara Albanese

The Netherlands: Frederiek Westra van Holthe, Wouter Schrier, Erik Arends

South Africa: Troshini Naidoo and Sivule Manoxi

Spain: Eloi Arisa

United Kingdom: Libby McKearney and Mark Armstrong

5

1. Background Project Objectives EUNAWE responds to the outreach part of the Coordinating Action for FP7-SPACE-2010-1. It meets the specific requirements of the call (Section 1.1.2.4). EUNAWE exploits inspirational aspects of astronomy and space to interest very young disadvantaged children in science and technology, broaden their minds and stimulate European and global citizenship. The proposal builds on Universe Awareness (UNAWE), a unique, innovative and proven programme for children aged 4 to 10 years. It will exploit the achievements of European (EU) and South African (SA) space sciences to inspire, excite and stimulate young children, when their curiosity is high and their value systems are being formed. Specifically, EUNAWE will: 1. Train and empower primary school teachers in 6 countries to include astronomy and space topics in the classroom.

2. Develop and translate hands-on material, where appropriate emphasising EU and SA science and technology.

3. Provide a network for exchange of expertise and material between educators - Lay the groundwork for expansion of the programme throughout the EU, Associated Countries and ICP Countries.

4. Act as a showcase for EU and SA astronomy/space and related technologies, by disseminating the products among very young children, their teachers and their families.

5. Use astronomy/space products to stimulate awareness and strengthen public support for EU and SA space science research and technology.

6. Stimulate the next generation of EU and SA engineers and scientists, particularly girls.

7. Contribute to the integration of disadvantaged communities in participating countries.

8. Strengthen collaboration between EU and SA over mutually beneficial scientific, technological, educational and social topics.

9. Provide significant added value for Europe's expenditure on astronomy and space sciences for a modest incremental cost.

10. Pooling complementary expertise and resources of 6 partners gives a project whose whole is greater than the sum of its parts.

(Numbers added for reporting purposes) Project duration: 01/2011 – 12/2013 http://cordis.europa.eu/projects/rcn/97906_en.html

6

Evaluation aims As stated in the programme agreement, this programme will communicate the impact of EU UNAWE activities for co-ordinators, teachers and pupils, using quantitative and qualitative methods.

Front end and formative evaluation input It is clear that each country has undertaken considerable formative evaluation, as appropriate to their capacity and resources, in order to shape programme development.

The majority of these have used teacher questionnaires as a methodology. Some existing data show evidence of the summative programme impact, and therefore these areas will be highlighted in this report focussing on summative evaluation.

Summative evaluation

Front End Evaluation Assessment of need for programmes before activity

This report focuses on summative evaluation: the impact of the programmes.

Feeds into future planning.

Formative evaluation Gathering data about improvements to programme and participant response

7

2. Summative Evaluation Process

8

3. UNAWE Summative Evaluation Framework 3. Stage I: Impact Objectives- collaborative agreement In March 2012 at the EU UNAWE International Conference in Leiden, impact objectives were agreed by UNAWE co-ordinators at a series of evaluation workshops. Both National Project Managers and International colleagues were involved in this collaborative process.

1. Stimulate via images, playful activities, experiments and models children´s awareness of the sun, the moon, the Earth as a planet and their related phenomena (day and night, tides, eclipses, and seasons), other planets, the stars in our galaxy, other galaxies, and their clusters and beyond. In doing so try to use throughout daily-life materials in the activities, experiments and models such that children can reproduce them at home.

2. Stimulate children giving them time and the occasion to formulate their own questions and to broaden their view of the “world� while delivering the program.

3. Create the occasions for direct observations of the sun, the moon, the stars and the planets taking into account school breaks, time-lines, excursion and project days.

4. Motivate children to appreciate other cultures (via stories and Skype conversations with children living in other countries) making them aware of the place in which other children live, their common cultural features and differences.

This proposal suggests a new framework is appropriate for EU-UNAWE evaluation, covering a range of domains linked to impact objectives above.

1. Astronomy Awareness 2. Curiosity 3. Observation skills 4. Motivation and engagement 5. Global cultural respect

9

3. Stage II: Concept agreement In August 2012, Grace Kimble worked with Dr Cecilia Scorza at the Haus Der Astronomie in Heidelberg to further develop an impact evaluation framework appropriate for UNAWE programmes.

The visit included familiarisation with UNAWE session spaces and resources.

The key deliverable of this consultancy work was a draft evaluation framework, draft evaluation materials, online surveys, and template letters for teachers and parents.

The booklet including this work can be found as an appendix to this report- Appendix A. Image above: Heidelberg early years astronomy classroom

The draft concept draws on impact definitions used by the US Informal Science Education framework (National Science Foundation, 2008), Earthsmarts’ Conceptual framework (Nichols, 2012) and Generic Learning Outcomes (Museums, Libraries and Archives, 2008).

Using their models as a basis, the domains of learning were defined, starting from from shared goals. Corresponding evidence was identified, before methodologies and draft materials were produced for comment. These included pre and post visit activities appropriate for different age groups, using a drawing methodology derived from Personal Meaning Mapping (Falk and Dierking, 2000). For younger children, a behaviour observation template was outlined, using methodology derived from Barriault and Pearson’s work on behaviour analysis (2010). Dr Cecilia Scorza produced a visual game example, around which observation of behaviour could take place. Also included were surveys for children and teachers, a data collection process outline, and template letters for teachers and parents.

Ethical conduct codes were followed in the production of consent letters for participation and visual evidence such as photo or video.

During September 2012 Dr Cecilia Scorza used draft materials with children to assess the usability of materials.

10

Draft UNAWE Summative Evaluation Framework

1. Motivation

Definition

Evidence

1. Enjoyment 2. Inspiration 3. Creativity 4. Persistence

1.

2.

3.

4.

Smiles, laughter, positive exclamation Ideas related to UNAWE intervention Construction game/ activity related to UNAWE Overcoming setbacks to complete a task

1. Curiosity 2. Observation 3. Identification 4. Classification 5. Making interconnections 6. Changing perspective 7. Communication

1.

Questions, practical investigation, reading, internet research Using direct observation or equipment, sustained searching for visual clues Correct use of vocabulary or gesture to name objects and phenomena observed in sky, as photos or in video.

Grouping objects/ phenomena to indicate developing understanding of astronomy concepts.

May take the form of a game or discussion Verbal or visual evidence of linking new information with existing conceptions Demonstration of understanding what others will see in different countries Showing others their new knowledge e.g. friends or family

2.

3.

4.

4. Intercultural attitudes

3. Knowledge

2. Scientific Skills

5.

6.

7.

Observing, exploring and discovering: Direct observation and/or recording of naming, first 1. The Sun, Sun light (shadows), Day/night explanations, discussing, drawing, construction, creative cycle, the seasons, responses, movement, to demonstrate knowledge of one of the features listed. left the Moon, the Earth as a planet, Solar and Moon eclipses, awareness of the air, water, Sun light on Earth,.

2. The Solar system: planets characteristics and movements, wharf planets, asteroids, comets, and exoplanets 3. The Stars in the night sky, the constellations, Life-cycle of stars, formation of stars and planets 4. Our place in the Milky Way, Family of galaxies 5. Current developments in astronomy 1. Valuing different cultural perspectives 1. Demonstrating awareness of different cultures 2. Recognising different physical perspectives through dance, performance, gesture, discussion, 3. Positive attitudes towards astronomy interaction with objects, song, creative responses 2. Ability to recognise differences in phenomena in different countries and to respond verbally or physically 3. Statements of future activity with regards to astronomy

11

Teachers gain subject knowledge for teaching their pupils in the domains previously explained. In addition there is potential for the impact of the project to be multiplied through: a) teachers disseminating knowledge b) teachers critically appraising resources and activities, in order to make recommendations for future projects c) Embedding astronomy activities into curriculum

Erik Arends discussing demonstrations to teachers at a Teacher CPD workshop Leiden, April 2013.

12

3. Stage III: Collaborative Modification National Co-ordinators met in October 2012, where the draft evaluation framework was discussed. The presentation used as a basis for the framework can be seen here: http://www.slideshare.net/unawe/euunawe-programme-evaluation Following rationale, a demonstration of draft materials by Dr Scorza and discussion, a number of modification points became clear:

National Specificity • •

•

•

Clarity of vocabulary required owing to local interpretations (IT) Framework is sufficient for needs if it can be modified (SA) E.g. Discussion of ‘knowledge’ scope. Agreement that national school curricula largely define the ability of national co-ordinators to cover objectives, and that these are specific to each nation. However, the framework allows for evidence under a range of possible developments in conceptual understanding.

Owing to the range of ‘knowledge’ covered, it is not realistic to use level descriptors, such as those appropriate for school teacher assessment, across all programmes.

This is consistent with contemporary approaches in evaluation; assessing informal learning impact involves assessing a broader range of domains, taking a broader view than formal education assessment.

Google docs or online surveys are not good for every country

Programme Specificity •

Understanding that evidence can be gathered from programmes with a range of formats. For example, Italian puppet activities, South African visual activities . For example, the evaluation framework could be used before and after a mobile planetarium visit, or after a classroom activity

Questions •

National co-ordinators wish for a set of questions to accompany each domain.

13

Use of teacher class assessment •

Teachers frequently have a requirement to assess pupil understanding through elicitation activities. The framework guidance needs to facilitate co-ordinators in using some of this rich existing evidence in order to demonstrate the impact of the programmes on pupils.

•

In Italy, photos were taken of children’s notebooks as part of evaluation, to record spontaneous activity.

Pupil voice •

There is a range of experience in evaluating with children, therefore this presents the opportunity some programmes to demonstrate expertise.

•

Use more drawings for the children’s activity

Domains •

There was agreement that the relationship between domains and evidence needs to be a many to many relationship, rather than one-to-one (as in the draft)

•

Curiosity was felt to fit under the first domain, motivation

•

Scientific skills should align more closely with curriculum wording

•

A positive approach to inclusivity should be included under attitudes

Evidence •

Additional phrases were suggested, to broaden the range of evidence e.g. perseverance could be interpreted as tenacity.

•

It should be made clear that the framework evidence examples are not prescriptive or exhaustive.

•

Observation schedules could include tracking

•

Spontaneity should be recognised, independent selection of topic

The references and rationale documents were circulated, and National co-ordinators returned their interpretations of the framework by email.

14

UNAWE Summative Evaluation Framework

Motivation

(Revised Oct 2012) Objectives

Evidence

Enjoyment Inspiration Curiosity Tenacity

-Children are doing the tasks with pleasure -Children seem enchanted -Children react with diligence in front of the proposed activities -Children demonstrate attention -Children apply perseverance / tenacity -Children manifest inquisitiveness -Children introduce some complex questions -Correct use of vocabulary or gesture to name objects and phenomena observed in sky -Grouping objects/ phenomena to indicate developing understanding of astronomy concepts -Making conjectures available to be contrasted -Developing some experiences related to the hypothesis -Linking new information with existing conceptions of the same or different areas -Removing previous points of view according to new inputs -Sharing with others their new knowledge

Scientific Skills

Develop Scientific Thinking & Problem Solving Techniques; Planning & Conducting Investigations Observation, Identification, Classification, Making interconnections, Changing Perspective & Communication From SA curriculum: example of school wording of this section Discussing and Questioning Ask questions which can be answered through an investigation Use scientific language regularly in discussions Planning Plan and carry out a test to collect evidence Select information from a range of resources Observing Decide what observations need to be made Select appropriate equipment for observation or measuring results Interpreting Draw conclusions linked to scientific knowledge and understanding.

Recognise patterns and trends based on the observation or investigation Ideas and Evidence Recognise that scientific ideas are based on evidence which can be verified by observations Use the imagination together with scientific knowledge to understand and think about why something happen Recording Decide on an appropriate method of recording Present results using tables, graphs, pictures Evaluation Review the work and reflect on the results

15

Intercultural Attitudes

Universe Knowledge

Knowledge & Understanding Observing, exploring and discovering: 1. The Sun, Sun relative position, Sun light (shadows), Day/night cycle, time zones, the seasons, the Moon, the Earth as a planet, awareness of the existence of water and of the Earth atmosphere and Sun light for the development of life on Earth, Solar and Moon eclipses,...

2. The Solar system: planets characteristics and movements, wharf planets, asteroids, comets,...

3. The Stars in the night sky, the constellations, orientation, the Lifecycle of stars, the formation of stars and planets,...

4. Our place in the Milky Way, Family of galaxies,...

5. Current developments in astronomy 6. Magnetic fields (compass, northern lights,...)

Valuing different cultural perspective.

Recognising different physical perspectives.

Positive attitude towards astronomy.

Valuing inclusive education Working Individually & in Teams

Direct observation and/or recording of naming, first explanations, discussing, drawing, construction, creative responses, movements and dances, etc. to demonstrate knowledge of one of the features listed. left

-Demonstrating awareness of different cultures -Ability to observe and explain differences in phenomena in different countries -Statements of future activity with regards to astronomy -Act on an appropriate way in a frame of diversity

16

3. Stage IV: Resource Production In order to support both National Project Manager and the wider network in data collection, several resources were produced.

EU-UNAWE Evaluation Handbook (See Appendix A/ http://unawe.org/static/archives/guides/pdf/EU_UNAWE_evaluation.pdf).

This 38 page handbook covers: •

Project goals

•

Rationale

•

Methodology

•

Resources

•

Observation sheets

•

Letters for teachers

•

Suggested data gathering process

Postcards for evaluation with children These were produced in response to discussions with National Project Managers following the handbook publication. It became clear that a quick method of gathering opinions and demographic data was needed for a range of activities including school workshops and drop in events.

Postcards were distributed in October 2013 at Heidelberg International meeting.

17

Families and groups with children age 10 and under

An additional version was produced for early years children.

18

4. National Impact In order to investigate impact, different methodologies have been used, as appropriate to the resources available within each partner country.

Quantitative methods Teacher surveys have been used to gather evidence about the impact of teacher workshops.

These have used both closed questions (i.e. quantitative data) and open ended responses (qualitative data, which has been coded).

Qualitative methods In depth qualitative methods have been used in case studies to gather video evidence with teachers and pupils.

Case studies will be presented under the UNAWE evaluation framework learning domains.

Domain I. Motivation

Case study Evidence a. Children Italy- children’s work b. Teachers- German CPD survey

II. Scientific skills

a. Children- UK- video observation of class teaching b. Teachers- The Netherlands CPD interviews

III. Knowledge

a. Children- Italy b. Teachers- Spain

IV.

a. Children- Leiden- South Africa film evidence

Intercultural awareness

b. Teachers-Italy- South Africa teacher report

V. Legacy

a. Evaluation with teachers b. Evaluation capacity building c. Peer review site for resources

19

4. I. Motivation The domain of motivation refers to the following objectives: enjoyment, inspiration, curiosity and tenacity. The sort of evidence that would be acceptable for this domain is evidence from teachers that they had a positive association with Astronomy workshops, and also children’s behaviour: • • • • • • •

Children are doing the tasks with pleasure Children are engaged in the activity Children react with diligence in front of the proposed activities Children demonstrate attention Children apply perseverance / tenacity Children manifest inquisitiveness Children introduce some complex questions

(Wording chosen through collaborative process with NPMs, Florence October 2012)

Children at Brownhall Primary School, Dumfries

20

4. I.a. Motivation: Evaluation with children, Italy Location

School: Circolo didattico di Zafferana, Etnea Children made essays in response to taking part in a UNAWE activities over several weeks: •

Stars tube

•

Build an astrolabe

•

Build the constellation cube

•

Build the earth with the constellations around

•

Model of Orion with its Nebula

•

Sundials

•

Observe your shadow (open air)

•

Sunspots observation

•

Activity on the extraterrestrials

Lara and Alessandra had made sure that content was linked to the topic the children were already studying in class; so the Year 2 children had constellation stories linked to American Indians (their school unit). The Year 3 children had narratives tying in with Ulysses. Planning

21

to respect the existing curriculum units makes in more likely that activities will be used again.

Following the activities, children were asked to write freely about the night sky and its mythologies and phenomena.

22

Using an observation paradigm, NPMs and evaluator James Bradburne recorded children’s responses to a variety of activities:

1.motivation

Objectives reached in the spontaneous writing Enjoyment Inspiration Curiosity Tenacity

Evidence

Children are doing the tasks with pleasure writing spontaneity texts and drawing.

-Children react with diligence in front of the proposed activities and they write texts longer than requested or adding more essays.

-Children introduce some complex questions. Many of the texts relate to lives of the stars and the existence of extraterrestrial life.

23

4. I. b. Motivation: Evaluation with teachers, Germany Location

House of Astronomy, Heidelberg, 2012. Workshops 13a and b. N=78 This teacher survey, carried out after a teacher CPD session, presents evidence that teachers had positive associations with parts of the course. They were particularly positive about Cecilia Scorza and Natalie Fischer’s enthusiastic presentation skills (responses have been coded).

•

It has been fun to work on the theme "Astronomy", I'm looking forward to the 2nd part of the fall

•

Ms. Fischer is contagious with her storytelling and knowledge

•

Verve

•

Ms. Fischer is an excellent speaker and has given us the theme so exciting.

•

theory + practical work I find very positive

24

4. II. Scientific Skills The domain of Scientific skills refers to developing scientific thinking and problem solving techniques, such as observation, identification, classification, making interconnections, changing perspective & communication.

Evidence for this domain has been gathered from teachers in the form of self-report in questionnaires, observation of teacher training workshops using video, and teacher interviews. Evidence has been gathered from children in the form of discussion and group tasks, and school class observation following teachers attending training workshops.

The kinds of evidence that would demonstrate Scientific skill development is as follows: -Correct use of vocabulary or gesture to name objects and phenomena observed in sky -Grouping objects/ phenomena to indicate developing understanding of astronomy concepts -Making conjectures available to be contrasted -Developing some experiences related to the hypothesis -Linking new information with existing conceptions of the same or different areas -Removing previous points of view according to new inputs -Sharing with others their new knowledge (wording chosen through collaborative process with NPMs, Florence October 2012)

Teachers discussing science skill demonstrations with Wouter Schrier in Leiden, April 2013

25

4.II.a.Scientific Skills:Evaluation with children, UK Location

Brownhall Primary School, Dumfries, Scotland, 2013

Description Teachers who attended the Teacher training workshop in Dumfries were contacted to ask for permission to visit their classroom and observe a session related to the workshop content. One teacher was selected for a visit, two weeks after the teacher training workshop. Parental consent forms were distributed and collected for permission. The session was recorded, and children were interviewed using a semi-structured interview technique which covered the domains in the UNAWE framework: motivation, skills, knowledge, intercultural awareness and legacy. A class from Brownhall Primary School in Dumfries was observed taking part in an activity inspired by the workshop.

26

The eight year old girl in the image on the previous page is demonstrating measuring a paper mache earth she has made, in an activity designed to show the relative sizes of the moon, earth and sun. Through doing this she is using communication skills and measurement.

The nine year old boy shown below is demonstrating communication skills through explaining why people in different countries are awake at different times. He is turning the earth in the correct direction, and holding his left hand to represent the sun shining on different countries at different times. He identified where Dumfries was, and referred to the Pole star which is fixed to the classroom wall.

(This interview provides evidence for a range of domains, including knowledge and intercultural awareness, although scientific communication skills are being focussed on).

Permission was sought from parents for filming using a parent letter as supplied in the EU UNAWE evaluation handbook.

The film from this class observation is supplied as appendix C.

27

4. II.b.Scientific Skills: Evaluation with teachers, The Netherlands Location

Leiden University, The Netherlands, March 2013

Methods Teachers were interviewed following a teacher training workshop in Leiden, April 2013.

Filming consent forms were distributed for permission. They were asked the following questions, using the framework domains as a basis: 1 What did you enjoy about the course?

2 Did you gain knowledge about the universe? Please explain.

3 Did you develop any new skills or practice existing skills?

4 Did the course make you consider cultural perspectives differently?

5 Did you share aspects of this course with other staff? How? 6 Have you got any recommendations for improvements to your course?

7 Have you embedded any new materials into the curriculum as a result of this course?

28

Results were recorded and the wordle on the next page shows the skills which teachers commented on developing through the workshop. n=65 comments

The emphasis on moon interpretation and curiosity is consistent with the emphasis on explaining ways to show the implications of the relative movement of the moon, earth and sun, featured in this workshop.

The image below is a still from workshop observation by the evaluator in March 2013. It shows teachers practising demonstrations of sunlight.

In response to the question

29

‘Did you

develop any new skills or practice existing skills?’

Many teachers focussed on pedagogical approach as a skill they had acquired: •

new skills! inquiry learning and how I can use it in my classroom

•

how to explain clearly to children using visuals

•

the usefulness of discovery learning

The film of this teacher CPD observation is supplied as appendix D.

30

4. III.Knowledge EU-UNAWE programmes aim to develop teacher and pupil knowledge about astronomy through observing, exploring and discovering: 1. The Sun, Sun relative position, Sun light (shadows), Day/night cycle, time zones, the seasons, the Moon, the Earth as a planet, awareness of the existence of water and of the Earth atmosphere and Sun light for the development of life on Earth, Solar and Moon eclipses,...

2. The Solar system: planets characteristics and movements, wharf planets, asteroids, comets, 3. The Stars in the night sky, the constellations, orientation, the Life-cycle of stars, the formation of stars and planets, 4. Our place in the Milky Way, Family of galaxies, 5. Current developments in astronomy 6. Magnetic fields (compass, northern lights,)

(wording chosen through collaborative process with NPMs, Florence October 2012)

Trial postcard evaluation from South Africa.

31

4.III.a Knowledge: Evaluation with children, Italy.

Description Project evaluator James Bradburne worked with Italian National Project Managers Lara Albanese and Alessandra Zanazzi during 2012/13.

This work was presented at CAP 2013 in Warsaw, with the title: ‘Assessing and Evaluating A case study in the framework of EU-UNAWE Italy project’. For more information see: http://www.slideshare.net/unawe/cap2013-assessing-and-evaluating-case-study-by-unawe-it

Each country draws on different influences for teaching pedagogy, and specific to the Italian context is the use of The Reggio Approach. It is a view of learning which acknowledges prior experience and the social construction of learning (co-constructivist views of learning are also well established in teacher pedagogy internationally, although with different emphases).

Case study: Year 3 (age 7/8), Ronco Briantin, Milan Children saw a planetarium show, and then discussed which areas they were most interested in.

The areas they chose were: •

Birth and expansion of the Universe

•

The constellations and the stars

•

The Solar System formation

•

The forces of the Universe

•

The creation of the Earth

They then had a skype call with South African children, discussing those topics. The fourth event was a night in school, where children were taken out to star gaze. Finally, a party was held where children operated the planetarium, and made exhibitions of astronomy.

When documenting children’s conversations, it was clear that they had developed scientific knowledge about the sky. The wordle below shows the relative frequency of vocabulary in children’s conversations about the sky at the start of participating in the UNAWE programmes.

32

The following wordle shows their conversations one year after participating in the UNAWE programmes.

Therefore, there is evidence that children moved from a spiritual view of the sky, to understanding a rational scientific view and knowing key astronomy vocabulary.

33

4. III.a.Knowledge: Evaluation with teachers, Spain

Spanish programmes are baed at the Universitat Politècnica de Catalunya (UPC) Barcelona Tech offered teacher training programmes and summer schools which included specific training for pupils with special needs

The evaluations below are used to demonstrate improvement; clearly the effectiveness of workshops improved in response to teacher comment, and 88% of teachers consider that they learnt new content.

34

4. IV. Intercultural Awareness The domain of intercultural awareness is defined by development in being able to valuing different cultural perspectives, recognising different physical perspectives, and valuing inclusive education. Children need to be able to working individually and in teams.

Teacher self- report and workshop observation have allowed data collection for this aspect.

For children, evidence can be seen in the form of: -Demonstrating awareness of different cultures -Ability to observe and explain differences in phenomena in different countries -Statements of future activity with regards to astronomy -Act on an appropriate way in a frame of diversity (wording chosen through collaborative process with NPMs, Florence October 2012)

35

4. IV. a. Intercultural Awareness: evidence of children’s participation A number of links have been, and continue to be facilitated through skype, through connections made possible by the EU-UNAWE programme.

Locations

The vimeo clip above shows children from Nompumelolo Primary in East London, South Africa, and Morskring Primary in Leiden, The Netherlands. The project was conducted for one month, with the focus being on moon phases and seasons. Learners observed the different moon phases from the 21st of May to the 21st of June. Over the month the learners watched the night skies and recorded their observation on a moon calendar. This information was reinforced through various activities about the moon in the classroom. The learners made moon phase flip books, weighing on the moon and creating moon craters. http://vimeo.com/52849722

36

4. IV. b. Intercultural Awareness: teacher report Locations:

Italian co-ordinators travelled to South Africa during Science Week 2012 (July- August). They organised activities at 5 schools, seeing approx. 600 pupils and 20 educators.

Female South African astronomer Shazrene Mohamed offered a discussion session with pupils to initiate interest in the topic. Pupils then saw a planetarium show (the first time they had seen this). Italian teaching using puppets extended their understanding of the constellations Orion, Crux and Scorpio. A live link took place between South African teachers and Zafferana Etnea school in Sicily.

Teachers were surveyed about children’s ability following the exchange, and there is evidence that learners developed cultural awareness through teacher report.

The direct exchange projects are in addition to teaching which explicitly shares different cultural approaches to, for example, explaining the seasons. Throughout the project

37

resources which encourage appreciation of narrative in early astronomy experience have been produced and translated, such as the Earth Ball and activity book.

Italy’s Lara Albanese won the Anderson award for her work in developing narratives about science and technology: http://www.unawe.org/updates/unawe-update-1366/ For examples see: http://www.unawe.org/resources/education/

A number of links have been made between countries in the wider network. The key challenge is network connectivity for skype calls and direct communication.

Examples on the UNAWE video site provide further evidence of collaborative projects. http://vimeo.com/unawe/videos

38

4. V. Legacy Legacy refers to the ongoing impact of the project, beyond the completion of the project in December 2013. For teachers, this includes embedding astronomy learning from UNAWE into the curriculum, and continuing to teach using resources from UNAWE. They may share their knowledge with other teachers in school based continuing professional development courses, and change long term planning. For children, this includes children’s aspirations in studying astronomy, and assimilation of learning as outlined in domains listed above.

In addition to the legacy in terms of teacher and pupil learning, what are the long term benefits of this programme?

This was assessed through asking teachers and pupils about their long term intentions regarding astronomy and what they had learnt through participating in EU UNAWE programmes.

4. V. a. Evaluation with teachers Location

Case study: United Kingdom Teacher training responses, Nov 2011 , n = 48 participants In a survey, teachers were asked to state what they would do following the CPD course:

Initial Action Plan - as a result of this training I will: “Endeavour to deliver the programme to my own P6 class within our World Around Us planning /teaching. Disseminate today’s programme to my other 3 year group colleagues in P6 and possibly to the P7 year group. Explore resources indicated to us today and try to secure funding for those requiring purchases”.

39

Follow up: Have you ever had a chance to talk to your colleagues about EU-UNAWE to disseminate what you have learned?

“I disseminated some of the materials at a staff meeting. I have also incorporated the materials into a unit called Razzle Dazzle – all based on light, planets, constellations etc. and have shared it with two colleagues from other schools who have also used it”.

There are many examples of similar responses.

For example, the teacher below is a Scottish teacher at senior management level, who stated the intention to build the materials into existing astronomy curriculum areas. She said ‘we are aiming to develop experiences and outcomes (Scottish curriculum Es and Os) which will take learning from the primary curriculum into the secondary curriculum in biology, chemistry and physics’. She went on to explain that astronomy is covered in a two year cycle because the school size is small.

It is clear that teacher autonomy to include material for teaching in their own classes varies according to their level of seniority and the size of the school.

It is therefore recommended that teachers attend the workshop in pairs if possible, with one teacher a junior, enthusiastic member of staff, and the other being a more advanced member of staff from the same institution. In this way, the desire to take on ideas will be facilitated by the ability to overcome bureaucratic barriers which may arise.

In addition, it would be helpful to ask teachers to bring copies of the short, medium and long term planning in order that they can share and pinpoint exactly where the opportunities for modification of existing plans are. This will also help highlight any paucity in potential for change which needs to be addressed institutionally.

40

4. V. b. Evaluation capacity building workshop There was a greater legacy than anticipated initially in terms of building evaluation capacity for education professionals in both the partner countries and the wider network.

The need to develop understanding in evaluation was identified by network members, and therefore a three day evaluation workshop was held in Heidelberg, October 2013.

The workshop was supported by a wiki site where participants could contribute comments in response to the workshop: https://evaluationunawe.wikispaces.com/home Description: Tuesday 8th October- Thursday 10th October 2013, Heidelberg This workshop built on the International UNAWE workshop in March 2012 in Leiden.

Contents: •

current international educational evaluation context •

examples of good practice • •

methods

report writing

Participants were provided with slides via the wiki site; links are active via slide share.

Participant locations

41

The outcome of day one was a booklet of case studies, sharing information about evaluation in participant’s contexts: It is supplied as Appendix D to this report and can be seen at: https://issuu.com/g.kimble/docs/unawe_case_study_activity_example/1?e=3149744/5151133

Days two and three covered methods, reports and useful software.

42

4. V. c. Peer review site for astronomy education resources A need was identified to assess the quality of education resources provided via the website.

Therefore, the platform AstroEdu has been launched (Nov 2013) to provide a means to select beneficial and effective learning resources for publication: http://www.unawe.org/press/UNAWE1302/

43

5. International Impact The National Programmes were evaluated using the evaluation framework, with 5 domains, as described. However, this framework was devised during the project lifetime and it would be beneficial to move on from this programme by using the existing framework as a starting point for future projects.

It is clear, and the literature supports the fact that it is an ambitious task to bring together different learning approaches and philosophies under one common framework, and in reality local adaptation has been the solution to allow assimilation of evidence coherently, despite different philosophies.

This section will return to the original programme goals, and show where existing activities and resources meet the intentions of the programme.

Examples are drawn from both National programmes and the wider international network which has expanded throughout the programme timescale.

5. 1. Train and empower primary school teachers in 6 countries to include astronomy and space topics in the classroom.

Using statistics from the UNAWE website, it is clear that activities have taken place across a range of locations. The previous sections of this report detail the learning which has taken place in Case study format.

44

5.2 Develop and translate hands-on material, where appropriate emphasising EU and SA science and technology.

Universe in a Box is one example of a resource which includes hands-on resources, and has been widely distributed; see for example video link. http://vimeo.com/54010475 and University of Leiden highlight: http://nieuws.leidenuniv.nl/nieuws-2013/lespakkettenuniverse-in-a-box-gaan-de-hele-wereld-over.html In addition to this educators have been encouraged to develop their own hands on resources, and most importantly to share them with others. At the International meeting in October 2013 a prize was awarded for the best hands-on resource as an incentive for educators to share their work. Previously, the resource ‘Deadly Moons’ won the Science Prize for Online Resources in Education (SPORE) award, chosen by a panel from Science Magazine.

45

5.3 Provide a network for exchange of expertise and material between educators. Lay the groundwork for expansion of the programme throughout the EU, Associated Countries and ICP Countries.

As shown in the image on page 41, 47 countries have astronomy programmes using Universe Awareness resources. In addition to formal links, this is supported by social networking.

46

5.4 Act as a showcase for EU and SA astronomy/space and related technologies, by disseminating the products among very young children, their teachers and their families.

Space Scoop news service is unique in providing information about innovations in astronomy, which are suitable for children aged 8 and above, their teachers and families.

These updates are translated in a wide range of languages as shown below.

Other resources are for younger children such as the Earth ball and Universe in a Box.

5.5 Use astronomy/space products to stimulate awareness and strengthen public support for EU and SA space science research and technology.

UNAWE activities take place in a range of formats. National Project Managers and others have been involved in many festivals, attended by the public, where UNAWE resources have been displayed. Whilst the focus of this evaluation has been on teachers and children, members of the public have also seen UNAWE resources at some of the festivals attended by co-ordinators during the project lifetime, for example: •

EU Festival of Europe, Brussels, Belgium http://www.festivaldeuropa.eu/en

•

‘The Dreamgazer’ children’s astronomy story film premiered at a Leiden Film Festival, The Netherlands http://www.nachtvankunstenkennis.nl/

•

Festival Nauke, Belgrade http://www.festivalnauke.org/

•

SciFest, South Africa http://www.scifest.org.za/

•

BBC Stargazing live http://www.bbc.co.uk/programmes/b019h4g8

47

5.6 Stimulate the next generation of EU and SA engineers and scientists, particularly girls.

The goals of UNAWE activities are to motivate, develop skills and knowledge about astronomy as a potential foundation for future study in science and engineering.

UNAWE activities have contributed to some specific initiatives to inspire girls to study science and engineering at advanced levels, such as Green Light for Girls http://greenlightforgirls.org/. UNAWE took part in an event in Brussels, Belgium in October 2012, where 300 girls participated in interactive workshops and the UNAWE activity: designing an alien life form.

5.7 Contribute to the integration of disadvantaged communities in participating countries.

It is clear that there is an aspiration to work with disadvantaged communities, both within the partner countries and internationally. International outreach has focussed on disadvantaged communities for example: •

East Timor; see project report: http://www.unawe.org/static/archives/reports/pdf/TL_VT2012_project_EN.pdf

•

Spain- Universitat Politècnica de Catalunya (UPC) Barcelona Tech offered teacher training programmes and summer schools which included specific training for pupils with special needs

•

Northern Ireland initial project team meeting (2004) highlights the issues facing disadvantaged communities in their context: http://issuu.com/unawe/docs/unawe_armagh_report_236_

Collection of demographic information about schools participating in teacher CPD by NPMs has allowed comparison with school performance and indices of deprivation in catchment areas to identify a measure of disadvantage. Given the scale of the programme, there are a number of different criteria which can be used to indicate disadvantage; ranging from parental income to percentage free school meals in a given school. It is a feature of the literature about evaluating higher education outreach programmes that collecting individual socioeconomic data is problematic; for example it is clear to see how enquiring about parental occupation could prove problematic for early years children in disadvantaged countries.

48

5.8 Strengthen collaboration between EU and SA over mutually beneficial scientific, technological, educational and social topics.

•

Links between schools in Leiden, The Netherlands and East London, South Africa as described

•

Links between schools in Sicily and South Africa as described. The image below shows South African pupils using puppets (Italian teaching method) to extend understanding of the constellation Orion.

•

South African pupils visited the House of Astronomy, Heidelberg, Germany, A videoconference was organised between the European Parliament and Dr Cecilia Scorza, German National Project Manager.

5.9 Provide significant added value for Europe's expenditure on astronomy and space sciences for a modest incremental cost.

5.10 Pooling complementary expertise and resources of 6 partners gives a project whose whole is greater than the sum of its parts.

The aggregation of resources by UNAWE has brought together a network of astronomy professionals, whose activities are complementary. For example, links with astronomy education organisations such as Galileo Mobile, who recently toured Chile, Peru and Bolivia (see http://www.galileo-mobile.org/) has meant that a wider range of resources are accessible to partners.

Attendance at the International workshops provided catalysts for inspiration and cultural exchange between programme co-ordinators, and participants were specifically encouraged to set up links, with the aim of furthering the UNAWE principles of intercultural collaboration within their future programmes.

49

6. Recommendations This project has shown evidence of project goals. Owing to the size of the project and the limited evaluation capacity, a case study approach has been used to present examples. The evaluation framework collaboratively agreed was used as a basis for considering children’s and teachers learning. In order to maximise learning from this project for future programmes, it is important to consider recommendations for subsequent astronomy education programmes for early years children. These will be presented and explained using the project goals as a starting point.

7.1 Train and empower primary school teachers to include astronomy the classroom.

It is recommended that teachers are invited to CPD workshops in pairs, ideally with one senior and one junior member of staff, in order to maximise the ability of each school to embed new materials at strategic and operational levels. It is recommended that teachers are asked to bring copies of their short, medium and long term planning, in order that a part of the day involves sharing with other schools where they can fit the new information into timetables. The purposes of this are twofold; the first is for the institution to realistically plan. The second is for comparison with other organisations to bring a flexible approach to considering other ways of organising the curriculum.

7.2 Develop and translate hands-on material, emphasising EU and SA science and tech It is recommended that existing resources are peer reviewed using the website AstroEDU, in order to assess and improve quality.

7.3. Provide a network for exchange of expertise and material between educators - Lay the groundwork for expansion of the programme throughout the EU, Associated Countries and ICP Countries.

The network should continue to seek sources of funding to maintain the critical mass of astronomy educators and collective approaches which have been facilitated through this programme.

7.4 Act as a showcase for EU and SA astronomy/space and related technologies, by disseminating the products among very young children, their teachers and their families.

The infrastructure which has allowed dissemination and translation of resources should continue.

7.5 Use astronomy/space products to stimulate awareness and strengthen public support for EU and SA space science research and technology.

Festivals are ideal formats for engendering public support. Future programmes should include capacity for evaluating public opinion as well as children and teachers, in order to

50

assess the impact of public activities.

-There needs to be clear understanding of the difference in format needed for curriculum linked workshops, suitable for a specific age group and specific duration; and effective drop in activities which are flexible in time and audience. A worthwhile future activity would be summarising the reflections of NPMs on these subtle differences, in order to communicate the degree of astronomy activity literacy which the network has amassed.

7.6 Stimulate the next generation of EU and SA engineers and scientists, particularly girls.

Longitudinal evaluation would be needed in order to assess the long term impact for pupils who have participated in UNAWE activities. It is recommended that a methodology and capacity to carry out such in depth research in future would be beneficial to the field of astronomy education, science communication and higher education outreach literature. The records kept by NPMs showing the schools and teachers who have taken part would allow evaluation of pupils who have taken part during the current cycle of UNAWE activities. This could be done around the age where they make higher education choices, for example. A brand new method of iPad personal profile building is recommended for this task.

7.7 Contribute to the integration of disadvantaged communities A solution needs to be found to collect demographic data about disadvantage for future programmes, if they are to engage the public. Whilst school and teacher data are linked to catchment areas and schools’ reports of socioeconomic indices, it would be beneficial to devise an agreed method of gathering community characteristics.

7.8 Strengthen collaboration between EU and SA Network connectivity is improving all the time. Therefore the connection issues which are inevitable with skype calls to remote areas should improve in future. However, it is recommended that a checklist for setting up test calls, plus skype session guidance would be useful preparation to provide for schools taking part in international links. Light plug-in software such as ‘cover-it’ could be used to reach areas with low coverage.

7.9 Provide significant added value for Europe's expenditure on astronomy and space sciences for a modest incremental cost.

7.10 Pooling complementary expertise and resources of 6 partners gives a project whose whole is greater than the sum of its parts.

UNAWE co-ordinators should be encouraged to maintain personal connections through the range of related organisations, in order to build on the network. Realistically events and activities that span several organisations would be important milestones to keep momentum in the network.

51

8. Conclusions The programme has delivered goals according to the initial aims stated on page 5.

8.1 Primary school teachers have included astronomy in their classrooms.

The in depth case study presented here showed a teacher who attended a workshop in Dumfries, offered by EU-UNAWE Northern Ireland. The then went on to teach eight year olds using demonstrations shown in the CPD workshop. Interviewing and observing children in his class, approximately two weeks after the CPD workshop revealed that they had learnt scientific skills. Teachers should be invited in mixed experience pairs to sessions. They should be asked to bring curriculum planning documents with them, and this should be the basis for a task within the session.

Analysis and synthesis of a diverse range of evaluation was made possible through elucidation of the EU-UNAWE Evaluation framework covering the five domains Motivation, Skills, Knowledge, Intercultural attitudes and legacy in order to demonstrate learning.

This was necessary because the philosophy of teacher education varies worldwide. It is therefore important to consider what co-ordinators understand by the term ‘learning’.

The epistemology which underpins the UNAWE Evaluation handbook includes the five domains stated above. It has been beneficial to use a metalanguage when explaining data collection rationale. However, the evaluation capacity is low for NPMs and investment in evaluation at the start of the programme would be helpful in future. This would bring more time for educators to blend existing and new evaluation.

In practice, the evaluation has worked best where there is local adaptation. For example, modification of existing surveys in Northern Ireland, South Africa and The Netherlands. Italy and South Africa have worked with evaluators to produce additional reports; see Appendix B for example. This finding of the importance of integrating local contexts with a structural ‘bigger picture’ approach is consistent with international education evaluation literature (Levin-Rosalis et al., 2009:191 in Ryan and Cousins, 2009) which contrasts human agency level evaluation with ontological structural approaches.

Practically, this means that similar programmes could use the EU- UNAWE framework or a similar one as a starting point for local modification and evaluation.

There is evidence that all NPMs have put in considerable effort to achieve an ambitious programme of activities designed to inspire primary pupils about astronomy.

52

8.2 Hands-On materials Universe in a Box and other hands on learning materials have been produced and trialled. It is recommended that existing and future resources are submitted to the website AstroEDU for peer review and subsequent dissemination to teachers.

7.3. Provide a network for exchange of expertise and material between educators The network exists in 47 countries and should continue.

7.4 Act as a showcase for EU and SA astronomy/space and related technologies, by disseminating the products among very young children, their teachers and their families.

Space Scoop resources are an example of a unique resource which shares research regularly with young pupils, about astronomy. These should continue.

7.5 Use astronomy/space products to stimulate awareness and strengthen public support for EU and SA space science research and technology.

Festivals such as SciFest and BBC stargazing have engaged members of the public in UNAWE activities. Future projects could focus on drop in activities for a wide range of audiences, including the need for flexible activities which vary in duration and content pitch.

7.6 Stimulate the next generation of EU and SA engineers and scientists, particularly girls.

UNAWE activities involved female role models, such as Shazrene Mohamed in East London, South Africa. They have worked with Green for Girls science aspiration group in order to encourage more women astronomers. There is the potential to carry out long term evaluation of this project as pupils approach higher education choices 7.7 Contribute to the integration of disadvantaged communities Future programmes should identify indicators and locations of high need prior to commencing activities.

7.8 Strengthen collaboration between EU and SA Online links between schools are beneficial and should continue where possible.

7.9 Provide significant added value for Europe's expenditure on astronomy and space sciences for a modest incremental cost.

7.10 Pooling complementary expertise and resources of 6 partners gives a project whose whole is greater than the sum of its parts.

Seeking opportunities for collaboration, and the resultant links, is a particular strength of this workshop.

53

References •

Museums, Libraries and Archives Council (2008) Generic Learning Outcomes. UK http://www.inspiringlearningforall.gov.uk/resources/

•

Falk, J and Dierking, L (2000) Visitor Experiences and the Making of Meaning.

Altamira, UK

•

Friedman, A. (Ed.). (March 12, 2008). Framework for Evaluating Impacts of Informal Science Education Projects http://insci.org/resources/Eval_Framework.pdf

•

Nichols, B. H., & Zeidler, D. L. (2012). Teaching earth smarts: A pragmatic, nonpartisan educational construct for socioecological literacy. Presented at the Annual International Conference of the National Association for Research in Science Teaching, Indianapolis, IN.

•

Ryan, K. and Cousins, J.B. (2009) The SAGE International Handbook of Educational Evaluation. SAGE, Chicago

54

Appendix A- UNAWE Evaluation handbook

Appendix B- Italian Evaluation report Appendix C- UK film clip Appendix D- The Netherlands film clip Appendix E- Workshop Case Study Booklet

55

Image: Dario, age 9

56