4. IMPACT AND EVALUATION This section gives an overview of the impact and reach of the project outputs listed in Section 3, and gives a full breakdown and evaluation for each area of who the engaged audiences were, how they engaged with the project, and what impact the project had on their knowledge, attitudes, and behaviours11. Overall, findings include that the digital and social media audiences were up to fifty times higher than predicted and that the Pot Gan viewers and attendees reported very high levels of enjoyment and understanding. The Pot Gan has influenced even audiences with existing climate change and development expertise. In many places, testimonies and direct quotes from participants and viewers are used to show how the project truly made an impact for so many people.
4.1 Methods of Evaluation A core component of the project has involved monitoring and evaluation (see Table 3 for a full list of all activities). This has primarily involved carrying out video interviews and online questionnaire surveys to gather feedback about the project outputs. A total of 165 people completed interviews and surveys, consisting of viewers of the live Pot Gan performances in Dhaka, online viewers of the documentary in English or Bengali, online viewers of the Pot Gan performance video, and attendees at the public events held in the UK. This has been supplemented with analysis of social media and digital media engagement.
Table 3: Overview of methods used to evaluate engagement and impact
Live Pot Gan performances - Audience numbers and composition. - Feedback from online questionnaire surveys, including descriptive statistics and thematic analysis. - Follow-up semi-structured interviews with survey respondents, to see how they have put their learning from the Pot Gan into practice in their work and daily life.* - Video interviews - Pot Gan interactive session transcripts (copies in English and Bengali available on request) Teaching and learning - Total number of students involved in project. - Number of University of Manchester students that utilised digital resources. - Testimonies from University of Dhaka students and lecturers involved in project. - Feedback from online questionnaire surveys with University of Dhaka students on learning outcomes, including descriptive statistics and thematic analysis.* - Online questionnaire surveys with secondary school teachers that used the project resources, to estimate the total number of secondary school Geography students that were taught content from the project teaching resources.* - Number of downloads of KS3 Geography teaching resources.* - Feedback from online questionnaire surveys with Secondary School students on learning outcomes, attitudinal shifts, and behaviour change, including descriptive statistics and thematic analysis.* - Analysis of Geography homework to assess learning outcomes and attitudinal change.* - Number of pledges and thematic analysis of pledges made by Secondary school students to tackle climate change.* - Follow-up semi-structured interviews with Secondary school students that pledged to tackle climate change to find out which impacts actually arose.*
Public events - Audience numbers and composition. - Multiple choice questions during events, including hand counts and polling. - Feedback from online questionnaire surveys, including descriptive statistics and thematic analysis. - Follow-up semi-structured interviews with survey respondents, to see how they have put their learning from the Pot Gan into practice in their work and daily life.*
Social media** - Project web page views. - Reposts of videos and other project materials on external websites. - Twitter tweets, including retweets, likes, and use of the project hashtag. - Thematic analysis of Twitter conversations.10 - Facebook shares, likes, comments, reactions, conversations and engagement with the project Facebook page. - Semi-structured interviews with those behind key Twitter accounts that commented on project events and/or outputs, to triangulate the interpretation of the qualitative data from the thematic analysis of Twitter conversations.*
Table 3 outlines how impacts on behaviour are being monitored and evaluated. Section 4 mostly deals with potential impacts that the project has on behaviour, the specific impacts that actually arise will be captured later in the project cycle.
11
12 Thematic analysis of Facebook conversations is not presented in this report as posts and comments from users focused on promoting project events, sharing outputs, or project successes and providing positive feedback on the project, rather than generating wider discussion on the key themes and issues raised in the videos and other project material.
24 PAGE