UN CC: Learn Cities and Climate Change Course Evaluation

External Evaluation of Cities and Climate Change

An External Evaluation Submitted to: The UN Climate Change Learning Partnership Issued To: Jane Smith Organization: UN Climate Change Learning Partnership Street Address: 405 East 42nd Street City, State, Zip: New York, NY 10017 Primary Contact: Jane Smith Contact Email: jane.smith@un.org Issued By: A2 Evaluation Professionals Street Address: 1551 S. Rural Road City, State, Zip: Tempe, AZ 85027 Email: Michelle.King@A2Eval.org ; Sonali.Perera@A2Eval.org ; Tina.Cipolla@A2Eval.org Date: 4/18/2021

1

Executive Summary Overview A2 Evaluation Professionals of Tempe, Arizona were contracted to conduct a 6-month external evaluation of the Cities and Climate Change eLearn Short Course. This final evaluation report is being submitted by A2 Evaluation Professionals to the UN Climate Change Learning Partnership (UN CC:Learn). In this report, we share the results of our mixed-methods evaluation of the course, the comprehensive data set, and we make recommendations based on our findings.

Data A2 conducted a mixed-methods evaluation that included data from a variety of sources.

A2 used both quantitative and qualitative data in the evaluation in order to come to conclusions about the learning outcomes, the effectiveness of the instruction, the student experience, the flexibility of the design and delivery of the course, the technical functionality, and the materials.

The data sources used include: ● A document analysis of four relevant documents ● A survey of 545 learners who have completed the course in the past 12 months ● A survey of 378 STEM-school science teachers who have used the plug-and-play curriculum during the past 12 months. ● Final quiz scores of the 923 participants of the evaluation ● A Zoom-based focus group consisting of seven of the STEM teachers who completed the course and used the materials from the course in their own classrooms.

EDT 506: Evaluation Final Report

2

Evaluation Results and Implications The Cities and Climate Change eLearning course is modularly designed with the intention that teachers will reuse parts of it in standards-based curricula. What our evaluation revealed is that teachers are hesitant to use the content for several reasons, chiefly because it is too difficult for them to adapt the content for use in their classrooms. As it stands now, the course content is too complex for younger audiences and not engaging enough for secondary school students. Furthermore, our evaluation revealed that the course needs changes to be effectively used for professional development. STEM teachers feel that the course is not appropriate for their professional development because the teachers are already familiar with the concepts covered in the training. The course tries to be too many things to too many audiences and it is not being used as it was intended.

Our recommendations are that this course should be reimagined into two separate deliverables. The first deliverable should be a library of short, modular plug-and-play materials that teachers can use in their classrooms. These learning objects should be brief, focused on a very specific topic, and be targeted toward different age groups. The second deliverable should be a professional development course for STEM teachers that is paired logically with the plugand-play materials but that contains up to the minute materials on the latest findings about the reciprocal effects of cities and climate change. While we understand that this would require regular updates, to be an effective professional development tool, this is necessary.

EDT 506: Evaluation Final Report

3

Introduction Overview This final report for the evaluation of the UN Cities and Climate Change eLearning course contains a description of the procedures employed in evaluating the eLearning course, a set of evaluation data, and a set of recommendations related to the course’s implementation. It also includes the sources of data for the study, the evaluation procedures, and a breakdown of the project costs.

Program Description The course is approximately two hours long, and while developed as a self-paced course, it is also used as part of a larger instructor-directed standards-based learning experience. The course’s culminating activity is a knowledge assessment, which provides a UNCC: Learn Completion Certificate for the participant if completed. This module is available as PDF and PowerPoint Presentation for offline study or training purposes.

Program Objectives The objectives of the Cities and Climate Change program are several. First it aims to provide professional development for STEM teachers who teach climate science as a part of the larger environmental science curriculum. Next it aims to provide resources to those teachers to reuse in their own classrooms. Finally, it aims to achieve a public good by being free and available to anyone who wishes to participate.

EDT 506: Evaluation Final Report

4

Program Components UNCC: Learn has designed a series of eLearning courses that provide broad access to Climate Change Learning. When used in conjunction with The Resource Guides for Advanced Learning on Climate Change, and the Library of UN Materials Relevant for Climate Change Learning, the courses can be used in multiple contexts. The use cases include but are not limited to; self-paced training for individuals, professional development for science and climate educators, and as plug-and-play components for existing standards-based, climate change curriculums, i.e., science programs within the K-12 learning space.

Statement of Purpose and Major Questions Answered by the Evaluation The purpose of this learning evaluation is to evaluate the Cities and Climate Change course against its stated goals. A2 attempted to understand the learning outcomes, the effectiveness of the instruction, the student experience, the flexibility of the design and delivery of the course, the technical functionality of the course, and the evaluation of materials that support the course and the instruction. This evaluation has provided answers to the following questions (which were raised when this project was in its proposal phase) What learning outcomes are being achieved by the course?

Participants in this course learn about the reciprocal relationship between cities and climate change. They receive information that cities can be a cause of climate change as well as a solution for climate change when city planners take climate into consideration in the urban planning process.

How effective is the method of instruction?

EDT 506: Evaluation Final Report

5

The course is generally rated by participants as a good course, but not stellar. A2’s main concern with regards to effectiveness is that the course is not be used as it was designed as a reusable object. Further, the course is not an effective tool for professional development, because the STEM teachers did not learn anything new in the module.

What is the student experience?

Most students found the course to be too long, especially given that for STEM teachers there was not a lot of new content here. Some teachers admitted to not finishing the course and not reading all the material.

Is the design of the course flexible enough to be used modularly as intended?

The teachers generally found the content useful, but we found in the focus groups that teachers were not using the course as designed. They found the material too complex for students in younger grades and not engaging enough for the secondary school students. They also felt that for professional development there was nothing new here that would keep the interest of someone who was already well versed in environmental science.

Is the course technically functional?

Students found the user interface to be frustrating when they clicked on links and played embedded videos. They struggled to get back to where they were after opening links or watching a video and had the frustrating experience of getting lost in the course.

How useful are the materials that support the course and the instruction?

Of all the materials that support this course, we found that students found the embedded videos to be the most memorable and the most helpful. Not incidentally, the videos were the most reused content by the STEM teachers who brought components into their classrooms.

EDT 506: Evaluation Final Report

6

Evaluation Methods Participants In addition to data from survey results of learners who complete the UNCC: Learn’s Short Courses, the evaluators collected data through surveys and focus groups.

Based on the target audience of the eLearn Short Course, the evaluation participants were recruited from the learners who have completed the course in the past 12 months and STEM school science teachers who use the plug-and-play components for climate change curriculums.

The survey respondents included 545 individual learners who have completed the course in the past 12 months and 378 STEM school science teachers who teach grades 9-12 in public, private or charter schools who have used the plug-and-play curriculum during the past 12 months.

A2 conducted a focus group with seven STEM teachers who were a subset of the survey respondents. The focus group walked through different areas of the course to gain further insights into the areas of usability, design, and functionality.

EDT 506: Evaluation Final Report

7

Data Sources Detail description of Data Sources: The survey collected data on user performance, user satisfaction, and expert judgments.

We looked at data from the UN on course completion and the summative assessment results for the user performance metrics. A Likert scale survey (see Appendix B) was also designed to understand user satisfaction with the course. We used our expert evaluation in combination with data from focus groups of High School STEM teachers. The evaluation instruments were designed using the Kirkpatrick model combined with theory-based methodologies as outlined in the Evaluation Procedures section.

Using the Kirkpatrick Four-Level Evaluation model to evaluate “reactions, learning, behavior, and results” (Russ-Eft & Preskill, 2009, p. 101), Team A2 developed evaluation instruments in alignment with the description of the four levels to ensure both qualitative and quantitative data was collected: Level I: Reaction to learners’ opinion of the eLearning program.

Level II: Learning measures of the learners’ achievement of the program goals.

Level III: Application measures of how the learners are applying knowledge gained from the program.

Level IV: Results measure whether the program has achieved the impact of the organization’s goals.

EDT 506: Evaluation Final Report

8

Evaluation Procedures The evaluation procedures concentrated on the sources identified in the Data Sources section. Team A2 conducted a document analysis and developed the evaluation instruments in cooperation with the UN CC: Learn team and identified stakeholders. The Team A2 project manager and trained staff members collected data from participants via survey and focus groups.

The senior evaluator conducted a review with staff and stakeholders of data collected and supervised the development of the final report.

A2 developed evaluation instruments based on achieving the desired outcome of assessing the impact of the UN CC: Learn Cities and Climate Change eLearning Short Course in the following areas: learning outcomes; effectiveness of instruction; student/teacher experience, the flexibility of design and delivery of the course, technical functionality of the course and evaluation of the materials that support the course and the instruction.

We apply the logic model framework developed by W.K. Kellogg Foundation for every evaluation project. “The logic model approach helps create a shared understanding of and focus on program goals and methodology, relating activities to projected outcomes” (W.K. Foundation, 2004).

EDT 506: Evaluation Final Report

9

Figure 1 Visualization of the Logic Model

The framework for program evaluation adapted from the CDC framework informs the standards Team A2 applies to program evaluation. To ensure the highest standards, we use theory-based methods to reduce bias and “establish a common understanding of how a program is presumed to solve social problems” (Russ-Eft & Preskill, 2009, p. 84). One established theorybased method we employ is appreciative inquiry (Cooperrider, Srivasta, & Fry, 1990).

“Appreciative Inquiry may: a) increase the richness of the data collected, b) help the evaluator obtain important contextual and stakeholder information, and c) increase the efficiency of the data collection process, d) increase participants level of trust and participation in the evaluation, and e) respect the diversity of participants’ experiences

EDT 506: Evaluation Final Report

10

and opinions” (Russ-Eft & Preskill, 2009, p. 86).

Additional theory-based methodologies used for evaluation include Empowerment evaluation (Fetterman, 1994) (p. 80), Participatory/Collaborative evaluation (Cousins & Earl, 1992) (p. 79), and Organization learning theories “linking evaluation to the concept of organizational learning” (Russ-Eft & Preskill, 2009, p. 82).

Figure 2 Visualization of the Framework for Program Evaluation

Inventory Information (Resources/Inputs): Data provided by the UN CC: Learn team on learners who have completed the course in the past 12 months, results from the certification quiz, and data to identify high school STEM teachers who are using the plug and play program in their classrooms. Team A2 evaluators served as experts and conducted an evaluation of the eLearning based on their expertise in online, blended, and eLearning course development. Data collected was from the post-course survey for participants who completed the course or taught

EDT 506: Evaluation Final Report

11

the course in the past 12 months. Meetings with stakeholders to gather information on design approaches of the team on course design, technical functionality, and materials, provided critical information to avoid assumptions in the instrument development. Discussions with stakeholders helped determine if the program is accomplishing the objectives, identifying strengths and weaknesses, and identifying who may benefit the most or the least from the program.

Define

Evaluation (Activities): Using the inventory information, Team A2 defined the evaluation criteria based on theory-based evaluation methods as referenced above and the data inputs received from the UN CC: Learn team and other identified stakeholders. The result of the inventory activities create, validate, and finalize the evaluation instruments as follows: 1) survey designed for general participants who have completed the learning with the past 12 months; 2) STEM school science teachers who have used the plug and play components of the learning within the past 12 months and 3) planned activities for observational evaluation and feedback from four focus groups.

Gather Credible Evidence (Short/Medium term outcomes and Impact) Information provided by UN CC: Learn team on previous participants were used to send the surveys via email with an expected response rate of 15%. Participants will receive an initial email invitation, a follow-up email one week later, and a final email ten days after the initial email invitation. As survey results are received, they were analyzed to begin the process of building the data models. The staff of Team A2, who are trained and certified in evaluation methods from the American Evaluation Association, collected and reviewed the data. Staff and Team A2 analyzed trends in short term outcomes (changes in knowledge, attitudes, awareness)

EDT 506: Evaluation Final Report

12

reflecting immediate outcomes; medium term outcomes (changes in knowledge, attitudes, awareness) reflecting longer term outcomes of six months or more; and impacts (changes in organizations, communities, or systems) reflecting changes that happen at a systems and organizational level.

Analyze evidence (conclusion) Team A2 under the guidance of the stakeholders, UN CC: Learn team, lead evaluator and project manager analyzed the evidence to provide feedback on learning outcomes, effectiveness or instruction, learner experience, the flexibility of design, and delivery of the course, technical functionality of the course, and evaluation of materials. Team A2 will provide visual models and graphics to illustrate key findings to internal and external non-research audiences. The report will be delivered directly to key stakeholders before delivery to the broader stakeholder audience and has been made available in digital format before the scheduled project results meeting to ensure Team A2 provides results that meet the needs of the audience.

EDT 506: Evaluation Final Report

13

Results Demographics The data analysis of the Cities and Climate Change course survey showed 923 participants completed the survey. Participant were almost evenly split by gender with 448 female participants (48.5%) and 475 male participants (51.5%). The mean age range of participants is 43 years old, with the minimum participant age of 25 years old and the maximum participant age of 62 years old.

Pa r ti c i p a nt s - G e n d e r Fema l e

Mal e

48.54%

51.46%

Pa r ti c i p a nt s - A ge Mi n of a ge

25

Max of a ge

62

Avera ge of age

45 0

EDT 506: Evaluation Final Report

10

20

30

40

50

60

70

14

The survey was completed by participants across all 50 states in the United States, as well as the District of Columbia. The highest number of survey participants were from Texas (13.2%), California (10.6%), and Florida (7.5%). Given that the population of these states is the highest in the United States, it is reasonable to expect a higher number of survey participants. In addition, survey participants are likely to be college level teachers or learners and Texas (1.64 M), California (2.7 M) and Florida (1.07 M) have the highest population of students in the United States (Miller, 2021).

Re s p o n d e nt s - To p T h re e S tate s

33.91% 42.21%

Cal i forni a Fl ori da Texa s

23.88%

The focus group participants demographics were similar to the gender/age of the participants in the survey; however, they were more geographically diverse as the target group for focus groups was STEM teachers in K-12 and higher education.

EDT 506: Evaluation Final Report

15

Overall Analysis The overall analysis of the Cities and Climate change course focused on the areas identified in the RFP: learning outcomes, effectiveness of instruction, student experience, flexibility of design and delivery of the course, technical functionality of the course, and evaluation of the materials that support the course and the instruction. The analysis provides evaluation of the course based on the results from the evaluation instruments, archival data and professional evaluation conducted by Team A2.

Learning outcomes Survey respondents Survey respondent evaluated the learning outcomes of the course related to their expectations of the course and their ability to understand and describe the information in the course related to the course objectives. Respondents evaluated the learning outcome questions on a Likert scale of 1-5 with the descriptors: 1-Strongly Disagree; 2-Disagree; 3-Neither Agree/Disagree (Neutral); 4-Agree; 5-Strongly Agree. Survey questions 4-9 were designed to evaluate learning outcomes (see Appendix B) which revealed learners ratings skewed left with and average mean of 3.80 (Figure 1) and an average standard deviation of 0.735 showing a positive deviation to the right of the mean. No respondents chose “strongly disagree” or “disagree” when evaluating the course learning outcomes. Participants evaluated the course favorably with 60.2% responding “agree”, “strongly agree”, “excellent”, or “very good”.

Approximately 39.8% of respondents provided a neutral response when evaluating the course

EDT 506: Evaluation Final Report

16

learning outcomes indicating a recommendation to review course learning outcomes (see Discussion).

Learning Outcomes After completing the course, how would you rate your ability to identify specific adaptation and mitigation measures suitable for urban areas?

3.82

After completing the course, how would you rate your ability to explain how climate change adaptation and mitigation can be considered urban planning?

3.78

After completing the course, how would you rate your ability to explain how urban areas are affected by, and contribute to, climate change?

3.80

After completing the course, how would you rate your ability to describe the transformative role that cities play in achieving green and climate resilient development?

3.76

Afte r completing the course, how would you rate your ability to describe urban trends for the 21st century?

3.81

The Cities and Climate change course enhanced my knowledge the subject matter.

3.79

0.00 1.00 2.00 3.00 4.00 5.00

Figure 1: Learning Outcomes

Focus group respondents Focus group respondents provided qualitative data on learning outcomes of the Cities and Climate Change course. Although respondents felt similarly aligned with the learning outcomes of the survey respondents, they provided context that was not captured by the survey.

Respondents felt that the course provided an overwhelming amount of information for learners.

The information provided was rudimentary for the teachers as it did not provide information that was unknown to STEM teachers. Concurrently, the information in the course was too high level for most K-12 students. Respondents stated they could use information for teaching STEM

EDT 506: Evaluation Final Report

17

courses, but it was not appropriate for students in grades K-6. The agreed the most benefit could be derived from using the materials in Earth Science courses for 7-12 grade students and higher education environments. The respondents felt the course could be more useful if the contents were broken out into short videos or exercises rather than long sections of content. The focus group was able to respond to questions on the stated learning objectives with examples, demonstrating the objectives are aligned with the course effectively.

Professional evaluation A2 professional evaluation is aligned with the information provided in the archival data showing an average quiz score of 6 correct answers out of 7 and an average grade of 8.23 out of 10 (84.2%). Additional archival data showed trends supporting the results from the focus group and survey respondents with learners completing a survey designed by the UN CC: Learn team.

Results from the survey data provided from the UN CC: Learn team showed evaluations skewing left with a mean of 3.82 and a standard deviation of 0.741. Evaluation of learning outcomes parallels the archival, focus group, and survey data which reveals a strong correlation between the course and achieved learning outcomes. The considerable amount of information across the learning modules impacts the efficacy of the learning outcomes for course participants which correlates to the average learning outcome of 3.8 across data sources.

EDT 506: Evaluation Final Report

18

Effectiveness of Instruction Survey respondents Survey respondent evaluated the effectiveness of the course instruction related to their expectations of the course, their satisfaction with the course, the amount of information contained in the course, and the effect of their level of knowledge post course. Respondents evaluated the effectiveness of instructions via survey questions on a Likert scale of 1-5 with the descriptors: 1-Strongly Disagree; 2-Disagree; 3-Neither Agree/Disagree (Neutral); 4-Agree; 5Strongly Agree. Survey questions 1, 3, 4, 10, and 12 were designed to evaluate effectiveness of instruction (see Appendix B) which revealed learners ratings skewed left with and average mean of 3.81 (Figure 2) and an average standard deviation of 0.743 showing a positive deviation to the right of the mean. No respondents chose “strongly disagree” or “disagree” when evaluating the effectiveness of course instruction. Participants evaluated the course favorably with 60.5% responding “agree”, “strongly agree”, “excellent”, or “very good”. Approximately 39.5% of respondents provided a neutral response when evaluating the effectiveness of course instruction indicating a recommendation to review the structure and amount of information provided (see Discussion).

EDT 506: Evaluation Final Report

19

Effecti veness of Instructi on I would recommend this course to others

3.80

I am satisfied wi th the course overall.

3.83

The Cities and Climate change course enhanced my knowledge of the subject matter.

3.79

The course contained the appropriate amount of information to achieve the stated objectives.

3.78

The Cities and Cli mate Change course met my expectations

3.80

0.00

1.00

2.00

3.00

4.00

5.00

Figure 2: Effectiveness of Instruction

Focus group respondents Focus group respondents provided qualitative data on effectiveness of instruction of the Cities and Climate Change course. Although respondents felt similarly aligned with the effectiveness of instruction evaluation of the survey respondents, they provided context that was not captured by the survey. Respondents felt that the course provided an overwhelming amount of information for learners. The information provided in the course felt “overwhelming” and “more informational than directive”, causing STEM teachers to feel that some of the information was useful for Earth Science instruction but not within the current structure. Focus group respondents stated that some of the contents of the course were confusing to navigate impacting the effectiveness of instruction. Examples included portions of the course where there were several interactive elements (see Figure 3) which provided links to external material.

Respondents noted that some of the links did not open properly which impacted the effectiveness

EDT 506: Evaluation Final Report

20

of the instruction. Taking the learner out of the course causes task switching which does not have a positive effect on learning outcomes. “A 2018 study by Kaitlyn May and Anastasia Elder examined 38 articles from 2003 to 2017 that researched the effect of task switching on learning.

They found that trying to do too many things at the same time resulted in less effective learning, lower exam scores, and decreased study efficiency” (Lumen Learning, 2021). Focus group respondents felt the interactive elements often did not result in effective instruction within the course.

Figure 3: Interactive Element

EDT 506: Evaluation Final Report

21

Professional evaluation A2 professional evaluation is aligned with the information provided in the archival data showing learners comments reflected similar responses as the focus group respondents provided.

Additional archival data a drop off in interaction with elements as shown in Figure 3. A drop-off of time spent on some of the more complex interactive elements, particularly if the element required learners to leave the course, resulted in a 52% decline from other course elements.

Sections 4-6 show a drop off rate for learners of 33% from sections 1-3. Sections 4-6 contain a large portion of the interactive elements which is resulting in a decrease in effectiveness of instruction.

Student Experience, Flexibility of Design and Delivery of the Course, Technical Functionality Survey respondents Survey respondent evaluated the student experience related to flexibility of design, functionality, accessibility, course satisfaction, and course information. Respondents evaluated the student experience, design and delivery of the course via survey questions on a Likert scale of 1-5 with the descriptors: 1-Strongly Disagree; 2-Disagree; 3-Neither Agree/Disagree (Neutral); 4-Agree; 5-Strongly Agree. Survey questions 2, 3, 10, and 11 were designed to evaluate student experience (see Appendix B) which revealed learners ratings skewed left with and average mean of 3.82 (Figure 4) and an average standard deviation of 0.752 showing a positive deviation to the right of the mean. No respondents chose “strongly disagree” or “disagree” when evaluating the effectiveness of course instruction. Participants evaluated the

EDT 506: Evaluation Final Report

22

course favorably with 60.2% responding “agree”, “strongly agree”, “excellent”, or “very good”.

Approximately 39.8% of respondents provided a neutral response when evaluating the student experience, flexibility of design, and delivery of the course indicating a recommendation to review the navigation elements and amount of information provided (see Discussion).

Student Experience/Course Design The curriculum had accessibility features built into the program to assist viewers who may need it.

3.79

I found the course materials easy to navigate.

3.81

I am satisfied with the course overall.

3.83

The course contained the appropriate amount of information to achieve the stated objectives.

3.78

0.00

1.00

2.00

3.00

4.00

5.00

Figure 4: Student Experience/Course Design

Focus group respondents Focus group respondents provided qualitative data on the student experience of the Cities and Climate Change course. Although respondents felt similarly aligned with the student experience of the survey respondents, they provided context that was not captured by the survey.

The focus on student experience including navigation of the course, satisfaction of the course overall, and the amount of information provided in the course. Respondents felt that the course provided an overwhelming amount of information for learners which was reflected in archive

EDT 506: Evaluation Final Report

23

and survey data. Focus group respondents stated that some of the contents of the course were challenging to navigate like access the menu which appears at the top right corner of the course window. The menu font size is approximately 7.5-8-point size. The size of the font, placement of the menu and the font color contrast against the background made it difficult for students to locate (Figure 5, pink arrow). Respondents noted that there were no features within the course to expand the menu for visibility. In addition, although students could navigate the course with the tab key on their keyboards rather than using the mouse, the menu link was not included as accessible when using the tab key. Navigational elements that focus group respondents agreed were helpful are shown in Figure 5 (red arrows) and include slide count, information option, arrow navigations, play/pause feature for audio, and volume control. In addition, focus group respondents stated the downloadable materials including the syllabus and PowerPoint presentations were supportive of varied learning modalities.

EDT 506: Evaluation Final Report

24

Figure 5: Navigation Elements

Professional evaluation A2 professional evaluation is aligned with the information provided in the archival, survey and focus group data. The student experience is impacted by navigational elements that are inaccessible without a mouse (menu), general information without clear guidance on what students should/could do to engage in practices supportive of addressing climate change. The information is overwhelming general (lacking novelty) which does not address learner’s connection to real-world issues that are relevant to their ability to act. Learners may find it difficult to remember the key objectives of the course due to the structure of the course which focuses on global issues without clear connections with students lived experience.

EDT 506: Evaluation Final Report

25

Evaluation of the materials that support the course and the instruction Survey respondents Survey respondent evaluated the materials that support the course and the instruction.

The accompanying materials included a PowerPoint presentation, PDF of course, linked material within the course and the course quiz. Respondents evaluated the course materials via survey questions on a Likert scale of 1-5 with the descriptors: 1-Strongly Disagree; 2-Disagree; 3Neither Agree/Disagree (Neutral); 4-Agree; 5-Strongly Agree. Survey questions 3, 11, and 12 were designed to evaluate student experience (see Appendix B) which revealed learners ratings skewed left with and average mean of 3.80 (Figure 6) and an average standard deviation of 0.748 showing a positive deviation to the right of the mean. No respondents chose “strongly disagree” or “disagree” when evaluating the effectiveness of course instruction. Participants evaluated the course favorably with 59.7% responding “agree”, “strongly agree”, “excellent”, or “very good”.

Approximately 40.3% of respondents provided a neutral response when evaluating the supporting course materials, indicating a recommendation to review and revise the supporting materials (see Discussion).

EDT 506: Evaluation Final Report

26

Evaluati on of Materials I would recommend this course to others

3.80

I found the course materials easy to navi gate.

3.81

The course contained the appropriate amount of information to achieve the stated objectives.

3.78

0.00

1.00

2.00

3.00

4.00

5.00

Figure 6: Evaluation of Materials

Focus group respondents Focus group respondents provided qualitative data on the supporting materials provided for the Cities and Climate Change course. Although respondents felt similarly aligned with the student experience of the survey respondents, they provided context that was not captured by the survey. The focus on supporting materials including PowerPoint presentation and PDF version of the course, linked content within the course and the course quiz. Focus group respondents preferred the PowerPoint and PDF materials over the full course. The downloadable content allowed for STEM teachers to choose content for lessons rather than using the sections as whole units. More than 75% of the focus group respondents stated that they did not complete the course but used the supporting materials to gather the information they needed. The most significant negative response regarding supporting materials related to the links within the

EDT 506: Evaluation Final Report

27

course. The course contains 76 slides with 79 links to content external to the course. The focus group respondents felt this was incongruent with eLearning design and created a distraction for the learner. The links provided within the PowerPoint and PDF material contain the links but serve as optional or supporting information (Figure 7). The abundance of externally linked material impacted the focus group respondent’s response on whether they would recommend this course to others. The focus group agreed (100%) the quiz provided at the end of the course materials was well-designed and effective. Specifically, respondents thought the ability to review quiz attempts (questions and answers) as well as allowing for learners to make multiple attempts to achieve a passing score was an effective use of supporting materials.

Figure 7: Example PPT version

EDT 506: Evaluation Final Report

28

Professional evaluation A2 professional evaluation is aligned with the information provided in the archival, survey and focus group data. The supporting material provided with the course delivers options for learners and teachers to use the materials effectively. The inclusion of these materials increases access to the information for learners with different abilities or learning needs. The links within the course should be evaluated for efficacy due to the sheer number and placement of links. Review of the data reveals no sustainability plan for maintenance of the links leading to broken, missing, or irrelevant external information. The evaluation team concurs with the focus group respondents regarding the post course quiz. The quiz is relevant, accessible, provides relevant questions, opportunity to review the questions and option to retake (up to 3 times) the quiz. To increase the effectiveness of the quiz, the evaluation team recommends adding connections to the relevant course content section for review providing learners the opportunity to revisit the information within the course before attempting to take the quiz again.

Discussion The results from all data sources provides a clear picture of the perspective of learners and teachers of the Cities and Climate Change course. What emerged was a positive correlation between the learners/teachers and the course design elements, content, navigation and learning objectives. Overall, the survey respondents provided a 3.81 mean with a 0.739 standard deviation demonstrating a 76% average positive response to all survey questions overall. Data from the survey was highly consistent with no respondents selecting “disagree” or “strongly disagree” to the twelve survey questions. Focus group respondents provided more specific and

EDT 506: Evaluation Final Report

29

qualitative data which aligned with the survey data consistently. However, the focus group respondents provided specific feedback that correlated to the shifts in response on the survey on questions 3, 6, and 7 (see Appendix C). These questions relate directly to the information provided in the course and the learning outcomes. The data reflected a slight increase in neutral positive and neutral negative responses on average of 41.5%. Survey responses combined with focus group responses pointed to a single area of concern regarding the amount of information provided in each course section. This trend remained relevant through the professional evaluation by A2 evaluators when examining the archival data. Feedback from course surveys, course quiz, and data regarding user interactivity rates/timing reflect the underlying issue preventing learners from full engagement with the course materials which results in decrease in course completion rates.

Program components reflected as positively received by learners/teachers focused on the supporting materials and how both learners and teachers were able to engage with the course more effectively using the available PowerPoint and PDF. This reflects a negative impression of the eLearning course contents as being “overwhelming” but also “too general”. The data and evaluation provide an important revelation about the current structure of the eLearning. Focus group respondents emphasized the materials as being relevant for 7-12, higher education and professional development use exclusively. It is recommended the UN CC: Learn team revisit the recommended target audience for the eLearning course based on feedback. Respondents and archival data provided a positive response to the course navigation elements with exception of the menu feature. It is recommended the UNCC: Learn team to revise the course to align with

EDT 506: Evaluation Final Report

30

Web Content Accessibility Guidelines (WCAG) to ensure learners are able to navigate every course element effectively.

Evaluation Recommendations Team A2 evaluators recommendations based on survey, focus group, and archival data are as follows.

Learning outcomes Overall, the learning outcomes are clearly defined and reinforced throughout the course materials. The course, however, feels overwhelming for learners. The data shows a consistent response from learners/teachers related to the course content as being both too broad and too general for the course to be impactful. Learners who had taken the course over six months prior to the survey were more likely to respond as “neutral” to questions about learning outcomes on the survey. Focus group respondents clearly reiterated their feelings of overwhelm while taking the course resulting in a disconnection from the learning material. Recommendation: The UNCC: Learn team may revisit the course to assess how to arrange relevant topics as microlearning’s for their target audience. It is also recommended to add in knowledge checks throughout the course to reinforce knowledge and understanding.

Effectiveness of instruction Respondents were clear (as noted) that the information in the eLearning course was perceived as too broad and too general. The most significant concern relating to effectiveness of instruction was reflected in respondent’s frustration with the number of external links in the

EDT 506: Evaluation Final Report

31

course that interrupted the learning flow. The major concerns relating to external links in the course were links that were broken, links that lead to information where the relevance wasn’t clear, and the number of external links in the course. Recommendation: The UNCC: Learn team may revisit the links contained within the course for relevance, content, and function.

Elimination of unnecessary external information will create a better learner experience and a more effective course. A sustainability model for course links with structured timelines for review and updates will be necessary to ensure satisfactory learner experience.

Student experience, flexibility of design & delivery, technical functionality In addition to the above recommendations, data regarding student experience and course design and functionality reflected consistent feedback regarding navigation of the course.

Students struggled to locate the menu for the course due to the size (7.5-8-point font), location and color contrast. In addition, the interactive portions of the course were often unclear as to which element the student should engage first and the flow of the engagement.

Recommendation: The UNCC: Learn team may revisit the course design elements related to accessibility. The course is structured to use the keyboard for most but not all navigation and there are no accessibility tools to increase the size or change the contrast for learners who are differently abled. Where parts of the course contain multiple interactive elements, it is recommended that the UNCC: Learn team provide clear instruction on how to interact with the elements and the flow of the interaction.

EDT 506: Evaluation Final Report

32

Evaluation of the materials that support the course and the instruction Data derived from respondents and archival sources provided a positive reflection of the supporting course materials including the PowerPoint, PDF, and quiz. Many respondents felt the supporting materials were superior to the course overall. The materials created a flexibility not available in the eLearning course. The quiz was described as being well-designed and effective for evaluation. Recommendation: The UNCC: Learn team may visit other forms of supporting materials that could enhance the learning and teaching of the course. The team may also consider creating connection in the course material to the quiz results for learners to review if they mark a question incorrectly or want to access the section again. This addition would serve as reinforcement of the learning objectives and outcomes.

Project Cost This section itemizes the cost incurred during the evaluation process and to conduct surveys and focus groups and gather data to understand the effectiveness of learning outcomes, flexibility of design, functionality and delivery of the short e-learning course Cities and Climate Change. These costs represent the development of an evaluation design, key evaluation questions, logic models, the development of evaluation instruments, data collection and analysis, the review of archival and/or supporting program materials, and the preparation and delivery of the final evaluation. The following table details the itemized costs of the evaluation:

EDT 506: Evaluation Final Report

33

Project Budget - Detail - Jun 28th, 2021 Category #1 - PROJECT CONCEPT AND REVIEW PERSONNEL Tina - Lead Evaluator 240 hrs. @$86 Michelle - Evaluator 240 hrs.@ $72 Sonali - Junior Evaluator 240 hrs. @ $64 Project Manager interview & recruitment (including agency fee) Category #1 Total Category 2 - PROJECT DESIGN AND DEVELOPMENT - PROJECT DESIGN Project Planning 40 hrs. ($95) Instrument planning, design & development 24 hrs. ($95) Writing, editing, review & design of Summative and formative questionnaire survey 36hrs ($95) E-mail preparation, review 4 hrs. @ ($95) Email distribution for survey - e-mail chimp costs Focus group recruitment & confirmation 8 hrs. ($95) Focus group management 8 hrs. ($95) - DATA DEVELOPMENT Data collection: system run, extraction and review 18 hrs. @$95 Data Cleaning 16 hrs.@$95 Data Analysis 24 hrs. @$95 Run Reporting 8 hrs. @$95 Category #2 Totals Category 3 - MATERIAL & COMMUNICATION - MATERIAL Thank you, letters & envelopes, with Logo @ $5.00 each x 20 FedEx or mail services Other printing & copying Faxing and scanning, COMMUNICATION Telephone bill charges for staff members 3 persons x $100 Category #3 Total Category 4 - PROJECT PREPARATION & PRESENTATION Final Report development 16($95) Presentation meeting attendance 4 hrs. ($95) Category #4 Total Category 5 - Miscellaneous (unexpected costs) Category

EDT 506: Evaluation Final Report

$20,640.00 $17,280.00 $15,360.00 $ 1,500.00 $54,780.00 $3,800.00 $2,280.00 $3,420.00 $380.00 $45.00 $760.00 $760.00 1,710.00 $1,520.00 $2,280.00 $760.00 $17,715.00

$100.00 $80.00 $60.00 $50.00 $100.00 $390.00 $1,520.00 $380.00 $1,900.00 $215.00

34

#5 Totals Total

$75,000.00 Project Balance Budget Summary - June 28th, 2021

Allocated Budget

$75,000.00

Budget expense categories Category #1

$54,780.00

Category #2

$17,715.00

Category #3

$390.00

Category #4

$1,900.00

Category #5

$215.00

Total Budget

$75,000

Project Budget Detail assumptions for Project Manager are documented as follows: ● Project Manager is paid at the hourly wage of $95.00 ● Project Management hours allocated are total of 206 hours ● Cost of Project Manager recruitment services $1,500 (includes benefits, services, and contact fees) ● Total amount spent for Project Manager is: 206 hrs. @$95 = $19,570 ● Service Fee $1,500 ● Total cost for Project Management services = $21,070 (all costs built into the total overall Project)

EDT 506: Evaluation Final Report

35

Program Evaluator salary assumptions are as follows: The average Program Evaluator salary in the United States according to Salary.com is $159,623 the salary range typically falls between $149,538 and $198,910. ●

Lead Evaluator is paid an annual salary of $178,000 ($86.00 an hour)

●

Evaluator is paid an annual salary of $149,760 ($72.00 an hour)

●

Junior Evaluator is paid an annual salary of $133,120 ($64 an hour)

This budget represents approved cost details, simply the combination of costs; individual people and activities that the project must accomplish. Budget drives project funding information for stakeholders on how much money is needed, when it is needed and more importantly the ability to afford people, equipment, and materials required to complete the project evaluation.

By measuring the project's actual cost against the approved budget, we can determine if the project is progressing according to the plan or if corrective action is needed to manage costs.

EDT 506: Evaluation Final Report

36

References Lumen Learning. (2021). Introduction to Psychology. Lumen. https://courses.lumenlearning.com/waymaker-psychology/chapter/learning-hacks-the-mythof-multitasking/.

Russ-Eft, D. F., & Preskill, H. (2009). Evaluation in organizations: a systematic approach to enhancing learning, performance, and change. Perseus Publishing.

United Nations Institute for Training and Research (UNITAR) https://unccelearn.org/ “Cities and Climate Change.” One UN Climate Change Learning Partnership with , SWISS Agency for Development and Cooperation SDC,unccelearn.org/course In collaboration with UNCC:Learn Partners W.K. Kellogg Foundation. (2004, January 01). Logic model development guide. Retrieved March 24, 2021, from https://www.wkkf.org/resource-directory/resources/2004/01/logicmodel-development-guide

EDT 506: Evaluation Final Report

37

APPENDIX Appendix A: Document Analysis Results

Data source Who or document wrote it and when

What data is available

Peoples’ UNDP Climate Vote and Universit y of Oxford January 2021

Charts, public opinion, survey, statistics

charts, statistics, surveys

EDT 506: Evaluation Final Report

How Describe the data many within the participa document nts were included in the report

Does the data in this document support the evaluation (provides data that can be used to evaluate parts of the program that the team cannot evaluate or can be used to support results of the evaluation your team is proposing) Short answer not just Yes/No

1.2 million

Yes, the data in the document demonstrates specific information on whether people believe climate change is an emergency, which policies that governments should enact and the data is broken out into categories that align with categories in the UN course. The large size of the population provides rich data that can be considered archival records for analysis for our evaluation.

Poll questions were distributed through advertisements in mobile game apps in 17 languages asking people about their belief in the climate change emergency and which policies across six areas that they would like their government to enact.

38

Education for Sustainable Development (ESD): Partners in Action – Global Action Programme (GAP) Key Partners’ Report (2015-2019)

Published in 2020 by the United Nations Education al,

Integrating Action for Climate Empowerme nt (ACE) into Nationally Determined Contribution s: A Short Guide for Countries

Published in 2020 by the UNESCO

Survey results,cha rts, statistics, infographi cs

Scientific and Cultural Organizat ion (UNESC O)

UNESCO worked with A network of 97

The report presents the analysis of the quantitative responses of GAP Key Partners

GAP Key to each of the ten Partners indicators. from governme nt, civil society, the private sector, and academia

This guide aims to provide countries with advice on how to address the six elements of ‘Action for Climate Empower ment’ (ACE) – Education, Training,

EDT 506: Evaluation Final Report

197 Reporting countries as part of the Paris Climate Agreemen t

In five simple questions and answers, it provides suggestions for elements that can be integrated into the NDCs in order to make ACE an integral part of countries’ climate responses.

Yes, it demonstrates what success looks like in integrating a sustainable development component to teacher education.

Further, it shows how the teachers integrated sustainability principles into their classrooms and training settings for more than 37 million learners of all ages. GAP Key Partners supported programs and activities in more than 139,093 schools, training institutions, and other organizations.

This document provides a foundation for understanding that introducing environmental education to young people is a powerful way to change longterm habits and reach large numbers of people. This document provides the “why” for adding Cities and Climate Change to Environmental curricula in whole or part.

39

Country Progress on Climate Change Education, Training and Public Awareness: An Analysis of Country Submissions Under the United Nations Framework Convention on Climate Change

Published in 2020 by UNESCO and the UNFCCC Secretaria t

Status 194 report on Reporting the Countries implement ation of climate change education.

EDT 506: Evaluation Final Report

194 reporting countries as part of the Paris Climate Agreement, submitted nationally determined contributions as part of their commitment to emission reduction and implementation.

Data included the findings of those submissions including climate change education, public awareness, focuses on adaptation and mitigation, country comparisons, and data each country is gathering for their reports.

This document explains why the expansion of climate change education beyond mere awareness and toward formal education and training activities are necessary. This shows why adopting a course like Cities and Climate Change in whole or part moves beyond awareness and into the arena of formal education.

40

Appendix B: Survey Questions

EDT 506: Evaluation Final Report

41

EDT 506: Evaluation Final Report

42

EDT 506: Evaluation Final Report

43

EDT 506: Evaluation Final Report

44

EDT 506: Evaluation Final Report

45

EDT 506: Evaluation Final Report

46

EDT 506: Evaluation Final Report

47

EDT 506: Evaluation Final Report

48

EDT 506: Evaluation Final Report

49

EDT 506: Evaluation Final Report

50

EDT 506: Evaluation Final Report

51

Appendix C Data Analysis, Cities and Climate Change Survey Demographics The data analysis of the Cities and Climate Change course survey showed 923 participants completed the survey. Participant were almost evenly split by gender with 448 female participants (48.5%) and 475 male participants (51.5%). The mean age range of participants is 43 years old, with the minimum participant age of 25 years old and the maximum participant age of 62 years old.

Frequency Tables Frequencies for gender gender Frequency Percent Valid Percent Cumulative Percent F

448

48.537

48.537

48.537

M

475

51.463

51.463

100.000

0

0.000

Missin g Total

923 100.000

Descriptive Statistics age Valid Missing Mean

EDT 506: Evaluation Final Report

923 0 43.078

52

Descriptive Statistics age Std. Deviation

10.663

Minimum

25.000

Maximum

62.000

Frequencies for university university Frequency Percent Valid Percent Cumulative Percent learner

545

59.047

59.047

59.047

teacher

378

40.953

40.953

100.000

0

0.000

Missing Total

923 100.000

The survey was completed by participants across all 50 states in the United States, as well as the District of Columbia. The highest number of survey participants were from Texas (13.2%), California (10.6%), and Florida (7.5%). Given that the population of these states is the highest in the United States, it is reasonable to expect a higher number of survey participants. In addition, survey participants are likely to be college level teachers or learners and Texas (1.64 M), California (2.7 M) and Florida (1.07 M) have the highest population of students in the United States (Miller, 2021).

EDT 506: Evaluation Final Report

53

Survey Results Question 1 - The Cities and Climate Change course met my expectations.

The central tendency of responses for question 1 skewed left with a mean of 3.84 and standard deviation of 0.735 showing a positive deviation to the right of the mean (Figure 1). No participants chose “strongly disagree” or “disagree” when responding. Most respondents (61.3%) chose “agree” or “strongly agree” the course met their expectations, with 38.7% indicating they neither agreed nor disagreed with the question. The responses indicated a positive response to the question overall indicating the course met the expectation of the learner based on the description and objectives provided prior to taking the course. Although the result is favorable, the description and objectives should be revised to improve future survey outcomes.

Qualitative data will be gathered from the focus groups to establish key factors to improve the correlation between learners’ expectations and the course.

Frequencies for The Cities and Climate Change course met my expectations The Cities and Climate Change course met my expectations

Frequency

Percent

Valid Percent

Cumulative Percent

3

357

38.678

38.678

38.678

4

390

42.254

42.254

80.932

5

176

19.068

19.068

100.000

0

0.000

923

100.000

Missing Total

EDT 506: Evaluation Final Report

54

Figure 8- Frequency Chart Question 2 – The Curriculum had accessibility features built into the program to assist viewers who may need it.

The central tendency of responses for question 2 skewed left with a mean of 3.79 and standard deviation of 0.740 showing a positive deviation to the right of the mean. No participants chose “strongly disagree” or “disagree” when responding. Most respondents (60.2%) chose “agree” or “strongly agree” the curriculum had accessibility features built into the program to assist viewers who may need it, with 39.8% indicating they neither agreed nor disagreed with the question. The responses indicated a positive response to the question overall indicating the course had accessibility features built into the program to assist viewers who may need it.

Accessibility is a key consideration to ensure the course is accessible to everyone. With nearly 40% of respondents choosing neither agree or disagree, the accessibility features should be reviewed and usability testing with learners off differing abilities should be conducted to assess areas of improvement.

EDT 506: Evaluation Final Report

55

Frequencies for the curriculum had accessibility features built into the program to assist viewers who may need it.

The curriculum had accessibility features built into the program to assist viewers who may need it.

Frequency

Percent

Valid Percent

Cumulative Percent

3

367

39.762

39.762

39.762

4

379

41.062

41.062

80.823

5

177

19.177

19.177

100.000

0

0.000

923

100.000

Missing Total

Question 3 – The course contained the appropriate amount of information to achieve the stated objectives.

The central tendency of responses for question 3 skewed left with a mean of 3.78 and standard deviation of 0.752 showing a positive deviation to the right of the mean. No participants chose “strongly disagree” or “disagree” when responding. Most respondents (58.2%) chose “agree” or “strongly agree” the course contained the appropriate amount of information to achieve the stated objectives, with 41.8% indicating they neither agreed nor disagreed with the question. The responses indicated a slightly positive response to the question overall indicating the course had contained the appropriate amount of information to achieve the stated objectives.

However, the response with the highest frequency was a neutral (do not agree or disagree) which demonstrates that either the course contained extraneous information that did not satisfy the stated objectives or a lack of information needed to achieve the objectives. With over 40% of respondents choosing neither agree nor disagree, it is necessary to reexamine the content of the

EDT 506: Evaluation Final Report

56

course to the objectives to ensure participants receive the information needed to achieve the goals of the course.

Frequencies for the course contained the appropriate amount of information to achieve the stated objectives.

The course contained the appropriate amount of information to achieve the Frequency stated objectives.

Percent

Valid Percent

Cumulative Percent

3

386

41.820

41.820

41.820

4

356

38.570

38.570

80.390

5

181

19.610

19.610

100.000

0

0.000

923

100.000

Missing Total

Question 4 – The Cities and Climate Change course enhanced my knowledge of the subject matter.

The central tendency of responses for question 4 skewed left with a mean of 3.79 and standard deviation of 0.736 showing a positive deviation to the right of the mean (Figure 2). No participants chose “strongly disagree” or “disagree” when responding. Most respondents (60.5%) chose “agree” or “strongly agree” the course enhanced my knowledge of the subject matter, with 39.5% indicating they neither agreed nor disagreed with the question. The responses indicated a positive response to the question overall indicating the course enhanced knowledge of the subject matter. With nearly 40% of respondents choosing neither agree nor disagree, a review of the course information is essential. Qualitative data gathered from the focus groups will provide deeper insight into the survey responses to answer the question of

EDT 506: Evaluation Final Report

57

whether the information in the course is presenting information that is already well-known to the learners.

Frequencies for The Cities and Climate change course enhanced my knowledge of the subject matter.

The Cities and Climate change course enhanced my knowledge of Frequency the subject matter.

Percent

Valid Percent

Cumulative Percent

3

365

39.545

39.545

39.545

4

384

41.603

41.603

81.148

5

174

18.852

18.852

100.000

0

0.000

923

100.000

Missing Total

Figure 9 - Frequency Chart

EDT 506: Evaluation Final Report

58

Question 5 – After completing the course, how would you rate your ability to describe urban trends for the 21st century?

The central tendency of responses for question 5 skewed left with a mean of 3.81 and standard deviation of 0.753 showing a positive deviation to the right of the mean. No participants chose “poor” or “fair” when responding. Most respondents (60.5%) chose “very good” or “excellent” after completing the course, with 39.5% choosing “good”. The responses indicated a positive response to the question overall indicating the learners were able to describe urban trends for the 21st century. Respondents positive self-rating of their ability to describe urban trends for the 21st century demonstrates the course provided relevant information that directly met one of the course objectives. This data provides insight for Question 3 (The course contained the appropriate amount of information to achieve the stated objectives), where 41.8% of the respondents provided a neutral response despite the majority of respondents rating their ability to describe the information gleaned from the course as good, very good, or excellent.

EDT 506: Evaluation Final Report

59

Frequencies for After completing the course, how would you rate your ability to describe urban trends for the 21st century?

After completing the course, how would you rate your ability to Frequency describe urban trends for the 21st century?

Percent

Valid Percent

Cumulative Percent

3

365

39.545

39.545

39.545

4

367

39.762

39.762

79.307

5

191

20.693

20.693

100.000

0

0.000

923

100.000

Missing Total

Question 6 – After completing the course, how would you rate your ability to describe the transformative role that cities play in achieving green and climate resilient development?

The central tendency of responses for question 6 skewed left with a mean of 3.76 and standard deviation of 0.724 showing a positive deviation to the right of the mean. No participants chose “poor” or “fair” when responding. Most respondents (58.7%) chose “very good” or “excellent” after completing the course, with 41.3% choosing “good”. The responses indicated a positive response to the question overall indicating the learners were able to describe the transformative role that cities play in achieving green and climate resilient development.

Respondents positive self-rating of their ability to describe the transformative role that cities play in green and climate resilient development demonstrates the course provided relevant information that directly met one of the course objectives. This data provides insight for Question 3 (The course contained the appropriate amount of information to achieve the stated

EDT 506: Evaluation Final Report

60

objectives), where 41.8% of the respondents provided a neutral response despite the majority of respondents rating their ability to describe the information gleaned from the course as good, very good, or excellent.

Frequencies for After completing the course, how would you rate your ability to describe the transformative role that cities play in achieving green and climate resilient development?

After completing the course, how would you rate your ability to describe the transformative role that Frequency cities play in achieving green and climate resilient development?

Percent

Valid Percent

Cumulative Percent

3

381

41.278

41.278

41.278

4

385

41.712

41.712

82.990

5

157

17.010

17.010

100.000

0

0.000

923

100.000

Missing Total

EDT 506: Evaluation Final Report

61

Question 7 – After completing the course, how would you rate your ability to explain how urban areas are affected by, and contribute to, climate change?

The central tendency of responses for question 7 skewed left with a mean of 3.81 and standard deviation of 0.727 showing a positive deviation to the right of the mean (Figure 3). No participants chose “poor” or “fair” when responding. Most respondents (58.7%) chose “very good” or “excellent” after completing the course, with 41.3% choosing “good”. The responses indicated a positive response to the question overall indicating the learners were able to explain how urban areas are affected by, and contribute to, climate change. Respondents positive selfrating of their ability to explain how urban areas are affected and contribute to climate change demonstrates the course provided relevant information that directly met one of the course objectives. This data provides insight for Question 3 (The course contained the appropriate amount of information to achieve the stated objectives), where 41.8% of the respondents provided a neutral response despite the majority of respondents rating their ability to describe the information gleaned from the course as good, very good, or excellent.

EDT 506: Evaluation Final Report

62

Frequencies for After completing the course, how would you rate your ability to explain how urban areas are affected by, and contribute to, climate change?

After completing the course, how would you rate your ability to explain how urban areas are Frequency affected by, and contribute to, climate change?

Percent

Valid Percent

Cumulative Percent

3

351

38.028

38.028

38.028

4

401

43.445

43.445

81.473

5

171

18.527

18.527

100.000

0

0.000

923

100.000

Missing Total

Figure 10 - Frequency Chart

EDT 506: Evaluation Final Report

63

Question 8 – After completing the course, how would you rate your ability to explain how climate change adaptation and mitigation can be considered urban planning?

The central tendency of responses for question 8 skewed left with a mean of 3.78 and standard deviation of 0.727 showing a positive deviation to the right of the mean. No participants chose “poor” or “fair” when responding. Most respondents (60.4%) chose “very good” or “excellent” after completing the course, with 39.6% choosing “good”. The responses indicated a positive response to the question overall indicating the learners were able to explain how climate change adaptation and mitigation can be considered urban planning. Respondents positive selfrating of their ability to explain how climate change adaptation and mitigation can be considered urban planning demonstrates the course provided relevant information that directly met one of the course objectives. This data provides insight for Question 3 (The course contained the appropriate amount of information to achieve the stated objectives), where 41.8% of the respondents provided a neutral response despite the majority of respondents rating their ability to describe the information gleaned from the course as good, very good, or excellent.

Frequencies for After completing the course, how would you rate your ability to explain how climate change adaptation and mitigation can be considered urban planning?

After completing the course, how would you rate your ability to explain how climate change Frequency adaptation and mitigation can be considered urban planning?

Percent

Valid Percent

Cumulative Percent

3

366

39.653

39.653

39.653

4

392

42.470

42.470

82.124

5

165

17.876

17.876

100.000

0

0.000

Missing

EDT 506: Evaluation Final Report

64

Frequencies for After completing the course, how would you rate your ability to explain how climate change adaptation and mitigation can be considered urban planning?

After completing the course, how would you rate your ability to explain how climate change Frequency adaptation and mitigation can be considered urban planning?

Total

923

Percent

Valid Percent

Cumulative Percent

100.000

Question 9 – After completing the course, how would you rate your ability to identify specific adaptation and mitigation measures suitable for urban areas?

The central tendency of responses for question 9 skewed left with a mean of 3.82 and standard deviation of 0.741 showing a positive deviation to the right of the mean (Figure 4). No participants chose “poor” or “fair” when responding. Most respondents (62.1%) chose “very good” or “excellent” after completing the course, with 37.9% choosing “good”. The responses indicated a positive response to the question overall indicating the learners were able to identify specific adaptation and mitigation measures suitable for urban areas. Respondents positive selfrating of their ability to identify specific adaptation and mitigation measures suitable for urban areas demonstrates the course provided relevant information that directly met one of the course objectives. This data provides insight for Question 3 (The course contained the appropriate amount of information to achieve the stated objectives), where 41.8% of the respondents provided a neutral response despite the majority of respondents rating their ability to describe the information gleaned from the course as good, very good, or excellent.

EDT 506: Evaluation Final Report

65

Frequencies for After completing the course, how would you rate your ability to identify specific adaptation and mitigation measures suitable for urban areas?

After completing the course, how would you rate your ability to identify specific adaptation and Frequency mitigation measures suitable for urban areas?

Percent

Valid Percent

Cumulative Percent

3

350

37.920

37.920

37.920

4

387

41.928

41.928

79.848

5

186

20.152

20.152

100.000

0

0.000

923

100.000

Missing Total

Figure 11 - Frequency Chart

EDT 506: Evaluation Final Report

66

Question 10 – I am satisfied with the course overall The central tendency of responses for question 10 skewed left with a mean of 3.83 and standard deviation of 0.761 showing a positive deviation to the right of the mean. No participants chose “strongly disagree” or “disagree” when responding. Most respondents (61.2%) chose “agree” or “strongly agree” after completing the course, with 38.8% indicating they neither agreed nor disagreed with the question. The responses indicated a positive response to the question overall indicating the course was satisfactory to most respondents. Although the result is favorable, the focus group data may reveal more information about learner satisfaction.

Qualitative data will be gathered from the focus groups to establish key factors to improve learners’ satisfaction with the course.

Frequencies for I am satisfied with the course overall.

I am satisfied with the course overall.

Frequency Percent

Valid Percent

Cumulative Percent

3

358

38.787

38.787

38.787

4

363

39.328

39.328

78.115

5

202

21.885

21.885

100.000

0

0.000

923

100.000

Missing Total

EDT 506: Evaluation Final Report

67

Question 11 – I found the course materials easy to navigate The central tendency of responses for question 11 skewed left with a mean of 3.81 and standard deviation of 0.758 showing a positive deviation to the right of the mean. No participants chose “strongly disagree” or “disagree” when responding. Most respondents (59.9%) chose “agree” or “strongly agree” after completing the course, with 40.1% indicating they neither agreed nor disagreed with the question. The responses indicated a positive response to the question overall indicating the course materials were easy to navigate. Although the result is favorable, the focus group data may reveal more information about course navigation.

Qualitative data will be gathered from the focus groups to establish key factors to understand how course materials may be improved for easier navigation.

Frequencies for I found the course materials easy to navigate.

I found the course materials easy to Frequency navigate.

Percent

Valid Percent

Cumulative Percent

3

370

40.087

40.087

40.087

4

360

39.003

39.003

79.090

5

193

20.910

20.910

100.000

0

0.000

923

100.000

Missing Total

EDT 506: Evaluation Final Report

68

Question 12 – I would recommend this course to others The central tendency of responses for question 12 skewed left with a mean of 3.80 and standard deviation of 0.733 showing a positive deviation to the right of the mean. No participants chose “strongly disagree” or “disagree” when responding. Most respondents (61.1%) chose “agree” or “strongly agree” after completing the course, with 38.9% indicating they neither agreed nor disagreed with the question. The responses indicated a positive response to the question overall indicating the learners would recommend the course to others. Although the result is favorable, the focus group data may reveal more information about course recommendation. Qualitative data will be gathered from the focus groups to establish key factors to understand why respondents would or would not recommend the course to others.

Frequencies for I would recommend this course to others I would recommend this course to Frequency Percent others

Valid Percent

Cumulative Percent

3

359

38.895

38.895

38.895

4

390

42.254

42.254

81.148

5

174

18.852

18.852

100.000

0

0.000

923

100.000

Missing Total

EDT 506: Evaluation Final Report

69

References Miller, C. (2021, March 1). College Enrollment Statistics [2021]: Total + by Demographic.

Education Data. https://educationdata.org/college-enrollment-statistics.

EDT 506: Evaluation Final Report

70