Executive Summary
Evaluating with a Real Data Set (Part 3) Maria Paula Borras-Patino
EDIT 7350: Evaluation and Analytics Dr. Lauren Bagdy 11/019/2023
Executive Summary
Summary Program
This evaluation assesses a 2-year professional development program designed to empower K-5 teachers to integrate technology effectively into their classrooms. The evaluation utilizes surveys, observations, and interviews to inform a strategic plan for the program's future.
Context
and Evaluating
Goals.
The program was instituted in an elementary school (K-5) with a student population of 600 students and 30 teachers.
The goal of the program was to provide teachers with the skills and knowledge needed to integrate technology into their instruction on a long-term basis and engaging teachers in active learning through in-classroom coaching and support, modeling, and observations.
The resources of the program were a total of 107 computers for student use, an interactive whiteboard, 30 computers with camera and projector for the teachers, technical support and a 2-year mentor support.
The timeline of the program training efforts, directed by the mentor, was as follow:
1 month
2 months
Year One
5 Months
Technology setup and issue resolution.
Teachers’ technical skills and tailored lessons.
Reinforcement of use of technology for instructions
Pedagogical strategies for technology integration
Active learning- including modeling, peer coaching, co-teaching, and co-development of lessons with the mentor
Year two 5 Months
Facilitate the formation of teacher-led communities of practice
Note: The program included 5 months of support by the Teacher led communities practice but is not included here since it was not led by the mentor.
The evaluation seeks to identify the program's strengths and weaknesses, offering insights for future improvements.
Findings summary/results
Strengths
Analysis indicates that the program generally achieved its goals, fostering successful student engagement and positive teacher perceptions regarding training and technology application as well as making it part of their long-term instruction programs.
The following were the most notable findings of the evaluation.
It was observed that the lessons with the new technological changes were a success from the point of view of the students. Students were engaged and appeared to have acquire the technological skills needed on their part almost seamlessly. This observation in turn shows that the teachers were able to use the technology into their lessons to their advantage by applying the new skills to make lessons that provided clear objectives and with easily understandable directions.
The evaluation showed that teachers felt adequately trained and believe the technology was easy to apply and useful for teaching.
Weaknesses
While the mentor played a crucial role, challenges in infrastructure impacted training time. Limited mentor hours and the exclusion of some teachers from mentorship affected the overall program effectiveness.
The program aligned with six research-based principles for effective professional development on technology integration (Mouza, 2006). However, weaknesses were identified in the areas of duration and collective participation.
Extensive duration was integrated by extending the training over a two-year period. However, upon scrutinizing the timeline, it becomes apparent that the mentor's support was present for eight months out of the first school year and five months out of the second year. This affected the duration of interaction each teacher had with the mentor. Consequently, this variance in mentor availability contributed to divergent opinions among teachers about the extent to which the mentor was beneficial, as revealed in the interviews.
The collective participation aspect was included by working with teachers from each class, and by the formation of teacher-led communities of practice; however, the communities of practice were established by teachers from grades 3-5 and they only acted as the leaders for their grade level team, leaving out all the teachers from grades k to 2.
Conclusions and recommendations
In conclusion the program has many strengths as previously describe and it could benefit from focusing on addressing the following weakness:
Infrastructure, hardware and in house IT help
The program should benefit from ensuring that all hardware, software, and internet connectivity provided for the use of the teachers and students are in good working standing. Additionally, to ensure this is made possible, it is recommended that an in-house IT help should be hired when implementing this program in other schools or to keep the current school’s equipment working appropriately. This will ensure that no time is taken away from the teachers' training or the students' instruction time.
Mentor
Hiring another mentor for the full two years would be of maximum benefit for the schools. The mentor’s support has had the most impact on the professional development of the teachers. Those who didn’t have sufficient time with the mentor showed no change in their perspective on the use of technology in the classroom, as they didn’t acquire the necessary skills for technology integration and pedagogical strategies. In contrast, teachers fortunate enough to spend time with the mentor changed their perspectives on the use of technology in the classroom. As one mentioned in the interviews, 'I can no longer separate technology from my lessons. I can't function without it.' Having an additional mentor would bridge the gap in teachers' knowledge, leading to increased and permanent integration of technology.
Survey Data Items and Descriptive Statistics
Table 1 Reflection
Most teachers received timely help with technology problems (45% agree, 45% strongly agree). However, 30% had concerns about technology reliability (10% neither, 20% disagree). Further analysis is needed, but teachers generally find the available technology useful (40% agree, 55% strongly agree).
Teachers had a positive perception of their workload and administrative support. They found the expectations about using technology for content objectives clear. Many felt there wasn't enough time for technology integration, even though their initial time expectations were met.
Teachers believed they received adequate professional development and could easily apply the training. However, they lacked time to share technology lessons with their peers.
Teachers believed using computers improved student learning and made designing activities easier. However, it didn't necessarily make their job as teachers easier.
Chart 1
Table 3 Reflection
When reviewing the numbers reflected in Table 3, it becomes clear that the lessons were overall successful, according to the observers. For each lesson except for a “Book Report” lesson, the most commonly occurring answer was “Strongly Agree,” meaning that the instructors presented the material clearly, students were engaged, and technological issues were minimal. Because one “Book Report” lesson had “Strongly Disagree” as the most common answer, attention should be given to that lesson to see why it creates less engagement among students. That being said, the range of most answers is 3, meaning there is a wide variance in responses. In fact, the other “Book Reports” assignment has a range of 3 and an average of 3, which is lower compared to most of the other responses.
Comparing those lessons to the “Ed Heads Simple Machines” lesson and the “Sharks WebQuest One” lesson, which were very successful according to the data, could provide ideas for change. In fact, the “Sharks WebQuest” assignments, of which there are three total, did very well, with only one (out of four) receiving a “Disagree” response.
Table 4.1
Interview Data Items and Frequencies
Table 4.1 Reflection
For questions one and two, the teachers were asked about their challenges with technology. For 30% of the respondents, the biggest challenge with technology is keeping up to date with technology advances and 20% of teachers expressed frustrations about technology interruptions such as internet outages or software glitches. When asked about additional challenges, the teachers responded with 60% being frustrated with the technology interruptions and 50% feeling a lack of knowledge about using technology. This information about the challenges leads us to conclude that the school should look into its infrastructure with regards to internet connectivity as well as technology compatibility for software or other programs.
Table 4.2
Interview Data Items and Frequencies
Frequency
Theme
3 Over the pass2 years, have your beliefs changed?
Frequency
3.a Explain why Theme
Table 4.4 Reflection
For question five, "How has the severity of barriers changed? What did the mentor help most/least with?" 40% responded that their mentor helped with professional development and ideation of using the technology, with an even split of 20% responding with assistance with access, vision, and time. With regards to the least help from a mentor, 20% also said access and time, though 50% did not label a "helped least with" for the question.
Table 5.1
Findings Survey
Teachers received timely help with technology. X
Technology was useful in the classroom
Teachers can easily create activities that use technology. X
Teachers consistently provided clear objective lessons.
Teachers provided easily understandable directions
Observations
Interviews Explanation
According to the survey of teachers, the teachers received help fixing technology issues in a timely manner (Mean=3 35)
According to the survey of teachers, the technology was useful for training (Mean=3 5) In the interviews, 80% of the teachers believed that the technology will help the students in the future
According to the survey of teachers, it is easy to incorporate technology into the design of a lesson (Mean=3 05) In the interviews, teachers also discussed how improved technology makes it easier to incorporate it into lessons for students
According to the observations, teachers' objectives were clear when using technology in the classroom (Mean=3.9).
The observations indicate that the teachers provided clear directions when using the technology (Mean=3 8)
Findings Survey
Teachers effectively handled technological issues
Technology helped the students learn better X
Students were comfortable using laptops.
Students actively participated in higherorder thinking skills.
Over the past 2 years teacher beliefs about technology have improved
According to the observations, teachers handled tech issues well Although in the interviews, teachers mentioned tech issues as a barrier, they do express being more confident in handling technology in general.
According to the observations, technology helped students achieve objectives in a way that would not be possible without the technology (Mean=3.8). In the interviews, teachers also agreed that the technology improved student learning
According to the observations, students were comfortable with the basics of laptop usage (Mean=3.8). Teachers also commented on students being able to adapt to using laptops in interviews
According to the observations, student activities focused on higher order thinking skills, such as synthesizing and analyzing (M=3.68).
In the interview, 80% of teacher respondents indicate their beliefs about technology have changed, and this change is articulated positively (better software, more training, improved abilities).
Weaknesses
Findings Survey Observations Interviews Explanation
Technology reliability (internet issues, computer running slow)
Keeping Up-to-Date with the Technology Advances
Not enough time for technology integration
Teachers could easily apply the trainings
Teachers lack time to share technology lessons with peers
20% of the teacher interviews identified technologyinterruptions as one of the 3 biggest challenges with using the technology
In the teacher interview, 30% identified keeping up-to-date with technological advances as one of the 3 biggest challenges.
In the survey, the questions regarding time had the lowest responses (Time to Integrate, M = 1 85), which shows that more teachers disagreed than agreed when asked if they had sufficient time to integrate
According to the survey, the mean response of easily applying technology in the classroom was only 2 95 according to teachers In the interviews, teachers mention tech issues and lack of knowledge regarding some technology as barriers
In the survey, the teachers average mean of 1 75 with regards to having time to share technology lessons with other teachers This was reiterated during the interview, but not as heavily indicated since only 20% of respondents mentioned it
Weaknesses
Findings Survey Observations Interviews Explanation
Teachers express they lack time to prepare and plan lessons that use technology X
Students may lose skills that are reinforced outside the use of technology.
Time with Mentor X
Students require intervention when using technology
The median and mean were 1 5 for the question that asked if there was enough time to prep or plan. Furthermore, two interviews also indicated that there was not enough time to explore or practice with the technology
In the interviews, teachers mention students losing thesaurus and dictionary usage skills that would still be helpful to them, as well as losing handwriting skills This is a major concern
Professional development was generally perceived as a strength by most of the teachers, as indicated by the responses to the survey questions on professional development (mode=3) and in the interview responses. Additionally, the frequency data revealed that 70% of the teachers that answered the survey either agreed or strongly agreed that they felt adequately trained.
However, upon closer examination of the data (Mean=2 75) and program details, it became evident that there might be an issue related to the time allocated to teachers for training with their mentor The mentor was only hired for 30 hours a week, which posed challenges for some teachers, impacting their ability to receive adequate training
According to the observations, students needed help when working with the technology (Mean=3 1) When interviewed, multiple teachers also expressed that students need help with issues like being confused when the technology doesn’t work or finding files that they saved
Recommendations: Areas for Growth
Mentor Hours
Both the interviews and observations indicated that the teachers need more time with the mentor to master the technology. The school consists of 30 teachers and 600 students and the district hired one mentor for only 30 hours a week In order for teachers to receive more time from the mentor, the district should hire another person The mentor is in charge of creating and facilitating professional developments, which does not leave much time for teachers to work individually with the mentor
In-House IT Help
Infrastructure and Hardware Resources
The study mentions “district technical support three times a week,” but based on the interview questions, there is a need for in-house support on a day to day basis. From the interviews, 60% are frustrated with the technology interruptions which might be quickly alleviated with onsite assistance, though the teachers had training to fix the most common problems
Coordinate with IT to review technology compatibility with the teaching software being used as well as the internet connectivity The teachers have a tremendous amount of technology to use; however, it was mentioned that many of these resources did not work and often required troubleshooting
Time in Peer Groups
The interviews and survey suggest that teachers didn't have enough time to integrate technology into their lessons or to share technology-related lessons with their peers ( 45% disagreed and 35% didn't have a positive opinion). If teachers are provided with sufficient time to share and reflect on their newly acquired knowledge, the integration of technology could become easier for them.
