University of Alaska Fairbanks Modern Blanket Toss
2014 Summer Institute Student Data Report
Winter 2015 Prepared By: PO Box 83418 Fairbanks, Alaska 99708 (907) 452-4365 alarson@goldstreamgroup.com Jennifer Danielson, Evaluation Consultant Angela M. Larson, Principal Consultant Prepared For: Upward Bound University of Alaska, Fairbanks Fairbanks, AK 99775 Jon Monahan, Principal Investigator Pips Veazey, Co-Principal Investigator
Executive Summary The first summer session of the Modern Blanket Toss met at the University of Alaska Fairbanks in June and July of 2014. The student evaluation data show positive progress towards the projects objectives. Student pre/post surveys asked students to self-assess their skills and attitudes in several areas. The results showed statistically significant gains in the following areas: • Content knowledge • Leadership and communication skills • Attitude towards science • STEM career interest • Engagement in science topics. Weekly student surveys showed that over the course of the summer program, students • remained satisfied with the program throughout • increased interest in STEM careers • decreased their perception that the content of the course was challenging • increased leadership and communication skills • increased in ability to work as a team. Student interviews demonstrated that the students • were enthusiastic about the program • were confident in what they learned about UAVs • learned applicable leadership skills • made positive personal connections • were inspired to share their knowledge and skills with their communities.
2014 Summer Institute Student Data Report, 2
Introduction As part of the University of Alaska Fairbanks Upward Bound program, students from four rural communities in Alaska were given the opportunity to participate in Modern Blanket Toss, a course on UAV flying and applications. In addition to the Upward Bound goals of encouraging and supporting students in pursuing higher education, the Modern Blanket Toss program aims to encourage students to pursue STEM careers by giving them skills in communication, leadership and cutting edge technology. Community involvement is an important component of the project. The specific objectives of the six week course in the summer of 2014 were for the students to 1) learn how to safely operate small unmanned aircraft systems, 2) understand the basic concepts governing unmanned flight, 3) explore the applications of UAVs, 4) achieve a general understanding and awareness of STEM career fields, and 5) obtain knowledge of UAV applications, operations, and regulations. In the classroom the students learned safety rules and checklists. Despite an unusually rainy and windy June and July, the students practiced flying UAVs in teams several times each week. They focused their efforts on gathering data for a map of the UAF frisbee-golf course. Through their teachers, guest speakers and field trips they were exposed to several individuals working in STEM careers. Upward Bound students in the UAV 2014 summer course participated in three levels of evaluation. First, they completed pre and post surveys about their attitudes towards science and STEM careers as well as self-assessment of their science, leadership and communication skills. Second, they completed weekly surveys on satisfaction and recently covered content. Third, students and teaching staff were interviewed at the end of the course to share their opinions and successes.
Pre/Post Student Survey Survey Development The Pre/Post Student Survey was designed to measure levels and changes in the student’s 1) content knowledge and science practices (indicator 1.a); 2) leadership and communication skills (indicator 1.b); 3) attitude towards science (indicator 1.c); 4)STEM career interest (indicator 1.d); and 5) engagement and excitement in project topics (indicator 3.b). The survey included 5 open-ended questions and 58 Likert-scale items. The pre-survey was administered to students on paper in their first week of the Upward Bound Academy. The post survey was administered on the last day of classes at the Academy. A copy of the survey is in Appendix A.
Sample Eighteen students completed the pre-survey and seventeen of those students also completed the post-survey. Respondents identified themselves with an ID number on the surveys and pre and post data was matched for analysis; only matched pairs were included when comparing pre and post values.
Reliability The items in the survey combine to make 6 constructs or scales. To measure the reliability of our scales, we calculated Crohbach’s alpha coefficient for each scale using SPSS. Cronbach's
2014 Summer Institute Student Data Report, 3
Items
Item Mean
Min
Max
Range
Variance
Cronbach’s Alpha
Content Knowledge Science Practices Leadership Skills Communication Skills Attitudes Towards Science Attitudes Towards STEM Careers
n
Scale
alpha is a measure of internal consistency among items included in a scale.1 A Cronbach’s alpha that is greater than 0.7 is considered evidence that the items included measure an underlying construct.2 For all of our scales there was a high rate of internal consistency among the survey items. However, our small sample size (n=33) may bias the results.3 Results of the reliability analysis are summarized in Table 1. The survey items for each scale are included in Appendix B. Table 1: Reliability Analysis for Pre/Post Student Survey
29 31 31 31 33 32
14 11 7 12 4 6
5.552 2.842 3.207 2.992 3.121 3.479
3.483 2.548 3.065 2.161 2.909 3.313
7.586 3.129 3.355 3.774 3.333 3.625
4.103 0.581 0.290 1.613 0.424 0.313
2.059 0.035 0.008 0.287 0.045 0.018
0.963 0.899 0.881 0.937 0.818 0.887
Analysis and Results For each construct, we calculated a scale score for each respondent. The scale score is the average of the response to items included in the scale. For “Content Knowledge,” respondents rated their own knowledge from 1 (low level of knowledge) to 10 (high level of knowledge) so the scale score also has a range of 1 to 10. For the “Communication Skills” items, respondents chose “not comfortable at all” (equal to 1), “somewhat comfortable” (equal to 2), “moderately comfortable” (equal to 3), or “very comfortable” (equal to 4). The scale score is an average between 1 and 4. All other items asked respondents to “strongly disagree” (equal to 1), “disagree” (equal to 2), “agree” (equal to 3), or “strongly agree” (equal to 4), so those scales also range from 1 to 4 with 4 as a more positive score. We used an independent samples t-test to test whether the mean scale scores were significantly different (P < 0.05) between pre and post surveys. All constructs showed a significantly positive shift. Students learned content knowledge, leadership skills, communication skills and science practices. Their attitudes towards science and STEM careers improved. Results are shown in Table 2.
1
Trochim, William. 2006. Research Methods Knowledge Base. Web Center for Social Research Methods. Retrieved from: http://www.socialresearchmethods.net/. 2 George, D. and Mallery, P. 2003. SPSS for Windows step by step: A simple guide and reference. 11.0 update. 4th edition. Boston: Allyn and Bacon. 3 Yurdugul, Halil. 2008. Minimum sample size for Crohbach’s Coefficient Alpha: A Monte Carlo study. H.U. Journal of Education 35:397-405. Retrieved from: http://www.efdergi.hacettepe.edu.tr/200835HAL%C4%B0L%20YURDUG%C3%9CL.pdf
2014 Summer Institute Student Data Report, 4
Table 2: Paired t-tests for Student Pre/Post Survey Constructs Scale
Survey
N
Mean
Std. Deviation
Content Knowledge
Pre Post Pre Post Pre Post Pre Post Pre Post Pre Post
16 16 16 16 16 16 16 16 16 16 16 16
3.82 7.27 2.68 2.98 3.10 3.33 2.77 3.10 2.94 3.27 3.34 3.55
1.45 1.46 0.45 0.49 0.45 0.45 0.72 0.63 0.48 0.47 0.40 0.42
Science Practices Leadership Skills Communication Skills Attitudes Towards Science Attitudes Towards STEM Careers
Mean Difference
t
df
PValue
3.45
7.55
15
0.000
0.30
3.68
15
0.002
0.23
2.70
15
0.016
0.33
2.80
15
0.013
0.33
2.95
15
0.010
0.21
2.27
15
0.039
Both surveys asked students about their previous experience with mapping and UAVs and if they were excited to learn about such things. Students were given the option to strongly agree or agree and more students strongly agreed in the post test. This is shown by the average score for each question. The more respondents chose “strongly agree,” the closer the average gets to 4. Results are shown in Table 3. Table 3: Experience and Excitement to Learn Skill
Pre Average
Post Average
Change
I am excited to learn about mapping software I am excited to learn about UAV’s I have used mapping software before. I have used UAV’s before.
2.88 3.63 1.94 2.75
3.13 3.69 2.44 3.63
0.25 0.06 0.50 0.88
It is valuable to know the average responses for individual items within each construct. Students were asked to rate their knowledge on each topic on a scale of 1 (low knowledge) to 10 (high knowledge). Predictably, students reported learning more about how to safely and legally fly a UAV, than how to use a 3-D printer or ask questions using GIS software. See Table 4.
2014 Summer Institute Student Data Report, 5
Table 4: Content Knowledge Items Skill
Pre Average
Post Average
Change
How to acquire data from databases How to use a 3-D printer How to acquire different types of data using UAV’s Applications of UAV technology How to use data collected to develop a map/image that represents reality How to ask questions using GIS software How to fly a UAV How to use GIS mapping software The amount of money I can earn from science, technology, engineering or math career The safety rules and regulations that apply to flying a UAV Principles of flight (e.g., lift, drag, gravity) The range of science, technology, engineering and math careers that are available to me How to build a UAV The level of math and science I will need for various science, technology, engineering, and math careers
2.56 1.25 3.00 4.31 2.75
7.38 5.69 7.19 8.40 6.56
4.81 4.44 4.19 4.09 3.81
2.00 5.50 1.81 5.06
5.75 9.19 5.06 7.94
3.75 3.69 3.25 2.88
6.25
9.13
2.88
5.67 5.06
8.50 7.87
2.83 2.80
2.88 5.50
5.56 7.81
2.69 2.31
Survey respondents rated their agreement with statements about their Leadership Skills. The scale ranges from 1 to 4 where 1 is equal to strongly disagree and 4 is equal to strongly agree. The largest shift is in respondent’s ability to “develop, implement, and communicate new ideas to others.” There was no change in respondent’s ability to “listen carefully to what other people are saying,” but most students already agreed they could do that in the pre survey. Results are shown in Table 5. Table 5: Leadership Skill Items Leadership I am able to develop, implement, and communicate new ideas to others. I am open to new and diverse perspectives. I am able to work effectively with a diverse team. I am able to collaborate and cooperate effectively with teams. I am able to leverage strengths of others to accomplish a common goal. I am able to set goals and prioritize tasks to meet goals. I am able to listen carefully to what other people are saying.
Count
Pre
Post
Change
16
3.00
3.50
0.50
16 15 16
3.06 3.00 3.25
3.38 3.31 3.50
0.31 0.31 0.25
16
3.00
3.13
0.13
16
3.13
3.25
0.13
16
3.25
3.25
0.00
The respondents showed a clear shift in communication skills. More students agreed they could 1) talk in a large meeting of acquaintances; 2) talk in a small group of strangers, 3) talk with a stranger and 4) talk in a small group of acquaintances at the end of the program. There was a
2014 Summer Institute Student Data Report, 6
smaller shift in the studentsâ&#x20AC;&#x2122; ability to 1) talk in a large meeting of acquaintances; 2) talk in a small group of strangers; 3) talk with a stranger; and 4) talk in a small group of acquaintances probably because most students already felt comfortable with those types of communication at the beginning of the course. Results are in Table 6. Table 6: Communication Skill Items Communication Talk in a large meeting of acquaintances. Talk in a small group of strangers. Talk with a stranger Talk in a small group of acquaintances Present a talk to a group of strangers. Present a talk to a group of acquaintances. Talk in a large meeting of friends. Present a talk to a group of friends Talk in a large meeting of strangers Talk in a small group of friends Talk with an acquaintance. Talk with a friend.
Count
Pre
Post
Change
16 16 16 16 16 16 16 16 15 15 16 16
2.56 2.25 2.06 2.81 2.06 2.75 3.06 3.25 2.00 3.60 3.19 3.75
3.19 2.75 2.53 3.27 2.44 3.13 3.44 3.60 2.31 3.69 3.27 3.81
0.63 0.50 0.47 0.45 0.38 0.38 0.38 0.35 0.31 0.09 0.08 0.06
Survey respondents rated their agreement with statements about their attitudes towards science and STEM careers. Three of the STEM career related statements are negative, so for those items a negative shift in value represents a positive shift in attitude. See Table 7. Table 7: Attitudes about STEM Careers and Science Items Attitudes toward science Looking at scientific data in different ways helps me learn. Science is an interesting subject to study. Using a map to study scientific data helps me to learn. My scientific investigations are better when I begin by asking a question. Attitudes toward STEM careers I am interested in the way that science, technology, engineering, or math can be used to help people. Scientists/engineers are really cool people. A career in science, technology, engineering, or math would be financially rewarding. A career in science, technology, engineering, or math would be dull and boring. Scientists/engineers are boring people. Pursuing a degree in a science, technology, engineering, or math field does not interest me.
Count
Pre
Post
Change
16
3.06
3.44
0.38
16 16 16
3.13 2.75 2.81
3.50 3.13 3.00
0.38 0.38 0.19
16
3.44
3.75
0.31
16 15
3.25 3.27
3.38 3.31
0.13 0.05
16
1.56
1.38
-0.19
16 16
1.56 1.75
1.31 1.44
-0.25 -0.31
2014 Summer Institute Student Data Report, 7
Weekly Student Surveys At the end of each week, students were given a short satisfaction and content survey. Students were asked 10 Likert scale questions about their satisfaction with the Modern Blanket Toss program that week. They were also asked three open-ended questions about what they learned.
Sample Each week, between 15 and 17 students completed the satisfaction survey. Table 8 shows the counts for each week. Table 8: Weekly Survey Counts Week 1 Week 2 Week 3 Week 4 Week 5 Week 6
Week of June 13 June 20 June 27 July 4 July 11 July 18
Count 17 17 15 15 16 16
Analysis and Results Similar to the previous survey, students were asked how much they agreed with a series of statements and a numerical value was assigned to the strength of that agreement; “strongly disagree” equals 1 and “strongly agree” equals 4. The responses for each item, each week are averaged and compared. The larger the value, the more and stronger respondents agree. Table 9 reports the average values of the responses for each week. Satisfaction items remained high for all six weeks. STEM related items shifted in different directions. Communication and leadership skill items all increased over the 6 weeks. Table 9: Averages of Responses by Week and Item. Week 1
Count The modern blanket toss classes have held my interest this week. The presenters were well prepared. The environment (meaning the classroom, field trip location, or other place where instruction took place) was conducive to learning. I learned new science and technology concepts this week. The information was challenging to me. I enjoyed the classes this week and look forward to more. I am considering a career in a STEM field I feel confident communicating with others about science topics. I am becoming more of a leader through this class. I am comfortable working as part of a team
Week 2
Week 3
Week 4
Week 5
Week 6
17 3.35
17 3.35
15 3.47
15 3.27
16 3.38
16 3.44
3.29 3.41
3.35 3.41
3.67 3.57
3.33 3.27
3.38 3.19
3.25 3.50
3.06
3.24
3.13
2.80
2.81
3.13
2.69 3.53
2.71 3.59
2.80 3.47
2.27 3.33
2.56 3.50
2.38 3.38
2.88 3.06
3.06 3.00
3.20 3.13
3.13 3.27
3.38 3.44
3.31 3.38
2.65
2.76
3.07
3.07
3.06
3.25
3.41
3.53
3.47
3.33
3.69
3.53
2014 Summer Institute Student Data Report, 8
Items can be grouped into three categories. General satisfaction items are shown in Figure 1. Satisfaction remained high (average greater than 3.2) for all six weeks for all items.
Figure 1: Satisfaction Items from Weekly Surveys
2014 Summer Institute Student Data Report, 9
STEM Interest items shifted as expected. Students more strongly agreed that they were considering a STEM career as the weeks went by. The material covered became less challenging and the concepts were less new on the weeks when the students were mostly practicing their skills, not learning new material. See Figure 2.
Figure 2: Skill and STEM Interest Items from Weekly Surveys
2014 Summer Institute Student Data Report, 10
Figure 3 shows that item responses about communication and leadership skills generally increased over time.
Figure 3: Communication and Leadership Items from Weekly Surveys
2014 Summer Institute Student Data Report, 11
Open-ended Questions Students wrote similar descriptions about what they learned each week, suggesting they were similarly paying attention. For all weeks, students liked flying the UAVs best. Most students didn’t want to change anything about the course, but a few wanted more flying time. The following are illustrative: • I learned about UAV's because I didn't know anything about them and that they can film. I was amazed; I also learned how to control a UAV. (Week 1) • The leadership cabinet was a very fun experience and very convenient. • I really liked making the video about the 3-D printer and the flying. • Not much, seeing we already learned everything we needed to. I did get more comfortable with the Frolf course. I also feel like I refined my leadership and teamwork skills. (Week 5) • Nothing because I have made lots of memories with the people and places.
Participant Interviews Sample Fourteen students, the teacher and two teaching assistants were interviewed in the last week of the program to get more detailed information about the strengths and weaknesses of the course.
Analysis and Results Interviews were recorded and transcribed for analysis using Atlas.ti. Following is a summary of the analysis by primary themes: satisfaction, learning, leadership, plans after college, communication skills, sharing what they learned with their community, interest in learning more about UAVs, and different factors that influenced their learning. Both students and teaching staff agreed that the summer program was a great success. The following are examples of student responses. • •
This project was amazing, and I hope that we can continue it. I liked being part of something important, like this program. I think it's really important.
All agreed that they learned a great deal in the program. Student reported learning can be divided into 3 categories: 1) skills related to getting the UAV in the air and collecting data with it, 2) awareness of technologies they didn’t previously know about, and 3) leadership skills. All students learned about the importance and function of the flight safety checklist and the roles necessary for a team to safely and effectively fly a UAV to collect data. The following responses to “What did you learn?” are illustrative. • •
•
How to fly [A UAV] without panicking that much. The UAV aircraft kind of like has a brain … Like it has like a computer, and then the TX is the controller that controls aircraft, and those two…they kind of talk to each other. So, they kind of give each other some information, data, like where to fly and how high to fly. We learned about … the aircraft, like what the parts are and how to use those properly and correctly. We did a lot of experiencing how to fly aircraft… … a preflight, and then when you’re done flying there’s also a post-flight checklist. And when you’re doing your preflight checklist… instead of saying yes or I don’t know, he wanted us to say “check.”
Students learned about types of UAV beyond the ones they used in the course. They know that sensors and/or cameras can be changed on the UAV depending on the type of data a particular
2014 Summer Institute Student Data Report, 12
project needs. They know that 3D printers are a viable technology now and can be used to create parts for UAVs. During the leadership portion of the course, students learned four skills that help make a good leader (listen, learn, love, lead.) They practiced those skills in leadership cabinets where they solved a problem by following a protocol that gave everyone a say but also moved the decision process forward. Many students appreciated that the leadership skills were immediately applicable to their flight teams and that being a “facilitator” gave them the opportunity to practice leadership. The following are illustrative. • •
•
Yeah. This class made my leadership skills rise a lot. I feel more comfortable speaking in a bigger crowd than what I used to back at home. … we were working in groups, and we all worked together, and we took turns as roles. And I’d have to say [we learned leadership skills], because after the weeks have gone by of seeing progress in everyone’s ability to work together. Through the leadership classes I have learned that empathy is definitely a big thing when it comes to leadership, and you need to listen to what your other – what the other people are saying and then decide on an end result by compensating with everybody in a way. And a lot of the time most people don’t realize that when being a leader you have to listen to the other people. You can’t just make a decision. And what UAV flying is about is you have to have a team. You have to have a team to fly the UAVs, because there is copilot, pilot, payload operator, spotter. There are so many different roles that not just one person can do it all. You have to be able to work as a team, and in order for that you need to develop leadership skills, and I’d definitely say that my leadership skills have improved by being in this program.
Students had a wide range of interests they are considering pursuing in college, most of them STEM fields. Examples of topics mentioned as possibilities are: biology, physics, engineering, environmental science, herbology, math, medicine, nursing, geometry, criminal justice, business, architecture, liberal arts, biochemistry, and chemistry. Though many students were interested in STEM fields before attending Upward Bound, most also thought the Modern Blanket Toss program encouraged them further to pursue STEM fields in college. The following are illustrative. •
•
You need the science, technology, engineering, and mathematics to work with UAVs, and UAVs are really cool, so I think this has encouraged a multitude of people that normally wouldn't choose something 'cause they thought it'd sound boring. I’m already really interested in the sciences. I’m not too great at math, but when I can relate it to physics, I can do the math much more – much better than just math in general. So, when I – and when it comes to science and physics, if I relate my mathematics to science and physics, I can understand it better, so I’d definitely say that I want to do something that involves science.
Students in general did not answer enthusiastically when asked if they had learned communication skills in the program, though they did all prepare presentations for the class. A typical response to the question about communication skills is: •
I don’t know, I’m not that much of a – or I’m not used to being around a lot of people that I’m not used to, and if I probably do it, I probably won’t be able to speak in front of them.
However, they were ready and excited to share their knowledge of UAVs with their communities. •
I think I'm ready to share to my friends back at home about UAVs and how they work and how to operate them and what they're used for. 2014 Summer Institute Student Data Report, 13
•
•
I’m able to teach them how to operate one and how to do the payload checks, the checklist and to know what’s wrong with a UAV so we wouldn’t have to waste money on a crash. Pretty well prepared. I don’t know that much, but I know enough… like how to operate one, the payload checklist, how to find out more for the payload checklist by experience, and camera lenses, because they get oily, and those are not in the checklist… And how to teach them a little bit about mapping and collecting data.
Two students felt they could share their knowledge with some support from peers. • •
If I had help with others, 'cause it's hard for me to put my thoughts into words and say it out to others, but like if I had time before telling them. I've learned a lot of stuff, but I'll probably need help from the other people that came to explain more.
Both students and teachers agreed there was more to learn. Six weeks was not much time to cover the technological goals and there was just not enough time to cover everything that the students were ready to learn. Students became proficient at data collection, but only a few of them learned about transferring files to the GINA website and the Frisbee-Golf course data was not compiled and/or edited into a map before the students went home. • •
•
•
I feel like there’s a lot more that we need to learn in that class. I feel like we’re missing out, like we didn’t have enough time. The thing I would like to change is actually put layers of the codes into the computer like a GIS program software like we did last year and use the data we collected from the UAVs and put that into software and create a world of what we collected. Yeah. What would I change? I’d say nothing. I love the learning process you guys put us through. It helped out a lot. Rather than just those skills, like communication skills also and leadership skills. I would probably just throw in more mapping software time for us. That’s about it. I think it should be a little longer here, like six weeks is kind of short to learn about all this.
Students mentioned the friendships and the people connections as being an important part of the success of the program.
•
I thought it was fun because we got to fly in different areas with a lot of different people we've never met before in the first week, and now we're laughing and we're getting closer, like friendship close, and, yeah, I've pretty much liked the past – most – well, all of the six weeks I've been here because I have so many memories now.
•
From where I’m from, I call it my nerdy side, but a lot – there’s not a lot of people that you have that you can talk to about science without them scoffing, but in a place like this and in the setting that I’m in, whenever I’m with the UAV class, I can talk about my nerdy side in a way, and I can bring out the things that I like about science, and it’s much more comfortable for me to talk about the sciences because it’s what I enjoy doing. And I don’t exactly have that option at home, but I’d definitely say that this has made it more comfortable for me to talk about the sciences.
At the conclusion of the interview students were asked to rate how different factors influenced their learning. Results are presented in Table 10. Students most frequently reported that hands on projects, being an Upward Bound student, and the instructors were “a lot” of help in learning. Writing things and the expectations of others were the least helpful to student learning.
2014 Summer Institute Student Data Report, 14
Table 10: Factors that help in student learning
Doing a Presentation Things you write about Your own inner motivation to learn Being an Upward Bound student Expectations that other have of you Your friends or peers Hands-on projects Your instructors Technology
Not very much 0% 0% 0% 0% 9% 9% 0% 0% 0%
A little
A lot
27% 64% 18% 9% 55% 27% 0% 9% 27%
73% 36% 82% 91% 36% 64% 100% 91% 73%
2014 Summer Institute Student Data Report, 15