TOP External Evaluation 2012

Modern Germany in the Minds of American Students: An Evaluation of the Transatlantic Outreach Program (TOP)

August‐December 2011

By Carolin Hagelskamp and Amanda Roy

1

Executive Summary This report presents the findings of a theory‐based evaluation study designed to examine the impact of the Transatlantic Outreach Program (TOP) on elementary and secondary school students’ knowledge, impressions, and interest in Germany and Germans.

In the fall of 2011 1,115 students from 30 schools across the nation participated in an online survey. Of these, 510 had been taught Modern Germany lessons by a TOP fellow (TOP students). The remaining 605 students made up the comparison group (comparison students). They were of comparable age and grade level, and attended the same schools as the TOP students, but had not had exposure to TOP materials and Modern Germany lessons from a TOP fellow.

This evaluation finds strong evidence that TOP positively impacts students’ knowledge, impressions, and interest in Germany and Germans. Compared to their peers who have not been exposed to Modern Germany lessons, TOP students in this study know consistently more about Germany and have more positive and less negative impressions of Germans. They also perceive greater similarity between Germany and the U.S. than comparison students and are more interested in learning more about Germany and Germans.

Specifically:

• The majority of TOP students enjoy Modern Germany lessons and rate TOP materials as engaging • TOP students are more likely than comparison students to know about TOP fellows’ visits to Germany and personal contacts with Germans • TOP students know more about Germany and Germans than comparison students • TOP students have more positive and less negative impressions of Germany and Germans • TOP students perceive more similarity between Germany and the U.S. than comparison students • TOP students also express greater interest in learning more about Germany and Germans than comparison students • TOP students and comparison group students indicate similar levels of comfort meeting a hypothetical German student at their school

The report describes the study methodology, analytic approach and the demographic characteristics of participating students. It presents the study results in detail and concludes with recommendations. A copy of the survey instrument can be found in the appendix.

2

Goals and Questions The goal of the Transatlantic Outreach Program (TOP) is to encourage cross‐cultural dialogue and to provide Social Studies educators with global understanding from an international perspective using Modern Germany as the basis for comparison and contrast. To meet this goal, TOP provides teachers, or TOP fellows, with both educational materials (e.g. textbooks, DVDs, maps, lesson plans) and a two‐week study tour to Germany where teachers can learn about Modern Germany first‐hand. In this way, American students learn about Germany in two ways: i) through exposure to engaging curricula and study materials on Modern Germany and ii) through their teacher who has visited Germany, met Germans, and made German friends.

The goal of this evaluation study is to assess the Transatlantic Outreach Program’s (TOP) impact on students’ knowledge, impressions, and interest in Germany and Germans. To this end, a student‐ focused, theory‐based evaluation was developed.

Figure 1 summarizes the logic model informing this work. The model was developed based on TOP’s aims and objectives. It was further informed by social psychological theories of intergroup contact, which define modes of contact that can increase knowledge, reduce prejudices and improve impressions of members of a different social group (e.g., a different nationality or ethnicity). Relevant to TOP are the notions of “extended contact” and “parasocial contact”1. Having a teacher who has visited Germany, met Germans, and made German friends provides TOP students with a form of “extended contact” with Germany and Germans. Additionally, TOP materials and Modern Germany curricula create a type of “parasocial” contact with Germans for TOP students. As outlined in the model, these modes of contact constitute the processes by which TOP is expected to ultimately impact student outcomes (i.e., their knowledge, impressions, and interest in Germany.)

Figure 1: How TOP impacts students’ knowledge, impressions, and interest

Processes •

TOP

•

Outcomes

Exposure to engaging curricula and study materials on Modern Germany A teacher who has visited Germany, met Germans and made German friends

• • •

•

Knowing more about Germany and Germans Thinking more positively about Germany and Germans Perceiving more similarities between Germans and Americans Having a greater interest in Germany and Germans

1

Wright et al. (1997). The extended contact effect: Knowledge of cross‐group friendships and prejudice. JPSP, 73(1), 73‐90.; Schiappa, E., Gregg, P.B., & Hewes, D.E. (2005). The parasocial contact hypothesis. Communication Monographs, 72(1), 92‐115.

3

The evaluation is designed to answer a set of questions that parallel the processes and outcomes outlined in the logic model.

Processes 1. Do students find TOP lessons and materials interesting and engaging?

2. Are students exposed to TOP more likely to know about their teacher’s visit to Germany and contact with Germans than comparable peers who have not been exposed to TOP?

Outcomes 3. How do students exposed to TOP compare to peers who have not been exposed to TOP in the following areas?

• Knowledge of Germany and Germans • Attitudes towards Germans • Similarities between Germans and Americans • Interest in Germany and Germans

Method An online survey was used to collect data from students. Data collection took place between September 2011 and December 2011. The survey was completed by 510 students who had been taught Modern Germany lessons by TOP fellows (TOP students) and 605 students who had not had this experience (comparison students), but were selected to be similar in age and grade to TOP students. Participating students were recruited from 30 schools in the U.S. with the support of 30 TOP fellows.

Recruitment of TOP fellows A sample of 100 TOP fellows who had participated in study tours to Germany between the years of 2008 and 2011 were invited to participate in the study. After TOP fellows expressed interest in participating, each teacher’s principal and superintendent were contacted and permission to conduct the study was requested. Principals and superintendents indicated permission via email or by completing an online consent form. In several cases, the evaluation needed to be approved by school districts’ Institutional Review Boards. In these cases, applications were submitted and formal approval was obtained.

A total of 30 TOP fellows recruited students to participate in the study. Although the majority of TOP fellows responded to the request for participation and expressed interest in participating, some were not able to for a variety of reasons including a lack of approval from superintendents and/or principals, scheduling conflicts, and time constraints.

Recruitment of TOP students and comparison students TOP fellows were encouraged to recruit both TOP and comparison students. However, in several cases this was not possible given the timing of the curriculum or students having graduated. In these cases, fellows were asked to recruit at least one of the two groups. Of the 30 participating TOP fellows, 25 recruited both TOP and comparison group students. Four teachers recruited only comparison students and one teacher recruited only TOP students.

Of the 26 TOP fellows who recruited TOP students, 9 recruited students who they had taught TOP materials to in the previous academic year, 10 recruited students who they had taught TOP materials to in the current academic year (i.e., September or October 2011), and 7 recruited both students who they had taught TOP materials to in the current year and some who they had taught in the previous year. Of 4

the 29 TOP fellows who recruited comparison students, 23 recruited students from either a current class or one they had taught in the previous academic year (but who they had not exposed to Modern Germany lessons). Six recruited students from a class they had never taught before.

Procedure Top fellows were asked to recruit 15‐30 TOP and comparison students, respectively. They were instructed to pick these students randomly (or invite an entire class to participate) without considering students’ academic standing or potential knowledge and attitudes toward Germany.

TOP fellows were further instructed to introduce the survey as a study on students’ attitudes toward social science and learning about other countries, and encouraged to debrief students after they had completed the survey about this study’s specific intent.

Prior to participation, all students received parental consent forms which informed parents about the study and allowed parents to decline their child’s participation. Furthermore, students were ensured that their responses were anonymous and confidential, that participation was voluntary, and that there were no consequences if they decided to opt out.

The large majority of TOP fellows arranged for students to complete the survey together in the schools’ computer rooms and under the supervision of the TOP fellow. In a few cases, students were allowed to complete the survey at home. One TOP fellow collected the data in a pen and paper format and sent the completed surveys to the evaluators.

In order to ensure anonymity, TOP fellows received two lists of unique ID codes for participating students; one set of codes to be used by TOP students and the other to be distributed to comparison students. Students entered their individual codes upon completing the survey. As such, survey completion of individual students could be tracked without asking students to provide their names. Only TOP fellows kept lists linking ID codes with student names, but they had no access to student data.

The survey instrument The survey instrument was designed to fulfill three requirements: 1) to answer the study questions, 2) to be sufficiently general to capture knowledge, impressions and interests from a wide range of students varying in age, grade level and the type of TOP materials they have been exposed to, and 3) for students to not perceive it as a test.

A copy of the survey can be found in the appendix. It includes a set of core questions designed to assess each process and outcome of interest: a) students’ rating of Modern Germany lessons and TOP materials (for TOP group only), b) students’ knowledge of TOP fellow’s visit to Germany and contact with Germans c) knowledge of Germany and Germans, d) impressions of and feelings toward Germans, e) perceptions of similarities and differences between Germans and Americans, f) interest in Germany and Germans.

The majority of these questions were adapted from widely used and validated social psychological survey instruments2. Questions referring to students’ knowledge of Germany and Germans, and those referring to TOP fellows and TOP materials, were created for this study.

2

Aboud, F. E. (2003). The formation of in‐group favoritism and out‐group prejudice in young children: Are they distinct attitudes?

Developmental Psychology, 39, 48–60.; Binder et al (2009). Does contact reduce prejudice or does prejudice reduce contact? A longitudinal test of the contact hypothesis among majority and minority groups in three european countries. JPSP, 96(4), 843‐856

5

In addition, the survey also contains questions on students’ attitudes towards school and social science in general, which are analyzed in the appendix, and several open‐ended questions about Germany, that served as warm‐up to the core questions of interest. There are also a set of questions that assess student demographics.

The majority of questions are designed such that students indicate their agreement with a statement on a seven point scale (e.g., 1=not at all, 7=very much). Some questions follow a true‐false or yes‐no response format.

Analysis and interpretation of results Average differences in responses between the TOP group and the comparison group are analysed for each question using multi‐level regression models. This approach accounts for the nesting of students within schools, therefore “adjusting” analyses to take into account the fact that responses from students within the same school may be more similar than responses from students from different schools. In addition, all models are adjusted for the fact that more students in the TOP group study German at school than students in the comparison group (see Participant Description). This allows for more precise comparisons and greater confidence that differences in outcomes are a result of exposure to TOP rather than demographic differences between TOP and comparison group students.

Differences between TOP and comparison students are described as statistically significant if the p‐ values in the regression models are less than .05 (p < .05). The p‐value is a test statistic that indicates the likelihood of finding the observed result if no real difference exists. A cut‐off of p < .05 is commonly used in Social Science research as an acceptable indicator of statistical significance.

Presentation of findings To ease interpretation, data is primarily presented as percent endorsed instead of averages. Responses that fall above the mid‐point of the response scale (e.g. 5, 6, or 7 on a 7‐point scale) are presented as an endorsement (i.e., “True”) of the question or statement.

Participant Description A total of 1,115 students completed the survey. Of these students, 510 have been exposed to TOP and 605 have not. Overall, there are slightly more girls (57%) in the sample than boys (43%), on average students are in 9th grade, and the average age is 14 (Table 1). The TOP and comparison group students are very similar in terms of demographic make‐up. The one exception is that there are more TOP students than comparison students who study the German language at school. Because many of the TOP students participating in the study were exposed to TOP in the current school year, and would have made the decision to study German prior to exposure, it is difficult to draw conclusions about whether TOP influences students’ decisions to study German at school.

6

Table 1. Participant description Comparison (N=605) 57% 43% 9.08 14.30 6% 3% 15% 48% 6% 14% 35% *Indicates statistically significant differences (p<.05) between TOP and comparison group students.

Findings Processes The logic model (Figure 1) outlines the processes through which TOP is hypothesized to impact student outcomes. Specifically, it is expected that TOP will impact students i) through exposure to engaging materials and lessons that focus on Modern Germany and ii) by knowing someone (the TOP fellow) who has visited Germany and has German friends and colleagues. Therefore, the evaluation aims to determine 1) whether TOP students find TOP lessons and materials engaging and 2) if TOP students are more likely than comparison group students to know about the TOP fellow’s trip to Germany and contact with Germans.

Do students value TOP lessons and materials?

The majority of TOP students reported that they liked learning about Germany (73%) and the materials used by the TOP fellows were fun (58%). Moreover, only a small percentage of TOP students felt that the materials were boring (9%) and that the teacher spent too much time talking about Germany (7% (Table 2).

Table 2. Interest in TOP lessons and materials

Female Male Average grade Average age Speak German Study German at school* Study French at school Study Spanish at school Have visited Germany Have relatives currently living in Germany Have German ancestry

Overall (N=1,115) 57% 43% 9.15 14.38 6% 6% 15% 50% 7% 13% 35%

Our teacher taught us interesting things about Germany The materials she/he used to teach me about Germany were fun The materials he used to teach about Germany were boring Our teacher spent too much time talking to us about Germany

TOP (N=510) 56% 44% 9.26 14.51 6% 10% 15% 51% 8% 12% 35%

TRUE TOP 73% 58% 9% 7%

In order to get a sense of how often and in what format TOP students are exposed to TOP materials, the 26 teachers who recruited TOP students answered several questions about their use of TOP materials.

Most fellows (42%) reported using TOP materials or discussing Modern Germany in three to five lessons with the TOP students they recruited for the study (Table 3). In addition, many fellows reported having 7

TOP materials always (48%) or sometimes (40%) on display in their classrooms. The TOP materials most commonly used with TOP students were fellows’ own lesson plans (89%), maps and collages (69%), and DVDs (60%) (Table 4).

Table 3. Number of lessons per class during which TOP fellows use TOP materials or focuse on Modern Germany

0 lessons 0% 1‐2 lessons 38% 3‐5 lessons 42% 6‐9 lessons 17% 10+ lessons 4%

Table 4. % of TOP fellows using each type of TOP material

Own lesson plans 89% Maps and collages 69% DVD 60% Transparencies 28% Online games 20% Workbooks 20% Other fellows' lesson plans 16% Textbooks 12%

Are TOP students more likely to know about their teacher’s visit to Germany and contact with Germans?

TOP students are more likely than comparison students to know about TOP fellows’ travel to Germany, and be aware that the TOP fellow knows German people and has German friends (Table 5).

Table 5. TOP students are more likely to…

TOP Comparison …know that their teacher (the TOP fellow) has visited Germany 94% 73% …know that their teacher (the TOP fellow) knows German people 82% 64% …believe that their teacher (the TOP fellow) has German friends 67% 50% Note: All comparisons presented in this table are statistically significant (p<.05)

Outcomes It is expected that students’ exposure to TOP will influence students’ knowledge, impressions, and interest in Germany and Germans. To test this hypothesis, students exposed to TOP are compared to similar‐aged peers who have not been exposed to TOP along four dimensions: 1) knowledge 2) attitudes 3) sense of similarity and 4) interest.

Knowledge of Germany and Germans TOP students are more likely to endorse positive and diverse descriptions of Germany than comparison group students (Table 6). The largest differences are for “Germany has a strong economy” (a difference of 21%), “In Germany, many people recycle” (a difference of 20%), “Germany is a friend to the U.S.”, “Germany is an environmentally friendly country” and “Many people in the U.S. use products made in 8

Germany” (differences of 16%) (Figure 2). In addition, TOP students report having more to tell a friend about Germany than comparison group students (Table 7).

Table 6. How true do you think the following statements are for Germany?

TRUE

TOP

Comparison Difference**

Germany has a strong economy

63%

42%

21%

In Germany many people recycle

52%

32%

20%

Germany is a friend to the U.S.

64%

48%

16%

Germany is an environmentally friendly country

56%

40%

16%

Many people in the U.S. use products made in Germany

44%

28%

16%

Germany provides a good education for children and youth

66%

52%

14%

Germany is on good terms with many countries around the world

50%

39%

11%

Germany has many immigrants

37%

28%

9%

Germany has many writers and artists

55%

48%

7%

Germany is a multicultural country

42%

35%

7%

Germany is an important member of Europe

71%

66%

5%

Note: All comparisons presented in this table are statistically significant (p<.05) **The percentage of comparison students subtracted from the percentage of TOP students

Figure 2. Differences in knowledge of Germany What comes to mind when you think about Germany?

A TOP student:

“We learned a lot about environmental efforts in Germany by the government and corporations. Also there are artists, musicians and athletes.

Mountains and rivers. Large thriving modern cities.”

80% 70% +21 60% 50% 40%

+14 +16% +7% +16% +20% +11% +16% +7% +9%

+5%

30% 20% TOP

10%

Comparison

0%

9

What comes to mind when you think about Germany?

A comparison student:

“The only thing we have learned about is the wars and the holocaust, I feel like there is more to learn than just that.”

Table 7. Knowledge of Germany

A fair amount/A lot

TOP

Imagine a friend asks you to tell her/him everything you know about Germany, how much do you think you could tell your friend?

35%

Comparison Difference** 23%

12%

Note: All comparisons presented in this table are statistically significant (p<.05) **The percentage of comparison students subtracted from the percentage of TOP students

Attitudes towards Germans Participating students were asked to indicate whether twelve traits are What comes to mind when you generally true for Germans and Americans. TOP and comparison think of Germans? students differ in their ratings of Germans on six of the traits: TOP A TOP student:

students are more likely than comparison group students to think of “Smart and very hard working Germans as friendly, innovative and smart and less likely to rate and looking to the future of the Germans as cruel, boring, and stupid (Table 8 and Figure 3). Although the planet.” differences between TOP and comparison group students are only significant for six of the twelve traits, it is interesting to note that the pattern of the differences is consistent across the majority of the traits: TOP students rate Germans higher on positive traits and lower on negative traits than comparison group students.

In contrast, TOP and comparison group students only differ in their ratings of Americans on one of the traits. Comparison group students are more likely than TOP students What comes to mind when you to rate Americans as interesting. This lends support to the think of Germans? hypothesis that the observed differences are due to exposure to A comparison student: TOP. For the most part, TOP and the comparison group do not differ

in their ratings of Americans. However, students exposed to TOP “Hitler, Jews, blonde hair, and blue eyes.” rate Germans more positively than comparison students.

It should be noted that in examining the percentage differences it would appear there are other traits (e.g. healthy) in which a difference would be significant, but is not marked as such. In these cases, it can be concluded that the apparent difference is largely related to learning German at school, rather than a result of TOP exposure per se (as the analysis took into account such differences in language learning between the two groups). At the same time, this more stringent analysis allows for greater confidence that the observed differences, when marked as significant, are being driven by exposure to TOP and not other factors.

10

Table 8. Endorsement of traits for Germans and Americans

Germans

TOP Comparison Difference** Friendly* 70% 60% 10% Friendly Innovative* 53% 47% 6% Innovative Smart* 80% 74% 6% Smart Cruel* 15% 22% ‐7% Cruel Boring* 10% 15% ‐5% Boring Stupid* 6% 10% ‐4% Stupid Healthy 61% 53% 8% Healthy Peaceful 45% 40% 5% Peaceful Selfish 15% 20% ‐5% Selfish Good 62% 58% 4% Good Weird 22% 25% ‐3% Weird Interesting 78% 78% 0% Interesting*

Americans TOP Comparison Difference** 82% 83% ‐1% 56% 55% 1% 76% 81% ‐5% 23% 24% ‐1% 22% 21% 1% 33% 34% ‐1% 41% 45% ‐4% 53% 56% ‐3% 54% 52% 2% 72% 74% ‐2% 40% 38% 2% 72% 78% ‐6% *Indicates statistically significant differences (p<.05) between TOP and comparison group students. **The percentage of comparison students subtracted from the percentage of TOP students

Figure 3. Differences in ratings of German traits +10 +6 +6 ‐7 ‐5 ‐4

Participating students were also asked to imagine that a new kid from Germany came to their school and they were asked to show them around. Students indicated how much they would feel five different emotions: relaxed, embarrassed, awkward, secure, and nervous. Student responses to these five items were averaged to create an index of comfort that ranges from 1 to 7 with 7 indicating the most comfort with a German student and 1 indicating the least comfort. On average, all students reported high levels of comfort with a new German student and there were no differences between TOP (mean = 5.30) and comparison students (mean = 5.26) (Table 9). This suggests that the majority of students, regardless of exposure to TOP, would be comfortable showing a new student from Germany around.

11

Table 9. How comfortable would you be with a new kid from Germany?

Comfort showing him/her around

TOP 5.30

Comparison Difference** 5.26 .04

Note: NONE of the comparisons presented in this table are statistically significant (p<.05) **The average of comparison students subtracted from the average of TOP students

Students were also asked how much it would bother them if a new kid from Germany engaged in four different activities. There were no differences between TOP and comparison students in the ratings of these items (Table 10).

Table 10. How much would it bother you if a new kid from Germany...

TOP Comparison Difference** …worked with you on a class project?

4% 6% ‐2% …joined one of your favorite classes?

5% 6% ‐1% …won first prize in a competition at your school?

7% 8% ‐1% …joined your sports team or after school club?

5% 5% 0% Note: NONE of the comparisons presented in this table are statistically significant (p<.05) **The percentage of comparison students subtracted from the percentage of TOP students

Sense of similarity between Germans and Americans Students were asked three questions about similarities and differences between Germans and Americans. On all three questions, TOP students report more similarities between Germans and Americans than comparison group students (Table 11).

What comes to mind when you think of Germans?

A comparison student:

“I think of people who are different than me and I

do not know much about.”

Table 11. Similarities between Germans and Americans

What comes to mind when you think of Germans?

A TOP student:

“I think that Germans are much like Americans. They are mostly educated and are often times scientists. Many Germans come to college in America and are well traveled. Germans have deep family history and strong traditions in their culture.”

TOP

Comparison Difference**

Germans and Americans have a lot in common

44%

33%

11%

American and German kids my age are similar to each other

49%

40%

9%

Germans and Americans are very different from each other

19%

25%

‐6%

Note: All comparisons presented in this table are statistically significant (p<.05) **The percentage of comparison students subtracted from the percentage of TOP students

12

Interest in Germany and Germans Students were asked to indicate their interest in participating in nine different activities that involve getting to know more about Germany and Germans. TOP students were more interested than comparison students in participating in five of the nine activities (Table 12).

Table 12. I would like to…

TOP

Comparison Difference**

…learn about Germany at school*

62%

54%

8%

…watch German movies*

38%

31%

7%

… meet German tourists in my town*

51%

44%

7%

… celebrate a German holiday*

46%

40%

6%

…visit Germany*

81%

76%

5%

…have a German exchange student stay with my family

46%

39%

7%

…travel to Germany as an exchange student

53%

47%

6%

…eat German food

66%

62%

4%

…learn how to speak German

55%

51%

4%

*Indicates statistically significant differences (p<.05) between TOP and comparison group students. **The percentage of comparison students subtracted from the percentage of TOP students

Students were also asked if a new kid from Germany started at their school how interested they would be in getting to know him/her in three different ways. TOP students are more likely than comparison group students to indicate that they would be interested in visiting the German student at their home (Table 13).

Table 13. Getting to know a German student

TOP

Comparison Difference**

Visiting her/him at their house*

44%

37%

7%

Inviting him/her to your house

40%

36%

4%

Becoming friends with him/her

63%

60%

3%

*Indicates statistically significant differences (p<.05) between TOP and comparison group students. **The percentage of comparison students subtracted from the percentage of TOP students

13

Conclusions and Recommendations

In summary, this evaluation finds strong evidence that TOP positively impacts students’ knowledge, impressions, and interest in Germany and Germans. Compared to their peers who have not been exposed to Modern Germany lessons, TOP students in this study know consistently more about Germany and have more positive and less negative impressions of Germans. They also perceive greater similarity between Germany and the U.S. than comparison students and are more interested in learning more about Germany and Germans.

Moreover, this study provides evidence that TOP students learn about their teacher’s (the TOP fellow’s) visit to Germany and personal acquaintance with Germans, and that TOP students enjoy TOP materials and lessons.

Finally, TOP students and comparison students report similarly high levels of comfort and low levels of irritation when meeting and spending time with a hypothetical new German student at their school.

The results of this study benefit from two important methodological and analytic strengths. First, TOP students are compared to peers of similar age and grade level. They attend the same schools and completed the survey at roughly the same time. Moreover, TOP and comparison students in this study all know the TOP fellow at their respective school personally. Observed differences between the two groups of students are thus most likely attributable to the fact that one group, yet not the other, has been exposed to TOP materials and Modern Germany lessons.

Second, all results are based on statistical tests that take into account the fact that more students in the TOP group study German at school. While this study cannot determine why this difference exists (and whether or not it is due to TOP exposure), it is important to note that none of the statistically significant differences can be attributed to the difference in language learning.

For the majority of the outcomes, the difference between TOP and comparison students is relatively small (less than a 10% difference). This is not surprising given that most teachers only use TOP lessons and materials in several lessons; it is not sustained over the course of the school year. However, it is still noteworthy that the differences between the TOP and comparison groups are as robust and consistent as they are. This suggests that even in their brief exposure to the lessons and material, TOP students are being influenced by the program.

Despite a strong design, this evaluation does not allow for definitive conclusions about causality to be drawn. Causal inferences can only truly be drawn when an experiment that employs random assignment is used. In the case of TOP this would mean randomly assigning classrooms of students to either receive TOP curricula or a comparable curriculum that does not focus on Germany. In this way, differences between students and classrooms (e.g. demographics, academic engagement, and teaching ability) that may also contribute to differences in knowledge and interest would be randomly distributed across the two groups. This would mean that one could be confident that any observed differences in outcomes are a result of exposure to the program and not a result of other differences between the groups.

14

There are several recommendation that can be made for program adaptation and future evaluations based on these findings:

• A very small number of teachers (12%) report using TOP textbooks in their lessons. This is much lower than teachers’ utilization of other TOP materials such as maps and collages (69%) and DVDs (60%). Therefore, efforts to make textbooks more engaging and user‐friendly may increase utilization.

• Differences between TOP and comparison students are the strongest and most consistent in the area of “Knowledge of Germany and Germans” and the smallest and least consistent in the area of “Interest in Germany and Germans.” This suggests that this might be an area to be targeted and further developed in TOP lessons and materials.

• In working with TOP fellows, it became clear that a number teach in communities with significant German ancestry and/or in schools that already focus on German culture and language through school‐wide events and American‐Germany history month celebrations. In these contexts, students are likely to already know more about Germany and be more interested in Germany compared to the “average” American teenager. TOP per se may thus have less of an impact on students’ knowledge and interest in Germany and Germans in such schools. In order to use TOP resources most efficiently, it may be important to consider teachers’ school and community context when evaluating fellowship applications.

• The funders might consider conducting an experimental, longitudinal follow‐up study that follows TOP fellows and their students over time. This would allow for causal conclusions to be drawn and the longevity of TOP effects to be assessed. In addition, TOP may want to consider conducting a national representative study of American teenagers’ knowledge, impressions, and interest in Germany and Germans. The results of such a study could serve a valuable baseline against which to evaluate the findings of this and future TOP evaluations.

The evaluators would like to thank the TOP fellows who volunteered substantial energy and time to this project and the students who were willing to share their thoughts.

15

Appendix

A. Additional analyses

Comfort with a new student Participating students were asked to imagine that a new student came to their school and they were asked to show them around. Students indicated how much they would feel 5 different emotions: relaxed, embarrassed, awkward, secure, and nervous. Student responses to these 5 items were averaged to create an index of comfort that ranges from 1 to 7 with 7 indicating the most comfort with a new student and 1 indicating the least comfort. On average, all students report high levels of comfort with showing a new student around. However, TOP students reported statistically significant lower levels of comfort (mean = 5.37) than comparison group students (mean = 5.53).

Academic engagement Students answered 6 questions about their feelings and engagement in school (e.g. I try hard to do well in school). Responses to these 6 items were averaged to create an index of academic engagement that ranges from 1 to 7 with 7 indicating the highest engagement and 1 indicating the least engagement.

There were no differences between TOP (mean = 5.34) and comparison students (mean = 5.41) in reports of academic engagement.

Learning about other countries Students answered 4 questions about their interest in learning about other countries in school (e.g. I enjoy learning about other countries in class). Responses to these 4 items were averaged to create an index of interest in other countries that ranges from 1 to 7 with 7 indicating the highest interest and 1 indicating the least interest. There were no differences between TOP (mean = 5.24) and comparison students (mean = 5.11) in interest in learning about other countries.

16

B. Evaluator Biographies Previously, Amanda Roy and Carolin Hagelskamp have worked together as research consultants to the United Nations Department of Peacekeeping and Department of Field Support (DPKO/DFS).

Carolin Hagelskamp (Ph.D.) received formal training in evaluation and intervention methodology and statistics at New York University, working with senior evaluation experts and statisticians. Her expertise is in evaluations of educational programs for children and youth. Her project experience includes a five‐year longitudinal study with 1,000 youth in NYC public schools, an evaluation of a literacy program in 100 primary schools in rural Kenya, and an impact analysis of a social‐emotional learning program on teachers, classrooms and youth in 60 middle‐ schools in Brooklyn, NY. She now works for the Public Agenda Foundation as Director of Research.

“As a former Youth for Understanding exchange student to the U.S. – and now a German living in the U.S. – I am particularly excited about the opportunity to evaluate TOP. In addition to having been an exchange student, my family hosted American exchange students in Germany, I mentored American exchange students in Germany and I organized a language camp for American exchange students in Germany. My experiences have taught me that positive contact and personalized, engaging education can build intercultural bridges and lifelong interests and connections.”

Amanda L. Roy (Ph.D.) is trained in Community Psychology, Developmental Psychology, and quantitative methodologies. Her expertise is the study of how setting‐level programs and characteristics can influence families and children. She has worked on several large‐scale projects including an experimental evaluation of how changes in welfare policy affect children, a longitudinal study of low‐income children’s development, and an experimental study of how housing quality matters for families and children. While employed as a research consultant for Chapin Hall Center for Children and the Education Development Center, she advised on evaluations of a neighborhood level intervention targeting the provision of services to families and an education initiative that introduced technology into school‐districts affected by Hurricane Katrina. She has extensive experience in study design, survey development, data analysis, and presentation of findings.

17

C. Survey

18

Dear Student, Welcome to the online student survey! We are contacting you from the Goethe Institute in Washington D.C. In order to find out what kids like you think about Social Studies and learning about other countries we are asking students from all over the country to complete a survey. In this survey there are no right or wrong answers. We simply want to know what you think.

The survey will take about 20 minutes to complete. Please respond honestly and try to answer as many questions as you can.

Your answers will not be shared with anyone outside of our research team (this means that your responses are CONFIDENTIAL). We will not ask for your name and your answers will not be shared with anyone from your school.

If you have any questions about this survey, please email us at TOPeval2011@gmail.com Click the button below to begin the survey. j I agree to participate k l m n

Please enter the 5 digit code that your teacher gave you.

Your school

Imagine a new kid came to your school and you were asked to show them around. How would you feel? ­ please respond to each ­ Not at all

2

3

4

5

6

Very much

Relaxed

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

Nervous

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

Secure

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

Embarrassed

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

Awkward

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

How true are the following statements for you?

Not at all true

2

3

4

5

6

Very true

I try hard to do well in school

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

I do not care about doing

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

I pay attention in class

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

School is fun

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

When I am in school I feel

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

well in school When I am in school I feel bad

good

The next questions are about things you might learn about in school. How true are the following statements for you?

I am interested when we

Not at all true

2

3

4

5

6

Very true

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

learn about other countries in class Learning about other countries in class is boring for me I pay attention when we learn about other countries in class I enjoy learning about other countries in class

Think about a time you learned about other countries in school. Which countries do you remember learning something about? ­ Please type out your answer in the box below ­ 5

6

Germany and Germans

One topic that some kids learn about in school is Germany. When you think of the country Germany, what comes to mind? Please write all of the words you can think of.

5

6

Now think of the people who live in Germany. When you think of Germans, what comes to mind? Please write all of the things you think of.

5

6

Imagine a friend asks you to tell him/her everything you know about Germany, how much do you think you could tell your friend? j Nothing k l m n

j A little k l m n

j A few things k l m n

j A fair amount k l m n

j A lot k l m n

Germany and Germans

Now we are interested in how true you think the following statements are for Germany.

Remember there are no right or wrong answers ­ we just want to know what you think.

Germany has many writers

Not at all true

2

3

4

5

6

Very true

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

and artists Many people in the U.S. use products made in Germany In Germany, many people recycle Germany provides a good education for children and youth Germany is on good terms with many countries around the world Germany is an environmentally friendly country Germany has many immigrants Germany has a strong economy Germany is an important member of Europe Germany is a friend to the U.S. Germany is a multi­cultural country

Germans and Americans

In general, choose the traits that you think are true for Germans. Check as many as apply.

Germans are .... c cruel d e f g

c freindly d e f g

c innovative d e f g

c healthy d e f g c weird d e f g

c interesting d e f g c stupid d e f g c good d e f g

c smart d e f g

c boring d e f g

c peaceful d e f g c selfish d e f g

Imagine a new kid from Germany came to your school and you are asked to show them around. How would you feel? ­ please respond to each ­ Not at all

2

3

4

5

6

Very much

Relaxed

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

Embarrassed

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

Nervous

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

Secure

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

Awkward

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

If a new kid from Germany started at your school, how much would it bother you if they did the following?

Won first prize in a

Not at all

2

3

4

5

6

Very much

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

competition at your school Joined one of your favorite classes Worked with you on a class project Joined your sports team or afterschool club

If a new kid from Germany started at your school, how interested would you be in doing the following?

Inviting him/her to your

Not at all

2

3

4

5

6

Very much

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

house Becoming friends with him/her Visiting him/her at his/her house

Germans and Americans

How much do you agree with the following statements?

Germans and Americans are

Not at all

2

3

4

5

6

Very much

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

very different from each other Germans and Americans have a lot in common American and German kids my age are similar to each other

Germans and Americans

Please tell us how much you would like to do the following things.

I would like to ....

Not at all

2

3

4

5

6

Very much

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

...eat German food

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

...watch German movies

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

...celebrate a German

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

...learn how to speak German

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

...meet German tourists in my

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

...learn about Germany at school

holiday ...have a German exchange student stay with my family ...travel to Germany as an exchange student

town ...visit Germany

Germans and Americans

In general, choose the traits that you think are true for Americans. Check as many as apply.

Americans are ... c innovative d e f g c smart d e f g

c selfish d e f g

c interesting d e f g c good d e f g

c boring d e f g c cruel d e f g

c stupid d e f g c weird d e f g

c peaceful d e f g c friendly d e f g

c healthy d e f g

Your teacher

Now think of XX, who is helping out with this project. Please tell us whether you think the following statements are true or not true.

She/he has visited

True

False

I don't know

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

Germany She/he visits other countries She/he knows German people She/he has German friends

Teacher X taught me things about Germany j Disagree a lot k l m n

j Disagree k l m n

j Neither disagree nor k l m n agree

j Agree k l m n

j Agree a lot k l m n

Think back to when XX talked to you about Germany in class. How much do you agree with the following statements.

She/he spent too much time

Not at all

2

3

4

5

6

Very much

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

j k l m n

talking to us about Germany The materials he/she used to teach about Germany were boring She/he taught us interesting things about Germany The materials (books, stories, pictures) she/he used to teach me about Germany were fun

About you

Are you a boy or a girl? c boy d e f g c girl d e f g

How old are you?

Years

6

Age

What grade are you in?

Grade this year

6

Grade

About you

What languages do you speak? Please choose all that apply. c English d e f g

c Spanish d e f g

c French d e f g

c German d e f g

Other (please specify)

Do you study a language at school?

j Yes k l m n j No k l m n

About you

What language do you study at school? c Spanish d e f g c French d e f g

c German d e f g

Other (please specify)

About you

Have you ever visited Germany?

j Yes k l m n j No k l m n

Do you have relatives who currently live in Germany?

j Yes k l m n j No k l m n

Did your parents, grandparents, or other relatives immigrate to the U.S. from Germany?

j Yes k l m n j No k l m n