LOR Example

PL E

LEARNING OUTCOMES

R E P O R T

School:

Destination:

EX A

M

Year:

1

The Dalton School South Africa 2014

programs • Envoys Pedagogical Model

• Areas

• Comparison to Staff

3 4 8 9

M

• Methodology

PL E

• Creating more meaningful

• Answering Choices and

EX A

• Data Gathering • Results Tabulation • Survey Results

10 10 11

Environmental Awareness

12

Positive Youth Development

13

Social-Cultural Awareness

14

Enriched Learning

15 2

Observations

• Digging Deeper

17 27

• Positive Youth Development

• Looking Ahead

29 30

• Annex Statistical Information

31

PL E

Creating more meaningful programs

M

At Envoys we are committed to measuring and reporting the effect of our programs. We believe that the best way to create more meaningful experiences is by investigating what students are learning during each program.

EX A

For many years experiential and global education programs have been an important part in schools curriculum, but there has been a lack of structured initiatives directed to measuring the learning associated with the experience.

While it is commonly accepted that students experience significant learning that becomes apparent when they come back, there are very few tools that allow us to measure the impact produced.

This study represents an effort to keep recording rigorous data to assess this important pedagogical aspect of our students’ lives.

3

In 2009 we began working with the American Camp Association and the International Camping Fellowship to develop tools that allow program administrators to assess the learning outcomes.

These efforts have lead us to design the tools used to measure the four Pedagogical Pillars that frame each one of our programs. These results allow Envoys and our partner schools to create follow-up plans to improve on the desired outcomes on an annual basis.

PL E

Envoys pedagogical model

M

Every Envoys program is designed and planed within a specific framework to ensure that our efforts are targeted on the development of specific skills and competencies.

EX A

We draw upon a variety of schools of thought to align the best of the formal classroom education world (CIA) with the experiential education world (REM) to promote our four Pedagogical Pillars: Positive Youth Development, Environmental Awareness, Social and Cultural Awareness and Enriched Learning.

4

Schools of thought 21st Century skills After school programming Experiential education Positive risk talking Multiple intelligences Behavior management through adventure

PL E

The best of two worlds The formal classroom education world

The experiential education world

C urriculum

R etention

E ngagement

EX A

I ntentionality

Students traveling in unfamiliar contexts have heightened recall of the experiences.

M

Improved curriculums that define the contents and experiences the program offers to students traveling.

Clear and specific learning outcomes that students need to accomplish through experience.

A ssessment

Specific methodologies to assess the degree of accomplishment that students are having with respect to objectives.

While in the field, students are engaged with and responsive to the environment around them.

M entorship

Students are open and receptive to guidance and mentoring from adults during their travel experience.

5

positive youth development

enriched learning

Students perform challenging tasks in foreign environments.

Students understand the process of building knowledge for effective action.

PL E

Pedagogical pillars

Students develop willingness to explore and confidence in trying new things.

Students increase self-direction and enjoyment of learning.

Students conduct meaningful academic work in challenging foreign environments and communicate their results.

M

Students feel able to foster meaningful change in the world.

CIA REM social-cultural awareness

Students understand the duty to act responsibly towards nature.

Students explore foreign communities and collaborate with people from different walks of life.

EX A

environmental awareness

Students gain respect for the environment and our place in it.

Students interact with nature and learn about the environment.

Students develop comfort in working with people form different social, culture and ethnic backgrounds.

Students have empathy and respect for others.

6

EX A

M

PL E

“Without data, you are just another person with an opinion”

~Andreas Schleicher, PISA

7

PL E

Methodology

EX A

M

Envoys has compiled a set of surveys based on the American Camp Association Youth Outcomes Battery and the ACA/ ICC International Camp Outcomes Project. The items measure student’s self perception at the time they are answering the evaluation and answers are marked on likert scales.

8

Our Baseline and Post-Trip surveys are aligned to provide a measure of comparision for each of our four domains. The Post-Trip survey allows for disaggregation of results within the domain of Positive Youth Development.

Students are also observed by staff throughout the Envoys program to ensure they are meeting criteria related to the four domains.

PL E

Areas

Environmental Awareness Did the program help students understand the natural world and the responsibility they have to protect the environment?

EX A

M

Social and Cultural Awareness Did the program help students understand and value social and cultural differences?

Enriched Learning Did the program help students to enrich their academic learning by transferring skills, exploring concepts, learning by doing and self-directing their learning?

9

Positive Youth Development Empathy Did the program help students to recognize emotions felt by others and undertake actions?

Teamwork Did the program help students to be more effective when working in groups of their peers?

Perceived Competence Did the program help students believe that they can be successful in the things they do?

Affinity for Exploration Did the program help students to want to experience new things?

PL E

Answering Choices and Data Gathering

M

The answer choices to each one of the questions of the survey were given on the likert scales that can be seen in figure 1 and a value was assigned to each one of these options to make a statistical analysis of the data.

For a detailed description on the psychometric testing made to assure that the American Camp Association Youth Outcomes Battery and the ACA/ICC International Camp Outcomes Project on which these surveys are based please see Annex 1.

EX A

Results Tabulation Decreased

Did not increase or decrease

Increased a little bit, maybe

Increased some, I am sure

Disagree a little

I Agree a little

Agree a lot

1. Becoming better at playing soccer.

Disagree a lot

1. I am good at playing soccer.

10

Increased a lot, I am sure

PL E M EX A

Survey Results

PL E

Unfortunately, Dalton students reported a decrease in their skillset for environmental awareness. This is most likely due the lower priority placed on engaging with nature during the program. While we engaged in coastal walks, they were largely constrained to urban environments. Furthermore, our safari experience took place while enclosed in private vans, which may have exacerbated student feelings of alienation towards nature.

5.0

3.35

2.5

M

Environmental Awareness

EX A

0.0

STATS:

PRE MAX: PRE MIN:

4.33 2.67

POST MAX: POST MIN:

3.07 2

PRE

12

2.43

POST

3.78

2.55

2.5

M

On average, Dalton students reported meaningful increases in the domain of Positive Youth Development. Students felt more willing to explore, higher degrees of self-efficacy, and greater connections with the world. Disaggregated results in this domain are discussed later in this report. A surprising individual note were that the two highest-scoring individuals for the pre-survey went on to report the highest and lowest scores for the post-survey. This intimates that there may be little correlation between previous levels of skill development and potential gains, and should be an area of further study.

PL E

5.0

Positive Youth Development

EX A

0.0

13

PRE

POST

STATS:

PRE MAX: PRE MIN:

2.95 2.21

POST MAX: POST MIN:

4.82 2.84

2.5

EX A

0.0

STATS:

PRE MAX: PRE MIN:

2.5 1.66

POST MAX: POST MIN:

4.66 2.77

2.43

3.35

M

All Dalton students reported meaningful gains in social-cultural awareness as a result of this program.

Throughout the program, Envoys instructors and Dalton faculty scaffolded students to confront the complexities surrounding their experiences, resulting in strong gains in understanding.

5.0

PL E

Social-Cultural Awareness

PRE

14

POST

2.5

2.43

3.35

M

This domain examines connections between academic study and the ‘real world’. Entering the program, Dalton students reported very low levels in this domain. The combination of teaching experience and focused discussions on international aid and poverty reduction resulted in meaningful increases in this domain.

5.0

PL E

Enriched Learning

EX A

0.0

15

PRE

POST

STATS:

PRE MAX: PRE MIN:

1.4 1

POST MAX: POST MIN:

4.28 2.56

PL E

Survey Results

Comparison to Staff Observations

EX A

M

Students are observed by staff throughout the Envoys program to ensure they are meeting each of the criteria through indicators that are designed to be specific, measurable, accurate and reliable.

Based on the number of indicators in evidence. Students receive a rating of Failure, Needs Improvement, Adequate, Proficient, or Advanced. These reports, along with staff comments, are sent to the students’ families at the conclusion of the program.

16

Envoys uses these staff observations as one means of confirming the results of the post-trip surveys. The domains are aligned to allow for comparisons between the means, minimums, and maximums.

As seen in the charts to the right, the Dalton student survey results were largely similar to the staff observations in all except the environmental awareness.

Staff felt that students exhibited greater degrees of comfort in nature than that reported by the students. This is likely due to a disconnect between internal feelings and outward behaviors during the experiential learning process.

Environmental Awareness

PL E MIN

MAX

EX A

Positive Youth Development

2.5

AVERAGE

M

5.0

Social-Cultural Awareness

Enriched Learning

0.0

5.0

POST

AVERAGE

STAFF OBSERVATION

MIN

MAX

2.5

0.0

PRE

STAFF OBSERVATION

17

5.0

AVERAGE

MIN

MAX

2.5

0.0

5.0

POST

AVERAGE

STAFF OBSERVATION

MIN

MAX

2.5

0.0

PRE

STAFF OBSERVATION

PL E

Environmental Awareness

EX A

M

The decreases in environmental awareness reported by Dalton students held true across the various disaggregators examined by Envoys. While there was some variation in entering levels of awareness reported by students dependent on their previous travel experiences, the final reported levels were very similar. This illustrates the uniformity of the opportunities (or lack thereof) for experiencing nature during the program.

18

We recommend targeting this domain in future programs with specific activities such as hikes and visits to wildlife reserves coupled with Leave No Trace training. While skills within the domains of social-cultural awareness and positive youth development are often (and rightfully so) given precedence on international programs, the opportunities for retention of environmental awareness while abroad should not be overlooked.

PL E M

Gender

Age 5.0

EX A

5.0

PRE

POST

PRE

5.0

POST

FEMALE

MALE

0.0

<17

19

PRE

POST

2.5

2.5

2.5

0.0

Previous Travel Experience

17+

0.0

<5

5 to 8

>8

PL E

Social-Cultural Awareness

EX A

M

The strong gains for social-cultural awareness held true across all disaggregators. In the free-response section of the survey, students noted that the substantial amounts of time and activities dedicated to interactions between Dalton students and Artworks peers, as well as the daily reflective and educational sessions on the South African context as critical elements for increasing their awareness.

20

It is interesting to note that the older students reported a higher final level of social-cultural awareness than their younger peers. This may be due to greater maturity allowing for a better sense of the â&#x20AC;&#x2DC;otherâ&#x20AC;&#x2122; and attempts to connect. It is also notable that previous travel experiences had little predictive effect on the reported gains. This illustrates the lasting power of the experiences and reflective sessions for this program, irrespective of how much a student has traveled previously.

PL E M

Gender

Age

EX A

5.0

PRE

5.0

POST

2.5

0.0

Previous Travel Experience

PRE

5.0

POST

MALE

0.0

<17

21

POST

2.5

2.5

FEMALE

PRE

17+

0.0

<5

5 to 8

>8

PL E

Enriched Learning

M

Increased connections between academic learning and the outside world held true for all disaggregators on this program.

EX A

In the free-response sections and staff observations, the male/female only learning groups were noted as particularly key to this process, as students engaged with their South African peers in an academic context and became motivated to learn more.

22

As one example of the continued motivation, during our layover in Johannesburg on the homeward trip, a Dalton student proudly purchased a book on international development that had been recommended by Envoys staff during the program.

Given these results, we can expect great things from these students during the upcoming academic year.

PL E M

Gender

5.0

5.0

EX A

5.0

Age

Previous Travel Experience

PRE

POST

PRE

2.5

0.0

POST

MALE

0.0

<17

23

POST

2.5

2.5

FEMALE

PRE

17+

0.0

<5

5 to 8

>8

PL E

Positive Youth Development

M

This domain examines a range of skills and attitudes that are necessary for global competency, including teamwork, empathy, willingness to explore, and perceived competency.

We examine each of these areas in the following pages.

EX A

When examining the disaggregators, we note statistically significant gains across all groups. It is interesting to examine the greater initial and final reports from the females on the program, similar to that reported for older students. As with the other domains, previous travel experiences did not have a significant effect on the final outcomes.

24

As a whole, students noted that the opportunities to live in and explore foreign environments as well as the challenge of being responsible for instructing their peers led to increases in these skills. A notable example was the use of student cooking groups, which were mentioned by several students as a highlight of the program for the development of selfresponsibility.

PL E M

Gender

Age 5.0

EX A

5.0

PRE

POST

PRE

5.0

POST

PRE

FEMALE

MALE

0.0

<17

25

POST

2.5

2.5

2.5

0.0

Previous Travel Experience

17+

0.0

<5

5 to 8

>8

PL E

â&#x20AC;&#x153;The belief that all genuine education comes about through experience does not mean that all experiences are genuinely or equally educative.â&#x20AC;?

EX A

M

26

~John Dewey

PL E

Digging Deeper

Within the domain of positive youth development, Envoys examines a set of skills that are critical to global competency. These include:

M

Affinity for Exploration The willingness to set out and interact with the unfamiliar, be it new foods, experiences, or people.

EX A

Perceived Competency The internal perception of ability to do things well, including both academic and non-academic tasks Empathy The ability to cognitively and emotionally understand the experience and perspective of others Teamwork The ability to successfully work as a member of a group towards achieving greater goals.

27

The chart on the following page illustrates the frequency of individual student responses within the post-trip survey for each of these areas, normalized to allow a better comparison.

PL E

Affinity For Exploration

11.00

Perceived Competency Empathy Teamwork

M

8.25

EX A

5.50

2.75

0.00

Decreased

No Change

Increased Slightly Increased Some 28

Increased A Lot

PL E

Perceived Competency

The bulk of student responses fell in the “increased slightly” or “increased some” areas, with a few students reporting “no change” for certain indicators. Given the opportunities inherent in this program, we recommend taking on more student-led collaborative projects to ensure stronger gains in teamwork.

Student responses were skewed towards “increased some”, demonstrating both positive results as well as room to grow. The lack of “increased a lot” responses is most likely an artifact of the time at which the survey was given. We expect that student self-image will continue to improve as this experience becomes a larger part of their identity after returning home.

Affinity for Exploration

Empathy

Students reported strong increases in affinity for exploration, with the bulk of the indicators at the ‘slight” or “some” increase level. While these are encouraging, we recommend building in more structured and unstructured activities in which students expand their comfort zones while abroad.

The domain of empathy saw the strongest gains for students, with several indicators reported at the “increased a lot” level. This demonstrates the efficacy of the experience--reflection cycle, as we regularly targeted the development of empathic skills.

EX A

M

Positive Youth Development

Teamwork

29

PL E

Looking Ahead

M

The Learning Outcomes Evaluation is a key component of Envoys programs. It represents a clear mechanism to gather relevant information and make thorough analyses of our programs.

EX A

Analyzing the information reported by students in every field reflects that this program has had a major impact on student development, and we expect to see continued progress made throughout the academic year.

Results of the survey are generally strong, though they do reveal a need for advanced planning for activities targeting environmental awareness. The large gains in enriched learning show the value of this program for the learning process, and the results of these evaluations should be used as evidence to the entire school community for supporting global education programming.

30

Finally, this report confirms the value of partnership in the process of planning, implementing and evaluating these programs. We do recommend that Dalton consider implementing the Envoys evaluation system across its global education offerings, as a means to better assessing the relative strengths of the various programs.

Such an initiative is a crucial element that allows Envoys to complement Dalton’s mission, by designing programs that target the school’s vision and purpose. In particular, we would like to build in more activities expressly targeting affinity for exploration, as we see this area as directly relevant to fulfilling the Daltonian objective to ‘go forth unafraid”.

PL E

Annex Statistical Information Youth camp outcomes questionnaires well as cross-structure analysis (inter-correlations among scale scores and correlations between scale scores and age)

2. Analysis of data supported the position that the Camper Learning survey can appropriately used as a unidimensional measure of generalized camp learning. The alpha reliability coefficient was .85. Itemto-total correlations ranged from .29 to .61.

Thirteen of the 14 coefficients exceeded .40, and eight of those exceeded .50. Random effects analysis of variance revealed no significant â&#x20AC;&#x153;interviewerâ&#x20AC;? variance.

M

Some individuals may be interested in the statistical details behind these questionnaires. The following information, while a bit detailed, may be helpful to individuals who must document the strength and appropriateness of these outcome questionnaires.

The following points highlight the construction and testing of the questionnaires:

EX A

1. A classical test theory foundation (e.g., DeVellis, 2003; Nunnally & Bernstein, 1994; Suen, 1990; Cronbach, 1990) was used in the four phases of this study: (1) identification of target outcomes, (2) operationalization of those outcomes, (3) data collection, and (4) data analysis and interpretation.

The questionnaires went through rigorous psychometric testing during the pilot testing phase (summer 2006) through the participation of eleven camps and a total of 791 campers. Psychometric evaluation included examination of the internal structure (reliability, item-to-total correlations, inter-item correlations) of the individual instruments as

3. Data from the Basic Camp Outcomes Questionnaires suggested that this format worked well. All reliability coefficients exceeded .90, and all item-to-total correlations were greater than .50. The final versions of each of these seven scales include 6-14 items and are scored on five point rating scales.

4.The Detailed Camp Outcomes Question-

31

naires format also worked very well. The change versions produced alpha reliability coefficients ranging from .87 to .93, and six of the seven alpha coefficients for the perceived status measures were above .80.

Criterion related evidence of validly based on the matrix of correlations among the scale scores supported the use of these scales for their intended purpose. The final versions of each of these seven scales (status and change) include 6-13 items and are scored on six point rating scales. http://www.acacamps.org/members/outcomes/statistical-information Recovered on April 15 2013.

PL E M EX A

Cambridge, MA 02142 One Broadway 14th Floor 1 (800) 515 6523 www.envoys.com info@envoys.com ENVOYS INTERNATIONAL Bogotá – Colombia Cra. 11B # 98-08 Of: 501 PBX: (571) 6910684