Dispositions to Stay Project: Annual Report 2008-2009 1. Background 1.1 Institutional Context The Dispositions to stay project is a collaboration between five Higher Education institutes: •
The University of Northumbria is the lead institution, where the principal researcher and research assistant are based. It uses the Effective Lifelong Learning Inventory (ELLI) across a range of academic programmes. At Northumbria, Student Retention is a priority remit for two key strategic groupings, the Student Affairs Committee and the Learning and Teaching Committee. The two Deputy Vice Chancellors involved both fully support the project. Both committees are being kept fully briefed on progress and the project is being overseen by the DVC (Learning and Teaching), who is chair of the steering group.
•
The University of Bedfordshire similarly uses ELLI across a range of its programmes. The university has a long-established Retention Action Group (RAG) which involves senior members of staff including the DVC (Academic), the Registrar, Dean of Students and the Director of Teaching and Learning. The RAG is fully committed to the project and to the dissemination of the findings. The University has established the Bedfordshire Guidance and Information Network (BEGIN) to ensure that staff and students receive consistent and supportive information that supports student retention and attainment. This provides another important conduit for the work of the project.
•
The University of Glyndwr uses ELLI with several cohorts of undergraduate and postgraduate students to raise awareness of the seven dimensions of learning power.
•
The University of Manchester uses ELLI with pre-university students on its Manchester Access Programme (MAP). At Manchester the project is supported by the Vice-President (Teaching and Learning) and reports formally to the institutional Teaching and Learning Group. It also reports informally to the Student Experience Office, and the Student Recruitment, Admissions and International Development Division through the Widening Participation team.
•
The University of Bristol developed the ELLI instrument and has made it available in the higher education sector for research and development. It supplies identifiers and passwords to students to enable them to complete ELLI profiles. .
1.2 Philosophy of the Project Three hypotheses to be tested were set out in the project’s funding bid: 1
• Some students are more disposed to stay at University than others • These dispositions to stay are malleable and can be strengthened by strategies to raise student awareness and support personal development • Whilst institution-wide strategies are important these need to be supported by personalised and situated approaches to be fully effective. Key to the project is the use of the Effective Lifelong Learning Inventory (ELLI) following its piloting in higher education through the ELLI in HE project, which involved thirteen universities. The online Inventory asks student 90 questions about their learning. 72 of these questions are used to provide a measure of their learning power and the remainder are currently being evaluated. The qualities and characteristics associated with each of the dimensions are described briefly below: Creativity This refers to the extent to which students are able to look at things in different ways and imagine new possibilities, rather than tackling every task in an identical, rule bound manner. Changing and Learning This is the extent to which students expect to see themselves grow and develop as learners, taking new approaches to expand their repertoire of learning skills, rather than believing that learning power is fixed and that difficulties simply reflect their limitations. Critical Curiosity This is the extent to which students want to ask questions and get below the surface of what is going on, rather than accepting what they are told and believing that ‘received wisdom’ must be correct. Meaning Making Meaning makers look for links between what they are learning and what they already know, enjoying seeing things ‘fit together’, rather than approaching learning sessions piecemeal and responding to them on their individual merits. Resilience and Robustness Students with this characteristic like a challenge and are willing to have a go at something when the outcome is uncertain; they carry on when learning is confusing or frustrating. Students without this characteristic are risk averse and react negatively to getting stuck or making mistakes. Learning Relationships 2
This refers the extent to which students are able to work effectively on their own or with others, striking the correct balance and not becoming isolated or dependent. It also refers to the extent to which they are willing to seek help where appropriate. Strategic Awareness Students with high levels of strategic awareness plan their time, like trying out different approaches and thinking about their own learning, being able to explain their reason for taking a particular approach. 1.3 Aims of the Project The project has three proposed outcomes: 1. To use ELLI to identify students at risk of disengagement 2. To actively explore the use of ELLI for planned retention interventions 3. To use ELLI to identify effective retention strategies In order to achieve these outcomes, evaluation will take place at several levels: Evaluation of Students’ Dispositions to Learn (outcome 1) Bringing together students’ ELLI profiles with data about progression and retention will facilitate an analysis of the types of dispositions that place students most at risk of leaving their programme without their intended award. The inclusion of programme related and demographic data will facilitate further analysis of whether there are dispositions that present a high risk among students with particular characteristics or who are studying a particular type of programme. In addition to the quantitative data, focus groups and individual interviews with staff and students will provide a more detailed insight into dispositions to learn. Evaluation of the Use of ELLI in Its Own Right (outcome 2) The use of ELLI as a retention tool in its own right will be evaluated in three ways: •
•
The process of using ELLI – how many staff and students at the partner institutions were persuaded to take part, steps that were taken to facilitate the use of ELLI (e.g. dedicated lab sessions), the convenience of the process for staff and students (as expressed through interviews and focus groups). The perceived effectiveness of ELLI – assessed by staff and students through focus groups and individual interviews. Where possible, comparisons will also be made between progression and retention rates before and after the introduction of ELLI, although care will be taken not to make unjustified claims of cause and effect. 3
•
Different methods of using ELLI – staff and students will be asked about different ways in which ELLI has been used (e.g. a tutorial may have been introduced to discuss a student’s profile directly after they completed ELLI). This evaluation will take place through interviews and focus groups. Where possible, the impact will also be measured using progression and retention data.
Evaluation of the Use of ELLI as a Diagnostic Tool Suggesting Further Interventions (outcome 3) Again, there will be evaluation of both the process and the outcomes of introducing new retention initiatives based on the findings from the quantitative data which combines a student’s ELLI profile, their demographic information and data about their marks/progression: •
•
Records will be kept of the number of interventions that are introduced on the basis of the identification of problematic dispositions to learn. The process of introducing these initiatives will be evaluated. The outcomes of these new initiatives will also be evaluated, in a manner that is consistent with the commitment in the bid to action research. Data will be collected from staff and students about the perceived effectiveness of the new initiatives. Where possible, the students selected will be those whose ELLI profile suggests that they are particularly at risk of ‘dropping out’. It is also hoped that the impact of new initiatives can be measured using progression and retention data.
In discussion with the external evaluator, David Baume, and with Helen May of the HEA a clear consensus has established that the results of the project must be used to bring benefits to as many students in higher education as possible. 2. Progress The project has received ethical approval from the School of Arts and Social Sciences at Northumbria University. Other Schools at Northumbria, and other institutions, have sometimes separately submitted details of the project to their own ethics committees; approval has been given on every occasion. The project has also been ethically audited as part of an institution wide process at Northumbria. One of the most challenging issues in the first year has been adapting the ELLI instrument to ensure that students are asked their permission for their data to be used in the research project. We are grateful to colleagues in Bristol for their assistance with this process, which is now complete. All students in the participating institutions who log on to ELLI are now asked whether they are willing for their data to be analysed as part of the research. 4
The lack of such a permission statement for much of the 2008-2009 academic year has limited the number of cases where quantitative data is available to be analysed. However, some excellent work by colleagues across institutions has led to manual permission statements being signed by students who have already used ELLI, meaning that there is enough data available for analysis in the coming academic year (see findings section below). In addition, some work has been undertaken with the SITS team at Northumbria which has made possible the creation of records incorporating both ELLI and SITS data. In relation to qualitative data, a highly successful event to promote the Dispositions to Stay project and to share information took place in Stevenage in March, organised by the University of Bedfordshire. In addition to sharing ideas, this produced qualitative data which is discussed below. As with the quantitative data, the amount of qualitative data that has been collected to date has been limited by practical factors, although the project team are confident that there will be more substantial coverage in the next academic year. However, it has still proved possible to collect qualitative data from a range of sources: •
A focus group has been conducted of Northumbria staff who are studying for the Postgraduate Certificate in Academic Practice. This group were in the unique position of being able to comment on the use of ELLI both as staff and students.
•
Four individual interviews have been conducted with student ambassadors at the University of Manchester. These are students of the university who support potential students on the Manchester Access Programme. They were able to comment on the use of ELLI both for themselves and for the access programme students.
•
Two focus groups have been conducted with students on the Manchester Access Programme.
•
An interview has been conducted with a member of academic staff in one of the partner institutions.
•
Data has been collected on a proforma from all academic staff who are known to have introduced ELLI to students.
There has been substantial dissemination of the early findings and concerns of the project, which have often doubled as promotional events to encourage academic colleagues to become involved, as discussed in the dissemination section below. 3. Findings In this section, the quantitative analysis is followed by the qualitative. The quantitative section is divided into two subsections: the first considers all those students where ELLI data was available, while the second 5
concentrates on those students for who ELLI and SITS data could be joined together. Some of the findings are followed by an expression in the form (p=). This refers to the likelihood that a relationship between variables could have occurred by chance and is not representative of a true pattern. So the expression (p=0.15) means that there is a 15% likelihood that an apparent relationship between variables does not exist in reality because the sample chosen was not representative of the population. Where p=0.05 or less, i.e. there is a 5% or smaller chance that a result could have occurred by chance, the finding is described as statistically significant and the relationship between variables is accepted as a true one. 3.1 Quantitative Data from ELLI 590 ELLI records were available for analysis: 502 provided by Northumbria University and 88 by the University of Bedfordshire. The 502 records from Northumbria were from the following academic programs: • • • • • •
218 first year Business Studies undergraduates studying a Human Resource Management module 150 first year Psychology undergraduates 66 first year undergraduates in the School of Computing, Engineering and Information Sciences 19 second year undergraduates in the School of Computing, Engineering Studies 43 staff who were also students, studying the university’s Postgraduate Certificate in Academic Practice (PCAP), which is a requirement for all new academic staff 6 students from various other undergraduate programmes
The students from the University of Bedfordshire were all first year undergraduates and were from the following programmes: • • • • • •
46 from Psychology 17 from Journalism 11 from Educational Practice 4 from Computing 3 from Biosciences 7 from various other programmes
The mean scores across each of the dimensions of learning power are shown below. These figures have little meaning on their own, but should help to place later discussions of changes in scores into context: 73.4 for meaning making 70.0 for changing and learning 59.3 for critical curiosity 6
58.8 for resilience 57.6 for strategic awareness 67.1 for learning relationships 50.7 for creativity Comparisons were made between the ELLI scores of different teaching groups. The PCAP students scored higher than some other groups in a number of areas. Specifically: •
On critical curiosity, they scored significantly higher than Business Studies (p=0.000), Psychology (p=0.000) and first year Computing students (p=0.001)
•
On meaning making they scored significantly higher than Business Studies (p=0.002), Psychology (p=0.002) and second year Computing students (p=0.005)
•
On resilience they scored significantly higher than Business Studies (p=0.000), Psychology (p=0.000) and fist year Computing students (p=0.001)
•
On strategic awareness they scored significantly higher than Psychology students (p=0.007)
There were no significant differences between the PCAP students and any other groups in the areas of Changing and Learning, Learning Relationships or Creativity. The Psychology students scored significantly lower on a number of learning dimensions than other groups of students: • • •
They scored significantly lower on critical curiosity than Journalism students (p=0.039) They scored significantly lower on creativity than both Journalism (p=0.025) and first year Computing (p=0.025) students They scored significantly lower on learning relationships than Educational Practice (p=0.009) and Business (p=0.033) students
The only other statistically significant difference between groups was that Educational Practice students scored higher than Journalism students in the area of learning relationships (p=0.02). There were no significant differences between the first year students and the second years in relation to the learning dimensions, although the second year scores were higher in every dimension except creativity (where the first years had a higher mean score by 0.1). The difference between the two year groups was: 8.1 in the case of resilience 6.6 in the case of learning relationships 7
5.6 in the case of critical curiosity 5.5 in the case of strategic awareness 4.8 in the case of changing and learning 0.1 in the case of meaning making Unfortunately, the demographic data that students provided when completing ELLI could not be downloaded, so comparisons between men and women, for example, could only be made where ELLI and SITS data could be combined, as discussed in the next section. It is hoped that this difficulty can be resolved next year and that ELLI records can be downloaded with demographic details included. 3.2 Quantitative Data Combining ELLI and SITS In the majority of cases, the ELLI records of the Northumbria students could be linked to their SITS data, ensuring that scores on the learning dimensions could be linked to characteristics of the students and to their academic marks. This was clearly not appropriate in the case of the PCAP students and there were other cases – including all those from the University of Bedfordshire where technical difficulties prevented matching. However, in 378 cases data could be matched: the Northumbria students were distributed between academic programmes as follows: • • • •
196 were first year Business Studies students 150 were first year Psychology students 19 were second year Computing Engineering and Information Science students 13 were first year Computing Engineering and Information Science students
Some of the characteristics of this student group were: • • •
166 were male and 212 were female 340 gave their nationality as British, 12 as Chinese and 26 as another nationality The ethnic origin of 283 was recorded as White, of 40 was recorded as White British, of 21 was recorded as Chinese and of 24 was recorded as another ethnic origin
Students had a wide range of previous qualifications. For ease of analysis, these were grouped into the following categories: • • •
A level or GNVQ level 3 (the classification that most students appeared under in the SITS data) Other qualifications above the level of A level (e.g. HNC, Foundation Degree, GNVQ level 4 or above, degree from the UK or overseas) Other qualifications equivalent to A level (e.g. Access Course, Foundation Course) 8
•
Not qualified to A level standard (other non-advanced, mature student or no formal qualifications)
The number of students falling into each of these grouped categories is shown in the table below: Table 1: Grouped Qualifications Qualifications
Number/Percentage of Students
Above A level standard A level or GNVQ level 3 Equivalent to A level standard Not qualified to A level standard Information not available
28 (7.4%) 303 (80.2%) 34 (9.0%) 5 (1.3%) 8 (2.1%)
Students were asked about the occupational background that they came from; the answers they gave were placed in one of seven standard categories. As can be seen from the table below, there was a substantial amount of missing data. Where data was available, approximately half of the students were from professional and managerial categories: Table 2: Occupational Categories of Students Occupational Category
Number/Percentage of Students
Higher managerial and professional occupation Lower managerial and professional occupations Intermediate occupations Small employers and own account workers Lower supervisory and technical occupations Semi-routine occupations Routine occupations Missing
60 (15.9%) 86 (22.8%) 40 (10.6%) 31 (8.2%) 11 (2.9%) 39 (10.3%) 17 (4.5%) 94 (24.9%)
Impact of Dispositions to Learn on Retention and Marks 11 first year students had a mean mark of 0 recorded – it was assumed that they had left the programme. These students were of particular interest to the research so were compared to the 345 first years who had marks of greater than 0 recorded. Although none of the differences were statistically significant, comparisons between the two groups showed that students who were assumed to have withdrawn had mean ‘scores’ on the dimensions that were: 5.1 lower in the case of critical curiosity 9
4.8 lower in the case of meaning making 2.7 lower in the case of changing and learning 1.1 lower in the case of creativity 1.2 higher in the case of strategic awareness 4.9 higher in the case of resilience While the numbers involved were very small, these findings raise some important questions. The higher scores recorded for the withdrawing students in the areas of strategic awareness and resilience suggest that leaving their academic programme may have been a well thought out decision that required determination. The nature of student withdrawal will be examined further in the second year of the project. Next the students who were assumed to have withdrawn were removed from the analysis and consideration was given to the relationship between first year marks and scores on the seven dimensions of learning power. Two dimensions were found to have a statistically significant impact on marks. In both cases the correlation was positive, meaning that high ‘scores’ on the dimensions were related to high mean marks: • •
Critical curiosity, had a correlation of .098 (p=0.034) with mean mark Changing and learning, had a correlation of .094 (p=0.04)
The correlations between the other dimensions and first year marks, although not significant, are listed below: • • • • •
Strategic awareness .06 (p=0.135) Resilience 0.041 (p=0.222) Learning relationships -.005 (p=0.465) Meaning making -.018 (p=0.372) Creativity -.082 (p=0.063)
In the case of learning relationships, meaning making and creativity, the minus figures indicate that higher scores on these dimensions were related to lower first year marks. However, as these findings are not significant, they may have occurred by chance, although this is least likely in the case of creativity. While the relationships were statistically significant, it should be noted that the impact of critical curiosity and changing and learning on the mean first year mark were limited – a correlation of .094 between changing and learning and mean first year mark indicates that only 0.88% of the variation in mean marks was explained by variation in a student’s changing and learning ‘score’. Differences Between Groups Men scored significantly higher on three of the learning dimensions than women: the dimensions were critical curiosity (p=0.000), resilience (p=0.003) and creativity (p=0.014). However, despite this, women had a higher mean mark for the first year (although not significantly so) – 56.1% compared to 10
55.8%. This provides further evidence that there was no simple relationship between scores on the dimensions and academic performance. Men and women were separated in the analysis to consider further the relationship between scores on the dimensions and marks. None of the scores had a statistically significant impact on mean first year mark for female students. However, for male students, there were three statistically significant relationships (in all cases high scores were associated with high marks): • • •
Changing and learning (correlation .2, p=0.007) Critical curiosity (correlation .191, p= 0.01) Strategic awareness (correlation .169, p=0.02)
When considering students studying different subjects, Business Studies students achieved significantly higher mean marks if they had lower scores for creativity (correlation - .122, p=0.045); a finding that echoes that of previous work conducted in schools. Extreme caution should be exercised when considering the results of the first year CEIS students, as there were only 13 included in the dataset, but the correlations with first year marks were strikingly strong in the case of three dimensions: • • •
Changing and learning .921 (p=0.000) Strategic awareness .808 (p=0.000) Critical curiosity .699 (p=0.004)
(In all three cases these were positive correlations, i.e. high scores were associated with high mean marks). Among the 19 second year CEIS students, there were no significant relationships between any of the dimensions and second year mean mark. However, some further analysis was undertaken to determine factors that had an influence on changes to their marks between the first and second years. 9 students achieved a lower mean mark in the second year than they had in the first, while 10 saw their mean mark improve, although in many cases the differences were small. In only one case was an increase in mean mark associated with higher scores on a learning dimension (although the difference was not significant) – the dimension in question was critical curiosity. In all other cases, the correlation was negative, i.e. higher scores on the dimensions were associated with a falling mean mark, and in the case of learning relationships this correlation was significant (correlation -.542, p=0.008). The small numbers again mean that this finding should be treated with caution but the possibility that high scores on some of the dimensions are associated with deteriorating academic performance is one that must be investigated further in subsequent years of the study. The two largest subject groups were subdivided by gender with the following results: 11
•
For women in the Business School, high marks were associated with high levels of resilience (correlation .271, p=0.006). However, they were also associated with low scores in the area of creativity (correlation -.186, p=0.043) and learning relationships (correlation -.189, p=0.04). There were no significant relationships among dispositions to learn and mean mark for men in the Business School.
•
For men in Psychology, high marks were associated with low scores on creativity (correlation -.326, p=0.042). For women in Psychology, there were no significant relationships.
When considering relationships between a student’s socio-economic background and the learning dimensions, a statistically significant correlation was found with learning relationships (correlation - .138, p=0.013). As the table below shows, students from routine occupations scored the lowest in this area: Table 3: ‘Scores’ on Learning Relationships for Students From Different Socio-Economic Backgrounds
socio-economic background higher managerial and
Mean
N
Std. Deviation
58.15
55
16.557
59.46
76
15.780
56.95
37
18.868
56.86
29
18.824
51.26
11
11.205
semi-routine occupations
56.16
37
15.413
routine occupations
48.70
15
15.462
Total
57.10
260
16.594
professional occupations lower managerial and professional occupations intermediate occupations small employers and own account workers lower supervisory and technical occupations
British students were compared with those of other nationalities according to their first year mean mark and scores. Non-British students scored significantly higher in the area of creativity (p=0.000) and strategic awareness (p=0.008).However, despite these differences British students had a mean first year mark of 56.3, compared to 52.6 for students of other nationalities, although this difference fell short of being statistically significant. Some more crude re-coding was necessary in order to make any comparisons based on ethnic origin. Students were divided into the categories of White, Black, Asian, South Asian and dual heritage. There were no significant 12
differences in mean first year marks or on any of the learning dimensions, except that Chinese students had higher levels of creativity than White students (p=0.046). When comparing students according to their re-coded previous qualifications, there were no statistically significant differences on any of the learning dimensions. However, students who were classified as having A level of GNVQ3 qualifications achieved significantly higher mean first year marks than those who had equivalent but different qualifications (p=0.003). 3.3 Conclusions Relating to Quantitative Data From the limited data on retention, and the much more substantial data on student marks, the following conclusions can be drawn. 1. It should not be assumed that students who withdraw from academic programmes are less powerful learners than those who progress. The findings suggested that the opposite is the case for some of the dimensions. 2. There was no simple relationship between scores on the dimensions of learning power and student marks. This was demonstrated by negative correlations between scores on some dimensions and mean first year marks and some groups – i.e. men and students of non-British nationality - achieving lower than average first year marks, despite scoring highly on some dimensions. 3. Critical curiosity emerged consistently as a factor positively correlated with student success. Given the mixed findings in relation to the data on other dimensions, it is worth emphasising the consistency of the findings in relation to this one: •
High scores on this dimensions were significantly associated with high mean first year marks. This was true when considering the student group as a whole, men only and first year CEIS students only.
•
Second year students achieved higher scores on this dimension than first year students.
•
For second year students, this was the only learning dimension where higher scores were associated with an increase in marks between the first and second years.
4. Changing and learning was another dimension where there was consistent evidence of a positive impact on academic performance. There was a significant positive correlation between high scores on this dimension and high marks in the case of all first year students; this result was repeated when considering men only and first year computing students only. 13
5. For two sub-groups – men and first year CEIS students – high levels of strategic awareness had significant relationships with high first year marks. However, there was little evidence of such positive impact among the student group as a whole . 6. Of all the dimensions, the value of creativity (at least as measured by ELLI) is questioned most by these findings. Again, it might be worth listing all the evidence in relation to this dimension: •
Among those students who submitted first year work, higher scores on creativity were associated with lower first year marks. This relationship was not significant for the student group as a whole but a significant negative correlation was found in the case of women in the Business School and men studying Psychology.
•
Men and overseas students had significantly higher scores on creativity than other students but achieved lower mean first year marks (although the difference in marks was not significant).
•
Creativity was the only dimension where first year students achieved a higher mean score (albeit marginally) than second year students.
7. In relation to meaning making, learning relationships and resilience there were mixed findings, making it difficult to assess their impact on academic success. The ambiguity was particularly surprising in relation to meaning making which was thought by academic staff and students to be an important dimension of learning power, as will be seen in the qualitative data section below. 8. Differences between students in different subject areas could be related to the nature of the subject: for example, positive learning relationships could be more important for those studying Educational Practice then those studying Psychology and creativity may be an important quality for journalists to have. Less easy to explain are men scoring higher across several dimensions than women and Psychology students scoring lower than other groups of students on a number of dimensions. Given that, in both cases, the dimensions involved included both critical curiosity and creativity, it is difficult to know how to interpret these findings. 9. Perhaps the most important findings in relationship to different groups was the lack of statistically significant differences on the key dimensions of critical curiosity and changing and learning. Men scored significantly higher than women, and Journalism students scored significantly higher than Psychology students, on critical curiosity. However, with these exceptions, there were no significant differences on the two key learning dimensions when considering differences based on gender, subject studied, nationality, ethnic origin, previous qualifications or socio-economic background. 14
3.4 Qualitative Data An Evaluation of the Use of ELLI as a Retention Tool in its Own Right As was noted above, the amount of qualitative data that could be collected this year was limited and skewed towards those students who were part of the Manchester Access Programme (MAP). While this limitation is important some very interesting key themes have emerged: Dispositions to Learn There are specific dispositions, felt by students and staff alike, to be significant for learning and thriving in higher education. Interestingly these findings run parallel to those from quantitative analyses on significant dispositions to learn. The qualitative data suggests that students and staff feel that critical curiosity is significantly the most important disposition for success in higher education. Changing and learning and learning relationships were also felt to be of considerable importance. Critical Curiosity •
Every time a respondent commented on the importance of one of the dimensions a note was made of this through a tally method – from this just under one third of the respondents felt that the most important quality to possess for success in higher education is the ‘critical curiosity’ – that is, a ‘thirst for knowledge’.
•
As one participant commented “critical curiosity is important for motivation in learning and I think it covers every school” (PCAP, p. 12)
•
Critical curiosity was seen as what could be described as the driving force behind either the possession or development of many or all of the other ELLI dimensions
•
For example a student ambassador for MAP students comments “if you’re critically curious and appreciate the material and what you learn, meaning making comes into play” (SAI, p.4)
•
MAP students found that the seven dimensions in general, but critical curiosity in particular, to be of huge benefit when thinking about the transition between school or college and higher education 15
•
For example one student notes that critical curiosity is the most important dimension in learning for dealing with “that transition between being spoon-fed and being more on your own” (MAP Focus Group 1, p.4-5)
•
Some staff have asserted that this dimension is a particularly difficult one to develop and improve upon feeling that “somebody is either curious or not but I would hope that most students have a level of it for coming into higher education” (PCAP, p. 14)
•
Whilst this last point is important this view considers the dimensions as fixed. However, one of the central principles underpinning research into ELLI and learning power is an understanding that all seven dimensions are malleable and situational.
Changing and Learning •
Changing and learning was another dimension that appeared to be of high importance in higher education for both staff and students.
•
Learning is seen as an ongoing progression, needing to be continuously considered and reconsidered. As a PCAP focus group participant noted “students need to see that there isn’t an end to learning. That it’s not a case of coming to university and learning everything they need to know and that’s the end of it” (PCAP, p. 12)
•
This dimension appeared to be particularly meaningful for students who have been outside of education for a while and have perhaps had some negative past educational experiences.
•
For example, as one staff member noted, “the dimension of changing and learning was very relevant for the mature students” (Practitioner view 1)
Learning Relationships •
Learning relationships is also regarded as an important dimension and skill to develop in higher education
Students in higher education need to have an understanding of the importance of working together to solve problems – a skill particularly relevant for beyond education settings i.e. the workplace, citizenship etc
There have been some observations that this dimension, however, can be slightly confusing given that it appears to have two opposing meanings; learning alone and learning in groups 16
For example “doesn’t seem to be a dimension; its two which are really contradictory things” (PCAP, p.3)
However, the important point here is that those who have strong learning relationships have effectively struck a balance between these opposing camps, that is, learning to seek help when and where it is appropriate
“Things get harder once you get to university so you can’t always do everything on your own. You need to be able to ask for help and work things out with other people who might have another approach” (MAP Focus Group 1, p. 6)
Creativity •
Creativity seems, across the board, to be a highly contested dimension.
•
Conversations with both staff and students suggest that creativity is the least important for higher education unless the programme of study is artistic in nature.
•
For example a MAP Ambassador asserts “I do think that creativity would be linked to subjects like English Literature, perhaps Art, Music, Design, that type of thing” (Practitioner View 3)
Staff and Student Views on the Process of Using ELLI and its Effectiveness Self-reflection and Personal Development in Higher Education Both of the student focus groups undertaken involved Manchester Access Programme (MAP) students. This group are particularly relevant with regards to generating an understanding of students’ views on ELLI, due to their pretransitional circumstances from school or college to higher education. MAP is a scheme that supports post-16 students’, who meet certain criteria based upon specific academic and circumstantial criteria, into successfully gaining a place at either the University of Manchester of another research intensive university. These students were asked to do ELLI in order to help them to understand the importance of self-reflection and personal development to the higher education experience. Overall the MAP students appeared to appreciate the importance of reflecting on their personal development and felt that ELLI was a very useful way of preparing them for what will be expected of them in higher education. MAP students were particularly worried about the transition and many asserted that ELLI dimensions, particularly critical curiosity, meaning making and learning relationships, have helped in showing them “how you would be able to tackle this and how you can improve on that” (MAP Focus 1, p. 5) 17
MAP students appeared confident about the possibilities of developing their own learning power through discussing, in groups, ways of developing weaknesses across the seven dimensions – this was particularly the case when discussions were centred on anonymised profiles. Another group of students interviewed, individually, about their experiences of using ELLI were Student Ambassadors, facilitating MAP students with their UCAS applications. Many of the Student Ambassadors could see the usefulness of ELLI themselves once they could see how they were able to use it to help the MAP students in understanding the significance of personal development and self-reflection in their UCAS applications. This suggests that peer activities could be a very useful way in helping students to understand what is being asked of them and develop solutions together. Staff who have used ELLI with their students have had mixed experiences with this process. Some staff agree that ELLI has “got them to think about themselves and their approach to their future”; however, others have commented that in the first instance ELLI “has not resulted in a significant amount of insight or understanding or changed behaviour.” Some staff have commented that the student reaction to self-reflection using ELLI has been mixed, in some cases being largely superficial. However, it is also felt that this is largely down to a general lack of understanding of the dimensions, and suggestions that there is further potential if staff had “the resource or skills to follow up the ELLI questionnaire with suitably facilitated follow up sessions” (Practitioner View 2) The Language of ELLI Many of the MAP students found the language of ELLI to be very useful when thinking about their strengths and weaknesses in the learning process. For example one MAP student remarks, “When you are thinking about it [selfreflection] it’s difficult to put into words what you mean ... it sort of puts the literature to what you are thinking” (MAP Focus 1, p. 5) However, some MAP students did have some difficulties in understanding what was being asked of them, in the first instance. A common initial reaction was “oh god ... it’s one of those horrible learning styles things ... you find out what kind of learner you are then the teacher won’t do anything about it. You’d be like oh you’re an aesthetic learner, great!” (MAP Ambassador 4, p.10) There has been much discussion around both the benefits and barriers that international students may face in working with ELLI. Again for students in transition ELLI provides a very useful way of helping international students to understand the differences between the education system in their home countries and what is expected of them from the system in the UK. However, staff also thought that perhaps the language of the questions may be inaccessible for students whose first language is not English. Whilst this is problematic it is also easily overcome through translation and/or guidance 18
Staff Perspectives Generally speaking whilst some staff appear to feel uncertain about ELLI and its model of dispositions to learn, there are others who are very enthusiastic, confident that ELLI can make a considerable contribution to tackling retention concerns. For many staff using ELLI has been a catalyst in the establishment of a structured place to have conversations about student centred learning through looking beyond the educational process to a focus on the needs of the student. For example: “ELLI has put a new slant on their past negative learning experiences.” (Practitioner View) “The positive aspect of using ELLI is that it introduces students into the understanding that they can redefine their learning by giving the process care and thought” (Practitioner view) Support and Sharing Ideas Having said this, and, as mentioned earlier, discussions indicate that that in order for ELLI to be effective there is a need for support and training to be given on the best means of delivery to students: “Staff need better trained on what the dimensions mean and what responses are appropriate to different profiles” (Practitioner view 2) In discussing how PCAP students could use ELLI alongside their teaching, one commentator explained that they had... “...looked at the definitions [of the seven dimensions] and thought hmmm, I would have liked a little bit more of an example, possibly because I wasn’t fully understanding it” (PCAP, p.3) This suggests that it is problematic to look abstractedly at the seven dimensions. Many of the enthusiastic staff who have used ELLI with their students this year have found very innovative ways of developing a working understanding of interpreting their profiles. For example: “The staff member should use their own profile or the profile of someone they know well to illustrate this topic” (Practitioner view 2) Difficulties
19
On a more technical note both staff and students have experienced some difficulties in, for example, logging-on and printing, an issue which will clearly need addressed in the coming years. Some staff found the spider diagram layout of the profiles slightly problematic in terms of instant recognition of what was being represented – some respondents remarked that perhaps a more linear arrangement would be better. Students also found this to be challenging. There was also some dubiousness on the manner in which the questionnaires were filled in. For example the production of a profile may well be skewed if the student lacks confidence or other outside/situational influences create difficulties. For some students it may well be beneficial to have ELLI guidance when filling in the questionnaire help with any such difficulties. Different Methods of Using ELLI ELLI has been used by staff across the partner institutions and beyond for a range of rationales, reflected in the ways ELLI has been used. Broadly speaking there have been three main methods of using ELLI: marginally integrated into the programme e.g. as part of a student support or guidance tutoring process; fully integrated into the programme; and as part of induction. These will be more fully investigated in the following years. The majority of respondents were given fairly brief introductions to ELLI before undertaking the questionnaire. After the questionnaire individuals were offered a more detailed discussion of the seven dimensions and of what their profiles signified. It was felt that these brief introductions were important as a more in depth discussion of the seven dimensions prior to undertaking the questionnaire may result in students feeling they should give the ‘right’ answer. Below are some ways that ELLI have been used with students this year and examples of possible future uses: 1. A fully integrated approach or as part of induction a. Before undertaking their very first assignments students will do their ELLI profile, have an explanation of the seven dimensions and a conversation about their relative strengths and weaknesses according to ELLI dispositions to learn b. Students are asked to carry out this assignment (and encouraged in others) with their ‘weaknesses’ in mind and are marked according to how well they have developed their ELLI ‘weaknesses’ c. For example if low on resiliency do the assignment on something you have found difficult in the past 20
2. Introduce students to ELLI by showing them a case studies (this could form part of next year’s analysis by picking examples of cross sections of ELLI profiles / different ways it was introduced / subject areas / types of conversations around ELLI and investigating the differences) a. “without seeing anyone else’s [profile] I didn’t know whether something quite close to a circle was actually good or not” (PCAPL, p. 2) b. “something that gives you an idea of being able to self-assess what’s actually being produced” (PCAPL, p. 2) c. “but to see that after you’ve filled out your questionnaire so that perhaps it doesn’t influence you beforehand” (PCAPL, p. 2) 3. Online tools to provide a better explanation of the purpose of ELLI, the nature of the dimensions and the range of suitable responses. One practitioner has suggested and is developing an ELLI game, where the student can build a profile by choosing different characteristics and considering the impact this has on the ELLI profile. By then choosing appropriate follow up activities students can learn how best to proceed with their own. It is hoped that this will also have a role in staff development (to train the tutors who are involved in supporting the small group seminar activity) and also to help students understand the ELLI concepts 4. A useful way of helping students to understand the different dimensions and to take ownership of their meanings is to ask students what they think each of the seven dimensions mean before having conversations about the definitions developed by ELLI 5. Some advice given is that there should be as little time as possible between completing the profile and having a conversation about it – whatever method that may take i.e. one-to-one with tutors / group discussions / peer discussions / role play Using ELLI as a Diagnostic Tool to Aid Retention Initiatives The following are some suggestions for possible interventions that were generated at the very successful ‘Dispositions to Stay’ event in Stevenage organised by Bedfordshire University: •
Recording – diary / portfolio to facilitate an understanding of positive changes
Reviewing previous work
21
Discussing ‘how’ to write reflectively – perhaps through informal study networks and conversations
Immediate feedback to boost confidence
Informal social events to incorporate strategies to overcome problematic relational barriers between students / tutors
Events that cut across barriers between disciplines / subject areas and national and international students
Modelling relationships through learning contracts to allow students to take more responsibility for their own learning
Online student-led support networks
More student placements
Starting career planning early
Effective messenger of services
3.5 Conclusions and Recommendations From Qualitative Data Whilst there have been some very positive reactions from staff across the three institutions, and beyond, there are some clear issues and challenges that need to be addressed and overcome for the next academic year. Engagement of •
Colleagues
Students
Management and heads of department
Practicalities •
When and how is ELLI best delivered?
When is the best time to introduce students to ELLI?
What forms of and/or how much administrative support needs to be provided?
What the long term prospects for access to ELLI beyond this research project?
Interpretation 22
•
There is a need for the sharing of ideas and support opportunities on the ways in which profiles can and should be interpreted
Staff need to be given opportunities to develop the skills that are necessary to deliver feedback with students
From the initial qualitative data there appears to be a great need for a staff guidebook for using ELLI with students given that most staff agree that for ELLI to have a positive impact, there needs to be a supportive explanation, both at the outset and when a student is presented with their profile. 3.6 Insights for the Grants Programme Meta-Analysis The qualitative and quantitative findings point to a number of issues that can inform learning across the wider sector. Most clearly, both sets of findings point to two attributes – critical curiosity and changing and learning – that institutions should be seeking to develop in students. Similarly, these attributes need to be developed in pre-university settings to ensure that students have skills that will be particularly important to them when they enter higher education. In terms of the curriculum, while many lecturing staff may already be seeking to encourage critical curiosity among students, institutions should review their methods of teaching, learning and assessment to ensure that they encourage development of this disposition. The second key disposition – changing and learning – can be easily linked to the use of ELLI itself. Students should be encouraged to reflect on the manner in which their learning power is growing; the use of ELLI, when introduced at the right time and in the right manner, can be one of the tools that facilitates such reflection. Conversely, while some academic programmes require students to be creative, the findings suggested that there is no need to seek to encourage the growth of this attribute among all students. An alternative interpretation would be that teaching and assessment practises should be reviewed to determine whether they are penalising creative students. In addition to the impact on the curriculum, the qualitative findings pointed to extra-curricula activities that might encourage students to consider their changing and learning such as learning contracts, more students placements and informal social events to break down barriers between students and their tutors. Suggesting extra-curricular activities to boost critical curiosity presents a greater challenge. At the level of the English higher education system, the results suggested that more attention should be paid to the way that institutions seek to develop their students as learners. English Higher Education Institutes were measured against performance indicators in six key areas in 2007-2008: widening participation of under-represented groups, widening participation among students who are in receipt of disabled students’ allowance, non-continuation 23
rates, module completion rates, research output and employment of graduates. (http://www.hesa.ac.uk/index.php/content/view/1446/141/). The findings suggested that at least two of these indicators – non-continuation rates and module completion rates – could be positively influenced by encouraging students to reflect on, and develop, the most important learning dispositions. The dissemination of these findings could be backed by guidance to institutions encouraging them to spend time with their students considering themselves as learners and pointing out the likely benefits in term of their academic success. However, an alternative interpretation of the findings is that institutions should not be assessed by non-continuation rates, as students may take a strategic and logical decision to withdraw. For as long as this is a performance indicator, and money is allocated to universities accordingly, there will be pressure to try to persuade students to remain on their academic programme, even when this may not be the option that best meets their long term interests. 4. Dissemination of learning Dissemination has begun at a very early stage of the project, rather than towards the end, reflecting the commitment of the team to an action research philosophy that leads to practical changes in teaching and learning. The project is intended to be a catalyst for innovation in teaching, learning and personal development; strategic support of the partner institutions has been given on this basis. Dissemination within Northumbria University in the first year has had the very specific objective of encouraging greater use of ELLI and facilitating discussion and debate about dispositions to learn. As a result, the research team have attended a large number of meetings of associate deans and teaching and learning committees. Presentations have also been made at two internal conferences: the Supporting Northumbria conference (for support staff) and the Programme and Subject Leaders’ Conference. The success of this approach is demonstrated by a commitment from all of the nine Schools in the university to use ELLI in the forthcoming year – a substantial achievement in itself. Further significant internal dissemination will take place at the Northumbria conference in September. Dissemination of good practice in the use of ELLI, and encouragement to staff across the partner institutions and beyond, was provided through the ELLI residential event programme organised by the University of Bedfordshire in March. The ideas discussed at this event have contributed to the data discussed in this report. In addition, staff widely reported that the event was encouraging and thought provoking in terms of the way that they used ELLI, providing many examples of different types of practice. Good practice is one theme of guidance materials which are being produced by Karen Williamson and Annika Coughlin. These materials will highlight some of the ways in which ELLI has been used, demonstrate the importance of critical curiosity and changing and learning, and suggest some approaches 24
to boosting these and other dimensions. The guidance will be distributed to all staff who are, or are considering, introducing students to ELLI. It will include guidance to students which staff may wish to distribute. Project staff are also contributing to placing the use of ELLI in the broader context of retention through participation in HEA events. Karen Williamson contributed to a workshop on the retention grant programme at the HEA conference, the team contributed a report to the second retention grants programme briefing, and Jamie Harding and Jamie Thompson will present a HEA seminar paper in December. Dissemination at two international conferences has also sought to increase the take up of ELLI, to stimulate debate about learning dispositions and to look for links between the use of ELLI and other factors that have a bearing on retention. The two conferences were the third international personal tutoring and academic advising conference and the European Learning Styles Information Network (ELSIN). These conference presentations led to a number of enquiries by institutions interested in the work and also highlighted other factors that could be measured alongside ELLI such as emotional intelligence. The team are considering whether there would be benefits to examining the relationship between these factors and ELLI. In addition, there has been international interest in the use of ELLI; as discussed below, the team are planning to give more attention to international comparisons in 20092010. Further dissemination has also been facilitated through the creation of a JISC mailing list and the creation of a Dispositions to Stay page on the Northumbria University website. 5. Plans and Challenges for 2009-2010 Data collection will be more substantial in 2009-2010. On the quantitative side, all students using ELLI in the participating institutions will automatically be asked whether they are willing for their data to be included in the research project. This is likely to mean that there are several thousand cases to analyse. The greater numbers will facilitate multivariate analysis, examining the impact of combinations of different learning dispositions. This may help to resolve the apparently contradictory findings in relation to meaning making in particular – a disposition that was closely associated by staff and students with critical curiosity and yet seemed to have little impact on student success. Learning relationships is another area where the data appeared contradictory and further investigation will be undertaken in 2009-2010. While much of the quantitative analysis in 2008-2009 necessarily focused on student achievement, as demonstrated by first year marks, next year there can be a clearer focus on retention across a number of areas of data collection: •
Information can be added to the dataset for the 378 students whose SITS and ELLI data has been combined to indicate whether they have enrolled for the second or third year. Some of the above analysis can 25
then be repeated with a clearer view of which students are likely to complete their programme. •
It is anticipated that several cohorts of students will complete ELLI profiles during induction. As the peak point for leaving a programme is during the first six weeks, collecting data during induction should ensure that there are more cases of students who do not complete the first year to compare to those students who do. Comparisons will be made at both the census point of 1st December and the end of the academic year.
•
Efforts will be made to interview students who have withdrawn from their academic programmes. While this is a notoriously difficult group to contact, it is hoped that even a very small number of interviews will provide important information about learning dispositions.
However, there will be some continuing work in the area of student achievement, with the second year mean marks being added to the data records of those students who participated in the research in 2008-2009. One aim of this continued work will be to determine whether the impact of students’ initial dispositions changes with time: in particular, whether initial high levels of creativity also disadvantage students in later years of their academic programme. Changes to students’ dispositions to learn, and the relationship between these and student success, will also be examined further as some students who have completed ELLI in 2008-2009 will complete another profile in 2009-2010. Qualitative data collection will also be more comprehensive and targeted. The guidance discussed in the previous section will be evaluated through interviews/focus groups conducted with staff and students who have used the appropriate materials. Questions asked will consider the helpfulness of the guidance as a whole, the most useful sections and any activities that were added to the suggestions made in the guide. Another element of qualitative data collection will focus on students selected because the data collected in 2008-2009 shows that they are either particularly ‘high’ or particularly ‘low’ in the key areas of critical curiosity and changing and learning. Analysis will seek to identify further differences in the dispositions of the ‘high’ and ‘low’ groups. The team also hope to make international and cultural comparisons of dispositions to learn a feature of their work in 2009-2010. Students who complete ELLI should form a more ethnically diverse group as a result of plans to include cohorts such as postgraduate students at the University of Bedfordshire and students of Middle Eastern Studies at the University of Manchester. Another group of interest will be Northumbria students who complete most of their programme in Grenada, but study at Northumbria for several months. Changes to their ELLI profiles will be compared to those of students on the same programme who remain in Grenada for the course of their studies. The team have also made HE contacts in South Africa, 26
Malaysia, India, Germany and Switzerland and will explore whether there is a possibility to make international comparisons in these countries. Students will play a greater role in shaping the project in 2009-2010. Two Northumbria students – one form education and one from social sciences – have agreed to take part in meetings to help to plan the work of the project. Student input will also play a key role in the planned conference at Northumbria University which is discussed below. At a practical level, discussions will take place with the University of Bristol about some of the difficulties arising from the use of ELLI in the 2008-2009 academic year, including the missing demographic data. Discussions will also focus on the formation of a sustainability strategy for the use of ELLI. 6. Dissemination Strategy For 2009-2010 Now that analysis has been undertaken of the key dispositions to learn, and work has been undertaken to develop guidance on the use of ELLI, the dissemination strategy will broaden in 2009-2010. There will be a focus on encouraging colleagues to use the new guidance material to introduce ELLI in a manner appropriate to their own subject discipline and to focus on the key dispositions. This encouragement will take the form of pre-induction briefings and ongoing training. Specific events at which dissemination is planned for the forthcoming year (some of which have been mentioned previously) are: • • • • • • •
A presentation and posters at the 2009 Northumbria conference in September. The Centre for Recording Achievement conference in November An HEA seminar, to be delivered in December, entitled Dispositions to Learn and Student Success The second retention summit on 20th January 2010 and the What Works? Student Retention and Success convention on 3rd – 4th March 2010 A workshop at the University of Manchester’s teaching and learning conference in May The HEA conference next summer At least one of the first year experience conferences (International First Year Experience or European First Year Experience)
The team will, of course, be keen to respond to further dissemination opportunities. They will also seek to publish articles in appropriate academic journals. An activity that will span data collection and dissemination is a conference at Northumbria University in the spring, to which student and staff representatives will be invited. It is hoped that students will write guidance as to how to boost each of the learning dimensions, particularly the key ones of critical curiosity and changing and learning, and that these ideas will be 27
presented to academic staff. Staff will consider this guidance alongside the findings from the main forms of data collection undertaken during the academic year, i.e. • • •
Qualitative findings about the use of ELLI and the existing good practice guidance Qualitative findings collected from students who were ‘high’ and ‘low’ in the areas of critical curiosity and changing and learning in relation to these two dimensions Quantitative findings on students who withdraw, making use particularly of the data relating to the 1st December cut off point
In the light of this material, conference delegates will be invited to contribute to the project’s strategy for the 2010-2011 academic year. 7. Outcomes The most significant outcomes of this year’s work are the triangulation of quantitative and qualitative data to identify the key dispositions to learn. In addition, many helpful suggestions as to how to encourage students to reflect on their dispositions to learn have been collated and disseminated. The project team require the same level and type of support from the Support and Co-ordination Team in 2009-2010 as they received in 2008-2009.
28