TW Report

Understanding Proxy Metrics

Measuringdisadvantage

There are many ways of understanding and quantifying disadvantage The simple splits between state and independent, selective and non-selective have never been sufficient to encompass the spectrum of multiple disadvantages that pupils face. Indeed, even on a more granular level, there is not a single metric that encapsulates this spectrum - hence the emergence of composite contextual bands that triangulate between multiple metrics.

The Access Office records participant data on the individual, school and postcode level. This gives a range of proxy data to understand the disadvantages faced by pupils. Due to the project's scope, it has focussed in particular on POLAR4.

POLAR4

POLAR4 is a nationally calculated geographical metric that assesses the lik

POLAR4: The National Picture

Q5 students outnumber Q1 students at university by a 3:1 margin.

The gap between Q5 students and all other quintiles becomes much more pronounced in top half institutions.

55% of Q1 students who go to university attend a top half institution, as opposed to 72% of Q5 students.

This graph tracks 'placed applicants’ (UCAS data) across the five quintiles in universities.

Unsurprisingly, students from less advantaged backgrounds make up a lower proportion of the student body at higher ranked universities. Only Q5 students make up greater proportions of the student body at higher ranked institutions than lower ranked institutions.

This graph displays this data on a trend level rather than individual institutions.

Programme Impact: POLAR4 Quintile 1

78%

of tracked students on a St John's programme attend a top third institution, with 10% attending one within the top three.

On average, only 38%of Q1 students who go to university will study at a top third institution.

Understanding this analysis

Howdoesthegraphwork?

On the X axis are university rankings. The higher-ranked universities (Guardian League Tables 2017-20) are further right.

On the Y axis is the cumulative% of students who attend a university of this ranking or less, from Q1 based on the national data

Each dot represents a university.

The red flag signifies the top third of universities; every dot to the right of this flag is a top third university. This is a group which has significant overlap with the Russell Group, but includes competitive, high-tariff institutions such as St Andrew's that are not otherwise included

Whatdothetwoshapesshow?

The upper row of dots represents national data available from UCAS. Red dots are universities at which students who have participated in St John's access initiatives have gone on to study. They are simply flagged but still use the national data for their position This works as a loose control

The lower, concave shape dots represents Inspire data. This represents the same universities as the upper row of red dots, but using the inspire data rather than national data.

Across the national average, only 38% of Q1 students who go to university will study at a top third institution. This is represented by the upper row of dots.

However, 78% of Q1 students who engage with St John’s access programmes attend a top third institution, with 10% attending one within the top three. This is represented by the lower row of dots

We can see this through the point at which each line intersects the red flag.

Programme Impact: POLAR4 Q2-5

Understanding this analysis

ThesegraphsapplythesameanalysistotheremainingPOLAR4Quintiles

There is a strong correlation between engaging with St John’s access programmes and attending significantly higher-ranked universities than peers from the same POLAR4 quintile.

The same curve is echoed across all five groups, which shows that the impact of the activities is not unique to students of any one background.

Whatelsecanwefindout?

These graphs also demonstrate that a high proportion of participants go on to study at the country’s most elite institutions. AfifthofparticipantsfromQ2-Q5 backgroundsgoontostudyatauniversityrankedwithinthetopthree(Oxford, Cambridge or St Andrews in these years).

This analysis also shows that there is a high floor for student performance, with over 80% of students attending a top third institution across Q2-Q5.

What does this mean?

Proving causality is challenging, as it would rely on both complete base data and the ability to conduct randomised control trials. This standard of evidence poses questions of feasibility and of ethics.

However, it is clear that ambitious and talented students (regardless of social background) were keen to get involved in the College’s programmes, and that those who did outperformed their national peers.

Whilst St John’s Access initiatives are not the only reason that programme participants significantly outperform their peers, it is reasonable to conclude that their work improves students’ outcomes and encourages them to consider elite universities as an achievable goal.