PISA 2012 Evaluating school systems to improve education
OECD EMPLOYER BRAND Playbook
Houston, 25 March 2014 Andreas Schleicher
1
Skills transform lives and drive economies
2
Increased likelihood of positive outcomes for adults with higher literacy skills (scoring at PIAAC Level 4/5 compared with those scoring at Level 1 or below) Odds ratio 3.0
2.5
2.0
1.5
1.0 Being Employed
High wages
Good to Participation in High levels of excellent health volunteer political efficacy activities
High levels of trust
Changes in the demand for skills
3
Trends in different tasks in occupations (United States) Mean task input in percentiles of 1960 task distribution
70
65
Routine manual
60
Nonroutine manual Routine cognitive
55
Nonroutine analytic Nonroutine interpersonal
50
45
40
35
1960
1970
1980
1990
2000
2006
2009
Source: Autor, David H. and Brendan M. Price. 2013. "The Changing Task Composition of the US Labor Market: An Update of Autor, Le vy, and Murnane (2003)." MIT Mimeograph, June.
4
PISA in brief • Over half a million students… – represen4ng 28 million 15-‐year-‐olds in 65 countries/economies
… took an interna4onally agreed 2-‐hour test… – Goes beyond tes4ng whether students can reproduce what they were taught… … to assess students’ capacity to extrapolate from what they know and crea4vely apply their knowledge in novel situa4ons – Mathema4cs, reading, science, problem-‐solving, financial literacy – Total of 390 minutes of assessment material
… and responded to ques4ons on… – their personal background, their schools and their engagement with learning and school
• Parents, principals and system leaders provided data on… – school policies, prac4ces, resources and ins4tu4onal factors that help explain performance differences .
5
Each year OECD countries spend 200bn$ on math educa?on in school
What do 15-‐year-‐olds know… …and what can they do with what they know? Mathema?cs (2012)
High mathematics performance Mean score … Shanghai-‐China performs above this line (613)
Average performance of 15-year-olds in Mathematics
580
Singapore
570
Chinese Taipei
560
Macao-China Japan Liechtenstein Switzerland
540 530
510 500 490 480 470
Fig I.2.13
Korea
550
520
Hong Kong-China
Poland Belgium Germany Austria Slovenia New Zealand Denmark France Czech Republic LuxembourgLatvia Portugal Spain Slovak RepublicUnited States Connec?cut Hungary
MassachuseJs
Florida
Israel
Netherlands Estonia Finland Canada Viet Nam Australia Ireland United Kingdom Iceland Norway Italy Russian Fed. Lithuania Sweden Croatia
460 450 440
Greece Romania
430 420
Chile
410 … 12 countries perform below this line
US
Serbia Turkey Bulgaria U.A.E. Kazakhstan Thailand Malaysia Mexico
Low mathematics performance
26% of American 15-‐year-‐olds do not reach PISA Level 2 (OECD average 23%, Shanghai 4%, Japan 11%, Canada 14%, Some es?mate
long-‐term economic cost to be US$72 trillion )
High mathematics performance Singapore Chinese Taipei
Hong Kong-China
Average performance of 15-year-olds in mathematics
Korea Macao-China Japan Liechtenstein Switzerland
Strong socio-economic impact on student performance
Poland Belgium Germany Austria Slovenia New Zealand Denmark France Czech Republic LuxembourgLatvia Portugal Spain Slovak RepublicUnited States Hungary Israel Greece Romania
Chile
Netherlands Estonia Finland Canada Viet Nam Australia Ireland United Kingdom Iceland Norway Italy Russian Fed. Lithuania Sweden Croatia
Serbia Turkey Bulgaria U.A.E. Kazakhstan Thailand Malaysia Mexico
Low mathematics performance
Socially equitable distribution of learning opportunities
2012 Singapore Hong Kong-China
Chinese Taipei Korea
Japan
Switzerland Poland
Liechtenstein Estonia
Netherlands
Canada Belgium Finland Viet Nam MassachuseJs Germany Strong socio-economic Austria Australia impact on student New Zealand Denmark Slovenia Ireland Iceland Czech Rep. performance 26 24 22France 20 18 16 14 12 10 8 6 UK Latvia Luxembourg Norway Portugal Italy Russian Fed. Connec?cut US Spain Lithuania Sweden Slovak Rep. Hungary Croatia Israel Florida
Romania Bulgaria
Chile
Greece Turkey
Macao-China
Serbia
United Arab Emirates Kazakhstan Thailand Malaysia Mexico
Socially equitable distribution of learning opportunities 4 2 0
Australia Austria Belgium Canada Chile Czech Rep. Denmark Estonia Finland France Germany Greece Hungary Iceland Ireland Israel socio-economic Strong Italy impact on student Japan performance Korea Luxembourg Mexico Slovak Rep. Netherlands New Zealand Norway Poland Portugal Slovak Rep. Slovenia Spain Sweden Switzerland Turkey UK US
2012
Korea
Switzerland Poland
Netherlands Belgium
Germany
Japan Estonia Canada Finland
Socially equitable
Austria Australia New Zealand Denmark Ireland distribution of learning Slovenia Iceland Czech Rep. opportunities France UK Luxembourg Norway Portugal Italy US Spain Sweden Hungary Israel Greece Turkey
Chile Mexico
Australia Austria Belgium Canada Chile Czech Rep. Denmark Estonia Finland France Germany Greece Hungary Iceland Ireland Israel Italy Japan Korea Luxembourg Mexico Slovak Rep. Netherlands New Zealand Norway Poland Portugal Slovak Rep. Slovenia Spain Sweden Switzerland Turkey UK US
Korea
Switzerland Poland
Netherlands Belgium
Germany
Japan Estonia Canada Finland
Austria Australia New Zealand Denmark Ireland Slovenia Iceland Czech Rep. France UK Luxembourg Norway Portugal Italy US Spain Sweden Hungary Israel Greece Turkey
Chile Mexico
Contribution of various factors to upper secondary teacher compensation costs, per student as a percentage of GDP per capita (2004)
Salary as % of GDP/capita
Instruction time
1/teaching time
1/class size
Difference with OECD average 15
Percentage points
10
5
0
-5
Slovak Republic
Poland
United States
Sweden
Finland
Mexico
Ireland
Iceland
Norway
Hungary
Czech Republic
Austria
Italy
Denmark
Netherlands
France
New Zealand
United Kingdom
Australia
Japan
Greece
Germany
Luxembourg
Korea
Belgium
Switzerland
Spain
Portugal
-10
Australia Austria Belgium Canada Chile Czech Rep. Denmark Estonia Finland France Germany Greece Hungary Iceland Ireland Israel Italy Japan Korea Luxembourg Mexico Slovak Rep. Netherlands New Zealand Norway Poland Portugal Slovak Rep. Slovenia Spain Sweden Switzerland Turkey UK US
Korea
Switzerland Poland
Netherlands Belgium
Germany
Japan Estonia Canada Finland
Austria Australia New Zealand Denmark Ireland Slovenia Iceland Czech Rep. France UK Luxembourg Norway Portugal Italy US Spain Sweden Hungary Israel Greece Turkey
Chile Mexico
Australia Austria Belgium Canada Chile Czech Rep. Denmark Estonia Finland France Germany Greece Hungary Iceland Ireland Israel Italy Japan Korea Luxembourg Mexico Slovak Rep. Netherlands New Zealand Norway Poland Portugal Slovak Rep. Slovenia Spain Sweden Switzerland Turkey UK US
Shanghai
2003 - 2012
Singapore Singapore
Switzerland Poland
Netherlands Belgium
Germany
Korea
Japan Estonia Canada Finland
Austria Australia New Zealand Denmark Ireland Slovenia Iceland Czech Rep. France UK Luxembourg Norway Portugal Italy US Spain Sweden Hungary Israel Greece Turkey
Chile Mexico
14
The American dream of social mobility In some countries it is close to a reality
Mexico Chile Greece Norway Sweden Iceland Israel Italy United States Spain Denmark Luxembourg Australia Ireland United Kingdom Hungary Canada Finland Austria Turkey Liechtenstein Czech Republic Estonia Portugal Slovenia Slovak Republic New Zealand Germany Netherlands France Switzerland Poland Belgium Japan Macao-‐China Hong Kong-‐China Korea Singapore Chinese Taipei Shanghai-‐China
300 325 350 375 400 425 450 475 500 525 550 575 600 625 650 675
15
Mathematics performance by decile of social background The children of room cleaners in Shanghai outperform the children of professionals in the US
High impact on outcomes
16 16 Lessons from high performers
Must haves
Quick wins
Catching up with the top-‐performers Low feasibility
High feasibility
Money pits
Low hanging fruits
High impact on outcomes
17 17
Quick wins
Must haves
Lessons from high performers
Commitment to universal achievement Capacity at point of delivery
Resources where they yield most Gateways, instruc?onal systems
Coherence Low feasibility
A learning system High feasibility
Incen?ve structures and accountability
Money pits
Low hanging fruits
High impact on outcomes
18 18 Lessons from high performers
❒
Quick wins Must htaves A commitment o educa?on and the belief that Commitment to universal competencies can be learned and atchievement herefore a ll children can achieve Capacity
educa?onal standards and Resources personaliza?on as at Universal point of delivery yield most the approach to heterogeneity where in the tshey tudent body… … as opposed to a belief that students have different Gateways, instruc?onal des?na?ons to be met with different expecta?ons, and systems selec?on/stra?fica?on as the approach to Coherence heterogeneity A learning system l Clear ar?cula?on who is responsible for ensuring Low feasibility High feasibility student success and to whom l
Incen?ve structures and accountability
Money pits
Low hanging fruits
19
Countries where students have stronger beliefs in their abilities perform better in mathematics
Fig III.4.5
OECD average
650
Mean mathematics performance
600
550
500
450
400
350
300 -0.60
Shanghai-China
Singapore Hong Kong-China Korea R² = 0.36 Chinese Taipei Macao-China Japan Switzerland Netherlands Estonia Canada Liechtenstein Finland Germany Poland Belgium Viet Nam Slovenia Denmark New Zealand Latvia Portugal Italy Austria Australia Russian Fed. Hungary Luxembourg Croatia Slovak RepublicSpain Greece Israel Norway Turkey Sweden Serbia Czech Republic Lithuania U.A.E. Iceland Romania United Kingdom Thailand Malaysia United States Ireland Bulgaria Kazakhstan Chile Montenegro France Costa Rica Mexico Uruguay Albania Brazil Argentina Tunisia Colombia Qatar Jordan Indonesia Peru
-0.40
-0.20
0.00
0.20
0.40
0.60
Mean index of mathematics self-efficacy
0.80
1.00
1.20
20
Perceived self-responsibility for failure in mathematics
Fig III.3.6
Percentage of students who reported "agree" or "strongly agree" with the following statements: France
Shanghai-China
OECD average
Sometimes I am just unlucky The teacher did not get students interested in the material Sometimes the course material is too hard
This week I made bad guesses on the quiz My teacher did not explain the concepts well this week I’m not very good at solving mathematics problems 0
US
20
40
60
%
80
100
21
The parent factor Students whose parents have high educa4onal expecta4ons for them tend to report more perseverance, greater intrinsic mo?va?on to learn mathema?cs, and more confidence in their own ability to solve mathema?cs problems than students of similar background and academic performance, whose parents hold less ambi?ous expecta?ons for them.
Parents’ high expectations can nurture students’ enjoyment in learning mathematics
22
Fig III.6.11
Change in the index of intrinsic motivation to learn mathematics that is associated with parents expecting the child to complete a university degree
0.50 0.45
0.35 0.30 0.25 0.20 0.15 0.10 0.05
Germany
Mexico
Macao-China
Croatia
Hungary
Portugal
Chile
Hong Kong-China
Italy
Korea
0.00
Belgium (Flemish)
Mean index change
0.40
High impact on outcomes
23 23
Quick wins
Must haves
Lessons from high performers
Commitment to universal achievement ❒
Clear ambi?ous goals that are shared across the
Capacity system and aligned wResources ith high s takes gateways and at point of delivery where they yield most instruc?onal systems l
Coherence Low feasibility
l
Well established delivery chain through instruc?onal which Gateways, curricular goals translate into instruc?onal systemss ystems, instruc?onal prac?ces and student learning (intended, implemented and achieved) A learning system High level of metacogni?ve content of instruc?on … High feasibility
Incen?ve structures and accountability
Money pits
Low hanging fruits
High impact on outcomes
24
❒ 24Capacity at the point of delivery Must haves
Lessons from high performers
l
l
l l
Quick wins
AJrac?ng, developing and retaining high quality Commitment universal achievement teachers and school leaders and tao w ork organisa?on in which they can use their poten?al Capacity Instruc?onal leadership and human resource Resources at point of delivery management in schools where they yield most Keeping teaching an aJrac?ve profession Gateways, instruc?onal System-‐wide career development … systems Coherence A learning system
Low feasibility
High feasibility
Incen?ve structures and accountability
Money pits
Low hanging fruits
High impact on outcomes
25 25 Lessons from high performers
❒
Must haves
Quick wins
Incen?ves, accountability, knowledge management
Commitment to universal achievement Aligned incen?ve structures For students Capacity Resources gateways clarity and nature of the incen?ves at point oHow f delivery affect the strength, direc?on, opera?ng on students at each where stage of tthey heir eyduca?on ield most l
l
l l
Degree to which students have incen?ves to take tough courses and study hard Gateways, Opportunity costs for staying in school and performing well instruc?onal
systems For teachers Coherence Make i nnova?ons in pedagogy and/or organisa?on A learning system l l
Low feasibility
l l
l
l
Improve their own performance and the performance of their colleagues Pursue professional development opportuni?es that lead to stronger pedagogical prac?ces
High feasibility
Incen?ve structures and A balance between ver?cal and lateral accountability accountability Effec?ve instruments to manage and share knowledge and spread innova?on – communica?on within the system and with stakeholders around it Money pits Low A capable centre with authority and legi?macy to hanging act fruits
26 26
Countries that grant schools autonomy over curricula and assessments tend to perform be`er in mathema4cs
Lessons from high performers
650
Fig IV.1.15
Shanghai-China
Mathematics performance (score points)
600
550
500
450
400
Chinese Taipei
Viet Nam
Korea
Estonia
Singapore
Hong Kong-China Japan
Poland Latvia Slovenia Czech Rep. Belgium Switzerland Canada Germany Finland New Zealand Lithuania Portugal Hungary Austria Croatia Italy Spain Serbia France Australia Macao-China Turkey Norway Iceland Denmark Slovak Rep. Bulgaria Greece Romania Kazakhstan Israel Malaysia Chile Uruguay USA Sweden Jordan Costa Rica Indonesia Brazil Luxembourg Tunisia Albania Colombia UAE Argentina Peru
Netherlands UK Thailand R² = 0.13
350
Qatar 300 -1.5
-1
-0.5
0
0.5
Index of school responsibility for curriculum and assessment (index points)
1
1.5
Schools with more autonomy perform better than schools with less autonomy in systems with standardised math policies
Fig IV.1.16
School autonomy for curriculum and assessment x system's extent of implementing a standardised math policy (e.g. curriculum and instructional materials)
Score points 485
480
475
470
465 460
Standardised math policy
455
No standardised math policy
Less school autonomy More school autonomy
Schools with more autonomy perform better than schools with less autonomy in systems with more collaboration
School autonomy for resource allocation x System's level of teachers participating in school management Across all participating countries and economies Score points 485
480
475
470
465
460
Teachers participate in management
455
Teachers don't participate in management
Less school autonomy More school autonomy
Fig IV.1.17
Schools with more autonomy perform better than schools with less autonomy in systems with more accountability arrangements
Fig IV.1.16
School autonomy for curriculum and assessment x system's level of posting achievement data publicly
Score points 478 476 474 472 470 468 466
School data public
464
School data not public
Less school autonomy More school autonomy
30
Quality assurance and school improvement
Fig IV.4.14
Percentage of students in schools whose principal reported that their schools have the following for quality assurance and improvement: Singapore
OECD average
Implementation of a standardised policy for mathematics Regular consultation with one or more experts over a period of at least six months with the aim of improving Teacher mentoring Written feedback from students (e.g. regarding lessons, teachers or resources) External evaluation Internal evaluation/self-evaluation Systematic recording of data, including teacher and student attendance and graduation rates, test results and Written specification of student-performance standards Written specification of the school's curriculum and educational goals 0
20
40
%
60
80
100
100
-50
Chinese Taipei Hong Kong-China Thailand Viet Nam Luxembourg Switzerland Indonesia Italy Kazakhstan Japan Czech Republic Netherlands Estonia Albania Ireland United States Hungary Sweden Korea United Kingdom Finland Denmark OECD average France Shanghai-China Australia Spain Slovak Republic Mexico Germany Austria Colombia Chile Canada Poland Jordan Argentina United Arab Emirates Portugal Peru Costa Rica Brazil New Zealand Malaysia Slovenia Uruguay Qatar
Score-point difference
Differences in mathematics performance between private and public schools shrink considerably after accounting for socio-economic status
50
Fig IV.1.19
Observed performance difference
After accounting for students’ and schools’ socio-economic status
75
Performance advantage of public schools
25
0
-25
Performance advantage of private schools
-75
-100
-125
School competition and mathematics performance
Fig IV.1.18
Adjusted by per capita GDP 650
Shanghai-China
There is no relationship between the prevalence of competition and overall performance level
Mathematics performance (score points)
600
Viet Nam
Korea
550 Estonia Germany Slovenia Portugal
Poland
Switzerland
Finland
500 Norway
Iceland
450
Montenegro
400
R² = 0.030
Japan
Chinese Taipei Hong Kong-China
Netherlands Latvia Singapore Czech Rep. Belgium Lithuania New Zealand Slovak Rep. Spain Italy France Serbia Macao-China Ireland Hungary Romania Austria UK Bulgaria USA Sweden Australia Turkey Thailand Greece Chile Uruguay Kazakhstan Malaysia Jordan Costa Rica Mexico Argentina Albania Brazil Tunisia Indonesia UAE Luxembourg Colombia Peru
350
Qatar 300 30
40
50
60
70
80
Percentage of students in schools that compete with at least one other school
90
100
High impact on outcomes
33 33 Lessons from high performers
Must haves
Quick wins
to universal achievement ❒ Commitment Inves?ng resources where they can make most
of a difference
Capacity Resources l Alignment of resources with key c hallenges (e.g. at point of delivery where they ytield mostt o the most aJrac?ng the most talented eachers challenging classrooms) Gateways, instruc?onal l Effec?ve spending choices that priori?se high quality systems teachers over smaller classes Coherence A learning system Low feasibility
High feasibility
Incen?ve structures and accountability
Money pits
Low hanging fruits
Countries with better performance in mathematics tend to allocate educational resources more equitably 700
Adjusted by per capita GDP
650
Mathematics performance (score points)
Fig IV.1.11
30% of the varia?on in math performance across OECD countries is 600 explained by the degree of similarity of educa?onal resources between 550advantaged and disadvantaged schools 500
450
Mexico Costa Rica
400
Shanghai-China
Chinese Taipei Korea R² = 0.19 Viet Nam Singapore Hong Kong-China Estonia Japan Poland Slovenia Switzerland Latvia Finland Belgium Canada Germany Macao-China Slovak Rep. New Zealand IrelandIceland France Austria UK Spain Denmark Australia Israel Croatia Hungary Romania Sweden Bulgaria Portugal USA Turkey Greece Norway Italy Serbia Thailand Malaysia Chile Uruguay Kazakhstan Jordan Brazil Indonesia UAE Montenegro Colombia Tunisia Argentina Luxembourg
Peru 350
Qatar 300 1.5
1
Less equity
0.5
OECD countries tend to allocate at least an equal, if not a larger, number of teachers per student to disadvantaged schools; but disadvantaged schools tend to have great difficulty in attracting -0.5 qualified teachers.0
Equity in resource allocation (index points)
Greater equity
High impact on outcomes
35 35
Quick wins
Must haves
Lessons from high performers
Commitment to universal achievement Capacity at point of delivery ❒
A learning system l
Coherence l
Low feasibility
Resources where they yield most
Gateways, instruc?onal An outward orienta?on to keep the system learning, technology, interna?onal benchmarks systems as the ‘ eyes’ and ‘ears’ of the system A learning system Recognising challenges and poten?al future threats to current success, learning from them, designing High feasibility responses and implemen?ng these Incen?ve structures and accountability
Money pits
Low hanging fruits
High impact on outcomes
36 36
Quick wins
Must haves
Lessons from high performers
Commitment to universal achievement
❒
Capacity at point of delivery
Resources where they yield most
Coherence of policies and prac?ces Alignment of policies across all aspects of the system Coherence l Coherence of policies over sustained periods of ?me Low feasibility l Consistency of implementa?on l Fidelity of implementa?on (without excessive control) l
Money pits
Gateways, instruc?onal systems A learning system High feasibility
Incen?ve structures and accountability
Low hanging fruits
High impact on outcomes
37 37
Quick wins
Must haves
Lessons from high performers
Commitment to universal achievement Capacity at point of delivery
Resources where they yield most Gateways, instruc?onal systems
Coherence Low feasibility
A learning system High feasibility
Incen?ve structures and accountability
Money pits
Low hanging fruits
Find out more about PISA at www.pisa.oecd.org • All na?onal and interna?onal publica?ons • The complete micro-‐level database
Thank you !
Email: Andreas.Schleicher@OECD.org TwiJer: SchleicherEDU
and remember: Without data, you are just another person with an opinion