PISA VET brochure

and Training Assessment

and the Centre for Skills

Directorate for Education and Skills and the Centre for Skills

International Vocational Education and Training Assessment

Foreword

The OECD’s Programme for International Student Assessment (PISA) is the world’s premier yardstick for comparing quality, equity and efficiency in learning outcomes of 15-year-old students across countries. The OECD is now launching the development of a PISA-inspired assessment for vocational education and training. This brochure sets out the occupational areas and employability skills as well as the target population proposed for this new assessment. The brochure also presents the timeline for the Development Phase of the project. This initiative builds on a range of national and industry-specific efforts to measure applied skills, but it would be the first time to apply these methods at scale and across countries. The initiative has the potential to pay huge dividends in terms of improving the labour-market relevance, quality and social status of vocational education and training. To achieve these benefits, the OECD, international experts, and participating countries must confirm the validity of the metrics and the international comparability of results that will come from this assessment. To address this, the OECD has established a step-by-step approach to the development of the assessment framework and the instruments with multiple opportunities for countries to review progress and decide on subsequent work.

Andreas Schleicher OECD Director of Education and Skills

International VET Assessment Development Phase

The International Vocational Education and Training (VET) Assessment project supports countries in strengthening their skills systems by developing, piloting and implementing an internationally standardised assessment of outcomes of initial VET programmes, the analysis of the results of which will enhance comparative policy insights on how to improve the relevance, quality, equity and effectiveness of initial VET programmes.

The scope of the assessment

What the initiative is aiming to achieve

The PISA-VET assessment project supports countries in strengthening their skills systems by developing, piloting and implementing an internationally standardised assessment of outcomes of initial vocational education and training (VET) programmes, the analysis of the results of which will enhance comparative policy insights on how to improve the relevance, quality, equity and effectiveness of initial VET programmes.

The initiative will do this by delivering a comprehensive dataset on VET learning outcomes and the factors related to these outcomes. This dataset will go beyond participation and employment rates and provide internationally comparable data on the performance of VET systems in key occupational areas. The initiative will also facilitate peer learning opportunities between countries and help to raise the image and promote the profile of VET.

The PISA-VET assessment will provide policy insights on a range of outcomes and related factors, including:

• comparative skill levels of students across countries and programmes

• differences between countries and programmes

• how to account for differences in programme design and learner background

• differences between learners and programme types

Five occupational areas

Five occupational areas have been selected as the focus of the assessments Development Phase, as follows:

• Business and Administration

• Electrician Craft /Industry

• Health Care, with a focus on Healthcare/nursing Assistants

• Automotive technician

• Tourism and Hospitality, with a focus on Hotel Receptionists

These occupational areas have been chosen for the assessment because each of them comprise a student population that is large enough in most participating countries to come to reliable and valid judgments on VET programmes that are important for the labour market.

These occupational areas are each of significant economic and social importance now and are likely to remain so well into the future. Moreover, the areas demand mainly professional skills that are sufficiently comparable at the international level and are linked to VET programmes that are comparable across countries.

In addition, there are sufficient resources available to support measurement of competences within these occupational areas.

This mix of occupational areas achieves a balance between those occupational areas requiring mostly STEM-related competences and those requiring mostly competences outside of STEM fields. Gender dimensions are a crucial feature of the assessment and, taken together, the five occupational areas selected for the Development Phase represent an acceptable gender balance in terms of student population.

Two of the occupational areas (electrician and automotive technician) are typically male dominated, one of the occupational areas (health care) is female dominated and two (business and administration and tourism and hospitality) represents both male and female learners by and large equally. The assessment at the same time promotes a less gender-stereotyped view of the occupational areas and associated VET programmes.

The international assessment prioritises the production of comparable measures of students’ procedural knowledge, practice-oriented skills and other professional skills in the specific occupational areas. The main focus of the assessment is thus the professional competence to act in practice and in the context of real work situations. The assessment of the competences specific to each of the five occupational areas may use three different types of instruments:

1. Scenario-based questions of technical and practical knowledge;

2. Digital simulations of problem-solving at the workplace ; and

3. Live demonstrations with expert judges.

The Development Phase of the initiative will determine which type of instrument is most suitable for the measurement of the different competences.

Example of scenario and test item from the ACT Workkeys assessment of applied technology

The band saw where you work will not start. This saw uses 240 volts, draws 25 amps, and has 30-amp cartridge fuses. These fuses (see diagram shown) are designed to protect an electrical circuit. Their main component is a fuse wire made of a low-resistance, low melting-point alloy. When a higher than tolerable current goes through such a fuse, this fuse wire melts. Your supervisor has told you to check the fuses in the band saw. By looking at the fuses, you cannot tell if they are good or bad.

You have turned off the power to the saw and removed one of the fuses. You check this fuse with a volt-ohmmeter (a device that measures resistance to the flow of electrical current).

If the fuse is good, the resistance (measured in ohms) for the fuse will be:

A. 0 ohms.

B. 10 ohms.

C. 50 ohms.

D. 100 ohms.

E. infinite.

Source: ACT Applied Technology Assessment: https://www.act.org/content/act/en/products-and-services/workkeys-foremployers/assessments/applied-technology.html

Employability Skills

Success in the world of work requires transversal competences in addition to job-specific knowledge and skills. In a changing world of work, employers increasingly seek employees who adapt easily and are able to apply and transfer their skills and knowledge to new occupational contexts. It is thus important to supplement the assessment of occupation-specific skills with information on a broader set of cognitive and social and emotional competences.

Four employability skills have been selected to be part of the assessment, based on their relevance for the labour market and lifelong learning and the availability of test instruments/frameworks. While certain of these employability skills are important across the occupational areas, others are more relevant and potentially occupationspecific in some areas than in others.

» Task performance as defined by the OECD’s Survey of Social and Emotional Skills (SES) includes a range of constructs that describe the propensity to be selfcontrolled, responsible to others, hardworking, motivated to achieve, honest, orderly, persistent and rule abiding. It includes the subdomains of self-control, responsibility and persistence.

» Literacy, as defined by the OECD’s Survey of Adult Skills (PIAAC), is the ability to understand and use information from written texts in a variety of contexts to achieve goals and develop knowledge and potential. This is a core requirement for developing higher-order skills and for positive economic and social outcomes. Previous studies have shown reading literacy to be closely linked to positive outcomes at work, to social participation, and to lifelong learning.

» Problem solving, as included in the OECD’s PIAAC as “Problem solving in technology rich environments”, and/or “Adaptive problem-solving”, as well as the OECD PISA’s “Creative problem-solving”. The former refers to the ability to use technology to solve problems and accomplish complex tasks. It is not a measurement of “computer literacy”, but rather of the cognitive skills required in the information age – an age in which the accessibility of boundless information has made it essential for people to be able to decide what information they need, to evaluate it critically, and to use it to solve problems. In the PIAAC survey, higherorder skills are identified along with basic proficiency.

» Collaboration as defined by the OECD’s Survey of SES, is the capacity to collaborate successfully with others and to do so by maintaining positive relations and minimising interpersonal conflict. Showing active emotional concern for others’ well-being, treating others well and holding positive generalised beliefs about others are all examples of collaboration. The focus on these “core” employability skills does not preclude the possibility of assessing other skills that are transversal in nature but show different complexity, characteristics and shape from a certain occupational area to another (e.g. communication in healthcare, numeracy in motor vehicle mechatronics etc).

Target Population

The target population for the assessment are students of initial formal VET programmes in the five occupational areas at the upper-secondary to short-cycle tertiary education levels (ISCED Levels 3, 4 and 5). In particular, VET programmes will be selected based on their comparability in terms of learning outcomes (levelled across levels 3-5 of the European Qualifications Framework (EQF) or equivalent for those countries not using EQF). This includes students in school-based programmes and in programmes that combine school-based and work-based learning (e.g. apprenticeships).

» The students will be assessed in the last six months of their programmes.

» The focus of the Development Phase of this assessment will be on learners in formal initial VET programmes in the five proposed occupational areas that have comparable learning outcomes, regardless of ISCED levels.

» The precise target population for each occupational area will be narrowed down during the development phase. At that stage, the focus will be on a set of VET programmes or qualifications within the selected occupational areas that are comparable in terms of learning outcomes.

» In recognition that for some systems adults are a significant and increasing proportion of the VET student population, adult learners returning to VET after beginning working life, may be included in the assessment, but they will be kept separate for analytical purposes – given their different educational backgrounds and work experiences.

Contextual data

The competences that students demonstrate through their performance on the assessment can be significantly influenced by multiple factors, including the family background of the student and their formal and informal education opportunities as well as institutional and system level factors. It is important to collect information on these contextual factors in order to correctly interpret differences across groups of students or countries. This is of particular relevance in the VET sector, given the substantial differences in how VET is organised and delivered between and even within countries.

The assessment will collect data on these contextual factors through the following instruments:

» Student questionnaire – each student that takes the test will also complete a background questionnaire

» Teacher questionnaire – all of the teachers responsible for the selected occupational areas in the sampled institution will complete a background questionnaire

» Institution questionnaire – the head of or administrator for each sampled institution will complete a background questionnaire

» Employer/trainer questionnaire - trainers in the work-based learning environment that are most knowledgeable about the participating learner’s training will be invited to complete a one-page questionnaire (only in countries and programmes that have a substantial work-based component).

» System level data questionnaire – each participating country will provide system level data on its VET system.

The assessment framework will outline the set of constructs to be included in these questionnaires and the modules that will form the basis of the assessment. These modules are organised around four distinct pillars:

» Learner success outcomes

» Foundations for successful VET programmes

» Demographic factors for assessing equality and equity

» Context factors

The assessment framework

Learning outcomes and foundations for success

The framework will outline the assessment and analytical modules that will form the basis of the assessment. These modules are organised around three pillars.

1. VET programme/learning outcomes

1.1 Job-specific learning outcomes (measured through the International VET Assessment)

1.2 Employability skills performance (measured through the International VET assessment)

1.3 Health and well-being

1.4 Educational attainment

1.5 Attitudes towards VET, work and learning

1.6 Work safety

1.7 Ethics (in the workplace)

Framework Assessment and analytical modules

2. Foundations for Success

2.1 Type of VET programme

2.2 Inclusive environments

2.3 Quality instruction

2.4 Learning time

2.5 Work-based learning (WBL) time

2.6 Material resources

2.7 Human resources

2.8 Family and community support

2.9 Social partner support

2.10 Labour market support and school interactions/exchanges with labour market

3. Demographic factors for assessing equality and equity

3.1 Age

3.2 Family structure and living arrangements

3.3 Previous educational/work experience

Gender

Socio-economic status

Language spoken at home and language of instruction

Urban/rural status

Immigrant status

Figure 1.

Activities

During 2023, the OECD has worked with experts nominated by participating countries together with other specialists to develop the draft framework for the assessment. The framework publication, launched on 19 March 2024, makes clear how the assessment is split between the core occupational areas, the employability skills and the contextual data. In particular, the framework:

» Defines the domains that the assessment targets;

» Outlines and defines the set of competences students are tested on;

» Provides specifications that orient the task design;

» Indicates appropriate data analysis and reporting approaches, including the description of different proficiency levels along a proficiency scale.

This initiative builds on a range of national and industryspecific efforts to measure applied skills, but it would be the first time to apply these methods at scale and across countries. The initiative has the potential to pay huge dividends in terms of improving the labour-market relevance, quality and social status of vocational education and training. To achieve these benefits, the OECD,

international experts, and participating countries must confirm the validity of the metrics and the international comparability of results that will come from this assessment.

To address this, the OECD has established a step-by-step approach to the development of the assessment framework and the instruments with multiple opportunities for countries to review progress and decide on subsequent work, and for industry experts to provide inputs.

Following the approval of the framework by the OECD’s responsible education committees (the Education Policy Committee and the PISA Governing Board) in November 2023, work has commenced on the development of the cognitive instruments for data collection.

The first tests will be administered in a handful of countries through a methodological study, the analysis of which will determine the extent to which the instruments and approaches provide valid metrics and internationally comparable results. The results of the Development Phase will be presented in a report in 2026 which also includes the plan for a Pilot Phase, the next stage in the development of the initiative.

Pilot Phase

Moving towards an international large scale assessment of VET

Subject to the agreement of the OECD’s education committees and following the successful completion of the Development Phase and the recruitment of a minimum number of countries that can commit resources to undertake further instrument development and data collection, the next phase, i.e. the Pilot Phase, would be launched in 2026 (taking approximately four years to complete).

This Pilot Phase would include the further development and validation of the assessment instruments trialled during the Development Phase, the implementation of the data collection in participating countries, the analysis of the data and the preparation of an international report.

After a successful Pilot Phase, the initiative could enter its third phase, the Large-Scale Implementation Phase – i.e. running a fully-fledged international large-scale assessment of skills of learners in VET programmes.

Timeline for the Development Phase of the project

What the OECD does

The Organisation for Economic Co-operation and Development (OECD) is an international organisation that works to build better policies for better lives. Our goal is to shape policies that foster prosperity, equality, opportunity and well-being for all. We draw on 60 years of experience and insights to better prepare the world of tomorrow. Together with governments, policy makers and citizens, we work on establishing evidence-based international standards and finding solutions to a range of social, economic and environmental challenges. From improving economic performance and creating jobs to fostering strong education and fighting international tax evasion, we provide a unique forum and knowledge hub for data and analysis, exchange of experiences, best-practice sharing, and advice on public policies and international standard-setting.

Education and Skills at the OECD

The OECD Directorate for Education and Skills seeks to help individuals and nations to identify and develop the knowledge, skills and values that drive better jobs and better lives, generate prosperity and promote social inclusion. It assists OECD countries and partner economies in designing and managing their education and skills systems, and in implementing reforms, so that citizens can develop the knowledge, skills, attitudes, and values they need throughout their lives.The OECD Directorate for Education and Skills focuses on helping countries to identify and develop the knowledge and skills that drive better jobs and better lives, generate prosperity and promote social inclusion, and accompanies them in the difficult process of policy implementation.

The OECD Centre for Skills supports countries to achieve better economic and social outcomes by taking a whole-of-government approach and engaging with stakeholders to develop and implement better skills policies. One of the key focus areas of the OECD Centre for Skills is vocational education and training (VET), contributing to strenghening VET systems around the world based on solid international data and policy insights.

The opinions expressed and arguments employed herein do not necessarily reflect the official views of OECD member countries.

This document, as well as any data and map included herein, are without prejudice to the status of or sovereignty over any territory, to the delimitation of international frontiers and boundaries and to the name of any territory, city or area.

© OECD 2024

Photo credits:

Cover: © Jacob Lund/Shutterstock; © Monkey Business Images/Shutterstock; © industryviews/Shutterstock; © NassornSnitwong/Shutterstock

Inside: © Jacob Lund/Shutterstock; © industryviews/Shutterstock; © alphaspirit.it/Shutterstock; © Tapati Rinchumrus/Shutterstock; © Monkey Business Images/Shutterstock; © Pixel-Shot/Shutterstock; © Kamil Macniak/Shutterstock

You can copy, download or print OECD content for your own use, and you can include excerpts from OECD publications, databases and multimedia products in your own documents, presentations, blogs, websites and teaching materials, provided that suitable acknowledgment of OECD as source and copyright owner is given. All requests for commercial use and translation rights should be submitted to rights@oecd.org.