Ngā Uri Whakatupu: An independent evaluation

Ngā Uri Whakatupu

Uri Whakatupu: An independent evaluation report

ALL RIGHTS RESERVED. Any unauthorised copy, reprint or use of this material is prohibited. No part of this content may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage and retrieval system without express written permission from Te Pūtea Whakatupu Trust.

Professional Disclosure Statement

AIKO Consultants have prepared this report based on the research and information available at the date of publication and has been obtained from and is based on sources believed to be reliable and up to date. No responsibility will be accepted for any error of fact or opinion based on such reliance. This report was prepared by AIKO Consultants for the use and benefit of our client for the purpose for which it was provided.

ISBN: 978-0-473-65910-3 (Print) ISBN: 978-0-473-65911-0 (Online)

Publisher: Te Pūtea Whakatupu Trust Wellington, New Zealand December 2022

Ngā Uri Whakatupu: An independent summative evaluation of Te Reo Matatini me te Pāngarau and Te Kete Aronui

Dr Chelsea Grootveld AIKO Consulting Limited

Ngā Uri Whakatupu

An independent evaluation of Te Reo Matatini me te Pāngarau & Te Kete Aronui literacy and numeracy programmes

HE MIHI

Kei te mihi atu ki ngā tamariki mokopuna i uru atu ki tēnei hōtaka whakahirahira kia whakapakari i ō koutou reo matatini me te pāngarau. Tēnā rawa atu koutou katoa. Me kī ake he manawa tītī koutou.

Ki ngā kaimahi ā kura i uiuia e mātou mai i Ōtautahi, Whakatū, Waihopai me Temuka. Ka nui te mihi ki a koutou mō te manaaki i a mātou, mō te whakarite uiui, mō te whakapuaki i ō whakaaro. I tino kitea tō aroha me tō manaakitanga ki ngā tamariki me ngā whānau i mahi tahi ai. Nā reira i whakanuia mātou ki te tūtaki ki a koutou me te rongo mō ō wheako - te pai, te kino me te wero.

He mea waimārie hoki mātou ki te kōrero ki ngā tamariki me ngā whānau me te rongo i tō rātou whakahīhī me tō rātou koa nā runga i ō whakatutukitanga.

Ki ngā kaimahi ō Te Pūtahitanga o Te Waipounamu me te Whānau Ora Commissioning Agency, kei te rere tonu ngā mihi ki a koutou katoa. I horahia mai e koutou ō koutou ake mātauranga ā-rohe.

E rere ana ngā mihi ki ngā kaimahi ō Te Whānau o Waipareira, Manukau Urban Māori Authority, Te Kura o Waatea, Te Kōhao Health me Te Rōpū Āwhina.

Ki ngā kaimahi anō hoki i Te Pūtea Whakatupu. Ka nui te mihi mō tō manawanui me tō whanaungatanga mai i hanga i roto i ngā tau e toru kua taha ake nei.

Ka mihi ki a koutou katoa mō tēnei waahi whakamīharo. Tika tonu nā mātou te honore nui.

Dr Chelsea Grootveld

Ngaitai, Ngāti Porou, Whānau-a-Apanui, Whakatōhea Director - AIKO Consultants Limited

EXECUTIVE SUMMARY

INTRODUCTION

This is the report on the summative evaluation of Te Pūtea Whakatupu’s Ngā Uri Whakatupu - Te Pūtea Whakatupu’s investment in literacy and numeracy. The evaluation looks at how well the investment in both Te Kete Aronui and Te Reo Matatini me te Pāngarau programmes has supported and contributed to improving numeracy and literacy outcomes for tamariki aged 5-12 living in targeted areas. The evaluation covers the three-year investment period from July 2019 to June 2022.

Information was gathered through a desktop review that included Te Kete Aronui Social Return on Investment report1, and qualitative interviews with selected stakeholders from Te Reo Matatini me te Pāngarau including tumuaki, kaiako, pouako, kaiāwhina, tamariki, whānau and Te Pūtahitanga o Te Waipounamu staff.

The data is presented through the key findings and three kura case studies. The case studies showcase approaches in three Te Waipounamu (South Island) locations and illustrate a mix of kura size, type and location. The quantitative data has its limitations and is based on survey results provided by partners and kura.

KEY FINDINGS

The evaluation found that targeted investment in two different and innovative kaupapa Māori numeracy and literacy approaches has resulted in positive education and wellbeing outcomes and impacts for tamariki and whānau across ten kura in Te Waipounamu (Blenheim, Whakatū, Ōtautahi, Temuka, Kaikoura, Ōtepoti, Waihopai) and four locations in Te Ika a Māui (West Auckland, South Auckland, Waikato, and Wellington). This is a significant achievement within the context of the current Aotearoa New Zealand education landscape.

The innovation was characterised by kaupapa Māori and Whānau Ora approaches centred on tamariki and whānau learning needs, whānau strengths, whānau aspirations and whānau wellbeing. When whānau are engaged in ways that work for them, they thrive. Te Kete Aronui and Te Reo Matatini me te Pāngarau have helped target and accelerate the learning needs of tamariki and whānau. Overall, tamariki and whānau engagement rates were high. Across the three-year period, over 1,600 tamariki and 1,000 whānau were engaged.

Implementation across both programmes was significantly impacted by the COVID-19 pandemic. In 2020, programme delivery was put on hold to ensure partners and kura could focus on supporting the needs of tamariki, whānau and community. However, the impacts of the pandemic have a long tail and are acutely felt to this day impacting negatively on student attendance rates, whānau re-engagement, and staff wellbeing and retention. Kura and partners continue to work hard to reengage tamariki and whānau.

Whakatupu: An independent evaluation report

1 Lakhotia, S., Mehr, E.S. (2021). Te Kete Aronui Report: Social Return of Investment. Whānau Ora Commissioning Agency. Unpublished Report.

An independent evaluation report

KEY FINDINGS (CONTINUED)

The findings showed that what works for tamariki and whānau includes:

» Being engaged in ways and at a pace that works for them and not giving up if whānau drop off the radar for a period

» Whānau Ora and whānau centred holistic approaches that support numeracy and literacy outcomes for the whānau and seek to remove barriers to learning

» Validated approaches like Te Kete Aronui

» Kura designed approaches like Te Reo Matatini me te Pāngarau

» Kaupapa Māori activities that support numeracy and literacy teaching and learning inside and outside the kura

» Fun activities that utilise games, competition, technology, creation of resources, incentives (prizes) and regular celebrations

In terms of what makes an effective programme:

» Delivery flexibility, consistency, and adaptability

» The right assessment and learning tools to measure progress and accelerate numeracy and literacy outcomes in English and te reo Māori

» A group of pouako or kaiako driving the programme so that it is not on the shoulders of one person

» Resource to support sustainable delivery

» Fit for purpose monitoring and reporting

Finally, what investors should think about when investing in numeracy and literacy programmes for Māori medium and kaupapa Māori settings includes:

» The potential for positively impacting whānau members beyond the tamariki enrolled

» The greater likelihood of long-term educational success (and other impacts like earning potential) of Māori when they have literacy and numeracy confidence from an early age

» Likely resourcing differences when investing in urban versus rural settings (to address staff retention and capacity)

» The potential for this programme to be building tamariki skill with digital technologies

CONCLUSION

The evaluation findings point towards the positive outcomes and impacts that can be achieved with targeted investment in two innovative, kaupapa Māori, Whānau Ora and whānau centred numeracy and literacy approaches.

The strong partnership developed with Te Pūtahitanga o Te Waipounamu and Whānau Ora Commissioning Agency over a number of years is important to the overall success of the kaupapa. Both agencies draw on their trusted and deep networks (kura and partners) to deliver the kaupapa using kaupapa Māori, Whānau Ora, and whānau centred approaches.

Te Reo Matatini me te Pāngarau – designed by kura and delivered for kura; and Te Kete Aronui – a validated kaupapa Māori numeracy and literacy programme delivered by Whānau Ora partners to four communities, have supported a previously unmet numeracy and literacy need for tamariki and whānau.

The recent launching of Hei Raukura mō te Mokopuna by the Ministry of Education provides the strategic framework to work with kura, whānau, hāpori, and agencies to design bespoke kaupapa Māori numeracy and literacy programmes. The investment made by Te Pūtea Whakatupu Trust in partnership with Te Pūtahitanga o Te Waipounamu and Whānau Ora Commissioning Agency provides a robust evidence base showcasing what can be achieved, what is effective and how to accelerate tamariki and mokopuna numeracy and literacy success –as defined by them.

Whakatupu: An independent evaluation report

INTRODUCTION

This is the report on the independent summative evaluation of Ngā Uri Whakatupu – the Te Pūtea Whakatupu Trust investment in kaupapa Māori literacy and numeracy programmes led by its partners, Te Pūtahitanga o Te Waipounamu and Whānau Ora Commissioning Agency.

The evaluation looks at how effective this investment was in terms of supporting positive changes (including literacy and numeracy outcomes and impacts) for tamariki and whānau. The report is focused primarily on the Te Pūtahitanga o Te Waipounamu programme – Te Reo Matatini me te Pāngarau. It includes a desktop analysis of the Whānau Ora Commissioning Agency monitoring and reporting to supplement the comprehensive SocialReturn-On-Investment report published in 2022.1

This evaluation builds on the baseline year one evaluation carried out by Aiko in 2019, and a preliminary report conducted in 2020.

Information was gathered through a desktop review, and semi structured interviews with tamariki, whānau, kaiako, pouako, kaiāwhina, tumuaki and Te Pūtahitanga o Te Waipounamu staff. Kura case studies are included to provide in-depth insights to the approaches kura adopted to support tamariki and whānau. The evaluation was carried out during March to August 2022.

1 Lakhotia, S., Mehr, E.S. (2021). Te Kete Aronui Report: Social Return of Investment. Whānau Ora Commissioning Agency. Unpublished Report.

BACKGROUND

INVESTMENT IN LITERACY AND NUMERACY

Te Pūtea Whakatupu Trust (the Trust) was established under the Māori Fisheries Act 2004 as part of the settlement of Māori fishing rights claims. As an independent charitable trust, its role is to provide strategic leadership in education, training, and workforce development for Māori, and to manage the Trust’s fund made available for these purposes. The Trust has an obligation to ensure benefits are made available to all Māori and ive regard to Māori who do not associate with their iwi and/or do not receive benefit from a mandated organisation.2

In 2018, the Trust made a strategic decision to re-invest in literacy and numeracy programmes targeted towards improving educational outcomes for tamariki (children) in Years 1 to 8, located in high need areas.3

This investment decision was part of a broader strategy review carried out by the Trust, which resulted in a sharpened focus on four specific whāinga:

» Te Whāinga Tuatahi – Ngā Uri Whakatupu: Futureproof the Next Generation For the 22nd century

» Te Whāinga Tuarua – Mentoring and Governance Training

» Te Whāinga Tuatoru – Innovation and Research for Māori Engagement and Growth

» Te Whāinga Tuawhā – Transforming the Trust

The Trust's strategic intent is supported by its outcomes framework which describes the outcomes the Trust seeks to achieve through its investment. The main activities supporting Te Whāinga Tuatahi are to:

» co-design the numeracy and literacy services that will be contracted with Whānau Ora Commissioning Agency and Te Pūtahitanga o Te Waipounamu

» develop a service plan for delivery commencing February 2019 to June 2022

» develop clearly defined outcome and social impact measures partner with the service providers to influence Government education funding into numeracy and literacy services for Māori. The investment will be the catalyst to evidence-based outcomes and look to grow investment in this area to $50 million

The Trust articulated a theory of change which demonstrates the connect between its investment in numeracy and literacy for tamariki Māori (and their whānau), and the outcomes and social impact the Trust seeks to achieve. The theory of change is based on the following assumptions:

» Kaupapa Māori based numeracy and literacy programmes work

» supporting and engaging whānau is critical to creating sustainable change

» Te Pūtahitanga o Te Waipounamu and Whānau Ora Commissioning Agency are critical investment partners with a proven history of efficiency, effectiveness, and excellence by, with and for whānau

» ongoing investment is critical to help create sustained change

2 Te Pūtea

3

Whakatupu: An independent evaluation report

Whakatupu Trust. (2017). Te Rautaki. Rua mano tekau mā whitu ki Rua mano rua tekau mā rua. Unpublished

KPMG. (2018). Te Pūtea Whakatupu Trust Business Case Support. Unpublished report paper.

Whakatupu: An independent evaluation report

PERSISTENT LOW LITERACY & NUMERACY RATES FOR TAMARIKI MĀORI

Being literate is vital to lifelong learning, wellbeing, and quality of life. Literacy includes more than reading and writing – it is also oral language, critical thinking, and digital literacies. Globalisation, increasing diversity, new technologies, social media, and digitisation have created both problems and opportunities and are changing how we live.4

In Aotearoa, there is huge variation in literacy achievement, and this adversely impacts more tamariki Māori and Pacific tamariki as well as those living or attending school in areas with high socio‐economic deprivation.5 In short, too many tamariki Māori are not receiving the support they need to succeed in their literacy learning which can negatively impact their whole lives.6

Māori or Pasifika students' achievements in mathematics, reading and writing is lower than Pākehā students.7 When comparing schools across deciles, there is little overall progress to report on overcoming inequalities in educational achievement in 2019.8 The achievement gap widened between 2017 and 2018, from 13.7% to 14.1%, continuing a widening trend since 2016.

Any progress being made to close the gaps is seen through declining achievement among students in socioeconomically advantaged schools rather than rising achievement for students at schools in disadvantaged areas. The percentage of Māori leaving school with less than NCEA Level 1 reduced by 25%, but the proportion is still 2.5 times higher than for non-Māori school-leavers.9

While there are standardised measures to evidence literacy attainment for students in English medium settings below NCEA, the Ministry of Education does not have equivalent evidence about te reo matatini attainment for tamariki in Māori medium education settings below NCEA, as data is not collected for these by any of the national or international studies and programmes.

The New Zealand education system has fallen well short in its responsibilities and obligations to support and accelerate the numeracy and literacy needs and aspirations of tamariki Māori.

4 Ministry of Education.(2021). Strengthening literacy and te reo Matatini Briefing Note March 7 April 2021.

5 McNaughton, S.(2020). The literacy landscape in Aotearoa New Zealand. Office of the Prime Minister’s Chief Science Advisor

6 Sutherland, D.(2019). Literacy Proficiency among Students in Aotearoa-New Zealand: Why the Gap between Māori and Pākehā?”. In Leiden, Perspectives on Indigenous Writing and Literacies. The Netherlands: Brill.

7 Ministry of Education.(2018). Achievement and progress in mathematics, reading and writing in primary schooling, Analysis of e-asTTle assessment data, 2011 to 2016 Ministry of Education, New Zealand

8 Measured by the proportion of students from deciles 1–3 and deciles 8–10 schools who left without NCEA Level 1 or better

9 Tanielu, R., Barber, P., & Wijeysingha, V. (February 2020). Tangata Whenua, Tangata Tiriti, Huia Tangata Kotahi: State of the Nation Report. The Salvation Army, Social Policy & Parliamentary Unit.

10 Ministry of Education (2022) Hei Raukura Mō Te Mokopuna Strategy A Te Reo Matatini and Pāngarau Strategy Ministry of Education, Wellington

TABLE 1: NGĀ URI WHAKATUPU LITERACY AND NUMERACY PROGRAMMES THEORY OF CHANGE 2019-2022

Inputs Outputs Outcomes Impact

Investment in 2018 pilot and evaluation

Insights from 2018 evaluation

Investment in programme from 2019-2022

Over 1,000 tamariki (and whānau) to access numeracy and literacy services by 2022

Comprehensive co-design numeracy and literacy services, including digital and science literacy

Government policy that implements kaupapa Māori strategies and increases funding to services for tamariki Māori

Publish and disseminate evaluation results

Improve and promote the educational achievement of tamariki Y1-Y8 in South Auckland, Hamilton, Wellington, and Christchurch

Whānau Māori engaging with their tamariki mokopuna

Tamariki are confident in reading their world, critically thinking and problem-solving.

Tamariki are achieving high standards of literacy and numeracy outcomes in kura and schools

RAUKURA MŌ TE MOKOPUNA STRATEGY

Te reo matatini refers to the many ‘faces or facets of language’ - ngā tini mata o te reo. It takes into account the multitude of ways that language can be linguistically, culturally, socially, spiritually, and academically acquired, represented and expressed. It is far more expansive than the term ‘literacy’ with which it has unintentionally become synonymous.

Te reo Māori is at the core of te reo matatini, alongside the ability to articulate the stories, histories, art, and philosophies of Māori. Mātauranga Māori is also critical to te reo matatini as a catalyst for the revitalisation and preservation of te reo Māori and tikanga Māori. Recognising the place of whānau, hapū and iwi and the collective responsibility required to realise tamariki potential is also important to a strategy for te reo matatini and delivering effective te reo matatini pedagogy through te reo Māori in Māori medium pathways. In August 2022, the Ministry of Education launched its ‘Hei Raukura mō te Mokopuna’ strategy – a te reo matatini and pāngarau strategy.10 The overall purpose of ‘Hei Raukura mō te Mokopuna’ is to promote and enhance a mātauranga and kaupapa Māori understanding of te reo matatini and pāngarau across the curriculum. This is expected to result in increased wellbeing, higher educational outcomes and qualifications, and better job opportunities for mokopuna.

The strategy objectives are outlined in the appendices, however what is important is that it marks the acknowledgement that central to this approach is working with kura and communities to lead and design the process and actions.

HEI

“The New Zealand education system has fallen well short in its responsibilities and obligations to support and accelerate the numeracy and literacy needs, and aspirations of tamariki Māori”

Whakatupu: An independent evaluation report

2018 LITERACY AND NUMERACY PILOT AND 2019 EXPANSION

Te Pūtea Whakatupu Trust contracted two whānau-centred organisations, Te Pūtahitanga o Te Waipounamu (based in Christchurch) and Whānau Ora Commissioning Agency (based in West Auckland), to deliver numeracy and literacy educational services in their respective regions. The pilots delivered literacy and numeracy programmes in both te reo and English across a range of school and community language contexts that included English-medium, bilingual settings, and Māori-medium.

The aim of the collaboration was to establish a tailored/mixed programme with multi-layered strategies which target accelerated achievement in numeracy and literacy. The intention was to provide opportunities to focus on Māori achievement in numeracy and literacy at school and within the whānau by building an understanding of how our tamariki learn and identifying what works for them.11

Te Pūtea Whakatupu Trust wanted to see innovation. Kaupapa Māori approaches were sought along with involvement of community and whānau, and programmes were to be tailored to the needs of the tamariki.

Aiko conducted a formative evaluation of year one of the programme and was asked to evaluate years two and three, providing a summative view of the pilot’s impact. Considerable learnings were gained from the first year of delivery which influenced changes in subsequent years.

The process evaluation found the pilot programmes achieved positive ‘successes’ within a relatively short implementation timeframe. This success was based on all pilot programmes achieving the contracted performance measures and being able to demonstrate positive shifts in key literacy and numeracy outcome indicators.

The evaluation identified three key areas to better support effective programme delivery:

» implementation – enable inter-programme shared learning; explore use of Whānau Ora navigators; resource professional learning and development for project leads; consider establishing an expert advisory group

» monitoring and reporting – resource a co-design process with the two commissioning agencies, kura and sites; develop a ‘fit for purpose’ monitoring and reporting framework

» impact assessment – develop a fit for purpose evaluation framework

Buoyed by the positive evaluation results and the Trust’s commitment to achieve its strategy, the Board agreed to increase its investment in the programme for an additional three-year period and broaden the reach to include tamariki and their whānau in more regions in Te Waipounamu, Waikato and Wellington.

11Te Pūtahitanga o Te Waipounamu (2019) First Report to Te Pūtea Whakatupu Trust May 2019.

MONITORING AND REPORTING FRAMEWORK

Table 2 presents the indicators used to measure progress towards the achievement of the Trust’s intended outcomes and social impact. Te Pūtahitanga o Te Waipounamu and the Whānau Ora Commissioning Agency also have their own measures of success based on the Whānau Ora outcomes framework which are presented to support and enhance the overall picture of how the programme has made a positive change to whānau wellbeing and educational outcomes.

Table 2: Ngā Uri Whakatupu Outcome Indicators and Measures 2019-2022

Outcomes Social Impact Measures & Indicators Data Source

Improve and promote the educational achievement of tamariki Y1-Y8 in South Auckland, Hamilton, Wellington, and the South Island.

Whānau Māori engaging with their tamariki mokopuna

Tamariki are confident in reading their world, critically thinking and problem-solving.

Tamariki are achieving high standards of literacy and numeracy outcomes in kura and schools

#tamariki participants #whānau participants % Increased positive engagement % Increased confidence to read and write % Increase confidence to speak % Improved reading and writing levels % Improved reo-āwaha and reo-ā-tuhi

Quarterly monitoring reports

Baseline and programme end testing

Survey data – tamariki and mātua

Qualitative interviews with whānau, tamariki and project leads

Whakatupu: An independent evaluation report

In line with their whānau centered commissioning approaches, both agencies also reported on progress against the Whānau Ora outcome areas or pou.

COVID-19 PANDEMIC AND REDIRECTION OF SUPPORT

On 23rd of March 2020 the Board of Te Pūtea Whakatupu Trust agreed to support their long-term partners through the COVID-19 lockdown period. As a first manifestation of that support, the Board recognised that the contract for the period 1st of April 2020 to 30th of September 2020 required variation. As a result, the numeracy and literacy services for the period 1st of April to the 30th of September 2020 was not required due to the impact of COVID-19.

During this period, Te Pūtahitanga o Te Waipounamu and Whānau Ora Commissioning Agency reprioritised their efforts towards COVID-related support for whānau and communities. Staff were redeployed and mobilised to stand up vaccination clinics, set up kai and medical supplies distribution and delivery centres, and support whānau in every way possible during this period of uncertainty and significant hardship.

“Te Pūtea Whakatupu Trust wanted to see innovation. Kaupapa Māori approaches were sought along with involvement of community and whānau, and programmes were to be tailored to the needs of the tamariki”

OVERVIEW OF TE REO MATATINI ME TE PĀNGARAU

Te Pūtahitanga o Te Waipounamu

Te Pūtahitanga o Te Waipounamu provides a comparative service for Te Wai Pounamu as the Whānau Ora commissioning agency provides in their region. In 2018, three kura were approached to be involved in the pilot – Te Pā o Rākaihautū, Waitaha, and Haeata Community Campus.

In 2019, the programme was expanded to include five new language contexts in four new regions; three kura kaupapa Māori – Te Kura Kaupapa Māori o Whānau Tahi (Christchurch), Te Kura Kaupapa Māori o Tuia te Matangi (Nelson), Te Kura Kaupapa Māori o Ōtepoti (Dunedin); one pā wānanga – Te Pā Wānanga (Blenheim); and one wharekura – Te Kura Kaupapa Māori o Arowhenua (Invercargill).

In 2021, two kura were included to expand the delivery regions into rural communities, Arowhenua Māori School (Temuka) and Hāpuku School (Kaikoura). A total of ten language settings in five regions are involved in the programme.

TABLE 3: TE REO MATATINI ME TE PĀNGARAU KURA

Provider Location Project Start year Description

Te Pā o Rākaihautū Ōtautahi Te Hā o Te Ora 2018 Reo ā-waha (Oral language)

Te Kura Whakapūmau i te reo tūturu ki Waitaha

Ōtautahi Kāinga Ora 2018 Reo-ā-waha, pāngarau, hāngarau

Haeata Community Campus Ōtautahi He Reo Tupu He Reo Ora 2018 Reo-ā-waha and pāngarau

Te Kura Kaupapa Māori o Whānau Tahi Ōtautahi Te Pikinga ki Runga 2019 Te reo matatini, hāngarau

Te Kura Kaupapa Māori o Tuia Te Matangi Whakatū Te Reo Āio 2019 Te reo matatini, hāngarau

Te Kura Kaupapa Māori o Ōtepoti Ōtepoti Tautāwhi Tamariki Kāinga Kōrerorero 2019 Reo-ā-waha

Te Pā Wānanga Ōpawa Te Pouhere o Te Pou Wānanga 2019 Reo-ā-waha

Te Wharekura o Arowhenua Murihiku He Kura Hai Kāinga 2019 Reo matatini, pāngarau, hāngarau Arowhenua Māori School Temuka Nōku te Ao 2021 Reo-ā-waha

Hāpuku school Kaikoura Wā Haututūtanga 2021 Reo-ā-waha

Kura were selected by Te Pūtahitanga o Te Waipounamu. Taking on board the lessons learnt from the three pilot kura based in Ōtautahi, the programme was purposefully expanded to include a diverse range of kura located across Te Waipounamu. Kura were able to deliver the programme in whatever way they deemed most effective to achieve the intended outcomes.

In 2019, a programme coordinator was appointed to work directly with kura and provide support, mentoring and advice. The coordinator was an experienced Māori medium educator and practitioner who understood the realities and challenges faced by kura kaupapa Māori, wharekura and bilingual kura.

OVERVIEW OF TE KETE ARONUI

Whānau Ora Commissioning Agency

The

Whānau Ora Commissioning Agency is the North Island commissioning agency formerly known as Te Pou Matakana. In 2018, Te Whānau o Waipareira (Waipareira) and Manukau Urban Māori Authority (MUMA) were approached to be involved in the Te Kete Aronui pilot partnership with Te Pūtea Whakatupu Trust. In 2019, two new Partners, Te Kōhao Health and Te Rōpū Āwhina were contracted to deliver programmes to their communities in Wellington and Waikato.

TABLE 4: OVERVIEW OF TE KETE ARONUI DELIVERY PARTNERS

Provider Location Start Description

Te Whānau o Waipareira West Auckland 2018

Manukau Urban Māori

Delivered at Waipareira for tamariki from different kura across West Auckland.

Authority - Te Kura Māori o Waatea South Auckland 2018 Delivered at Te Kura Māori o Waatea

Te Kōhao Health Waikato 2020

Te Rōpū Āwhina Wellington 2020

Partnered with Emberley Community Centre for tamariki attending homework centre

Delivered at Te Rōpū Āwhina for tamariki from kura across Te Whanganui a Tara

Developed in 2011, Te Kete Aronui is a programme that provides holistic education support to tamariki to achieve improved and sustained numeracy and literacy skills.12 From 2011 to 2019, Te Kete Aronui was funded and delivered by Te Whānau o Waipareira to a targeted group of tamariki and whānau in West Auckland.

Based on elements of the successful Kip McGrath approach to numeracy and literacy, Te Kete Aronui consists of assessments and enrolment processes, as well as whanaungatanga with the tamariki and their whānau, creating individualised plans and learning programmes for each child, teaching sessions and continuous review of tamariki progress and needs.

The teaching classes are a series of weekly one-hour numeracy and literacy sessions. Each provider usually delivers two classes per day with 2-3 sessions per week. Tamariki are enrolled in either numeracy or literacy or both sessions depending on their learning needs. Key to the success of Te Kete Aronui is its unique approach that underpins its delivery. Te Kete Aronui integrates Whānau Ora, kaupapa Māori, commissioning, and flexibility to provide an inclusive and fun learning environment through its capable kaiako and by working closely with other stakeholders.

An independent evaluation of Te Kete Aronui carried out in 2013 highlighted positive improvements in tamariki numeracy and literacy skills and achievement. 13 The holistic approach provides a path to more sustainable outcomes, as well as a safe and fun environment with appropriate teaching methods supports child-centred teaching and learning. These factors have enabled kaiako to accommodate different capabilities and learning styles of tamariki and offer flexibility to respond to the whānau and community contexts. 12

Whakatupu: An independent evaluation report

Whānau Ora Commissioning Agency.(2021). Te Kete Aronui Return on Social Investment Report May 2021. 13Te Maro, P.(2013). Ko Te Kete Aronui Report: Evaluative Research of the Te Kete Aronui/Kip McGrath Literacy and Numeracy Programme. Unpublished Report.

EVALUATION APPROACH

This is a kaupapa Māori summative evaluation that answers the following key questions:

» How well are the programmes working and what are the critical success factors?

» What are the impacts of this investment for tamariki, rangatahi and their whānau? The evaluation involved:

» Desktop review of key monitoring data, interviews and discussions with Te Pūtahitanga o Te Waipounamu to understand any changes to the context and current operating environment in year three. Documents reviewed included year two and three kura milestone reports, and kura survey data including videos

» Semi-structured interviews with Te Pūtahitanga o Te Waipounamu Coordinator, Te Pūtea Whakatupu Coordinator and kura representatives

» Kura case studies focused on three kura in Te Waipounamu – Te Kura Kaupapa Māori o Arowhenua, Te Kura Kaupapa Māori o Tuia Te Matangi and Arowhenua Māori School. These are included in the findings to provide in-depth insights into the success factors and barriers drawing on rangatahi, rōpū and whānau voices. Kura were selected with Te Pūtahitanga o Te Waipounamu and include a mix of kura type (Kura Kaupapa, Kura ā Iwi) locations (urban and rural) and kura at different stages of implementation

Uri Whakatupu: An independent evaluation report

TABLE 5: KURA CASE STUDIES

Case Study Description Rohe

Te Kura Kaupapa Māori o Arowhenua He kura hei kāinga Murihiku – Invercargill

Te Kura Kaupapa Māori ō Tuia Te Matangi Āio Te Reo

Whakatū – Nelson

Whakatupu: An independent evaluation report

Arowhenua Māori School Nōku te ao Temuka

More detailed information about the evaluation methodology is included in the appendices. Kura profiles for the remaining six kura are set out in the appendices.

LIMITATIONS OF THE EVALUATION

This evaluation includes qualitative and quantitative data for Te reo matatini me te pāngarau. The quantitative data has its limitations and is largely based on survey data received from the ten kura.

There are gaps in the evaluative data during the period from July to December 2020. This is when Te Pūtea Whakatupu approved a contract variation enabling kura to pivot and prioritise resources on supporting whānau during and post the COVID-19 pandemic.

Due to the enduring impacts of COVID-19, particularly for whānau and tamariki in Tāmaki Makaurau, and timing constraints, the decision was made not to carry out qualitative interviews with the Whānau Ora Commissioning Agency partners. As a result, this report is based on a desktop review of milestone reporting provided by the Whānau Ora Commissioning Agency from 2019 to 2022.

A comprehensive Social Return on Investment report was co-funded by Te Pūtea Whakatupu and Whānau Ora Commissioning Agency in 2021 to evidence the quantitative value of the investment. A summary of the findings is included in this report.

Ngā Uri Whakatupu: An independent evaluation report

OVERALL FINDINGS

Key findings have been aligned to the evaluation questions and, with some of the findings overlapping, we have set out a table (next page) to explain how these have been arranged. The overall findings from Te Reo Matatini me te Pāngarau and Te Kete Aronui data are presented in Table 6. While two different investment approaches were delivered, a set of common findings emerged.

There were also themes unique to each approach. These are presented separately in Table 7. Three Te Reo Matatini me te Pāngarau case studies and kura profiles are included in the appendices to provide more in-depth qualitative insights.

TABLE 6: OVERALL EVALUATION FINDINGS

Areas Covers

Key findings

Success factors

» High rates of engagement and positive education, numeracy, literacy and wellbeing outcomes and impacts achieved for tamariki and whānau

» Whānau need to be fully engaged to support tamariki success

» Whānau Ora and whānau centred approaches boosted engagement and outcomes

» The kaupapa was agile, kura and partners valued high trust relational approach

» Tamariki enjoyed fun and diverse numeracy and literacy approaches

» COVID-19 pandemic exacerbated existing whānau vulnerabilities

Barriers to success

Impacts

» Rebuilding momentum post-COVID 19 was a challenge

» Lack of equitable whānau access to devices and the internet

» Lack of te reo Māori numeracy and literacy theories, tools and pedagogies

» Intended numeracy and literacy outcomes and impacts achieved for tamariki and whānau in Te Ika a Māui and Te Waipounamu.

KEY FINDINGS

The evaluation found that targeted investment in two different and innovative kaupapa Māori numeracy and literacy approaches has resulted in positive education and wellbeing outcomes and impacts for tamariki and whānau across ten kura in Te Waipounamu (Blenheim, Whakatū, Ōtautahi, Temuka, Kaikoura, Ōtepoti, Murihiku) and four locations in Te Ika a Māui (West Auckland, South Auckland, Waikato and Wellington). This is a significant achievement within the context of the current Aotearoa New Zealand education landscape.

The innovation was characterised by kaupapa Māori and Whānau Ora approaches centred on tamariki and whānau learning needs, whānau strengths, whānau aspirations and whānau wellbeing. When whānau are engaged in ways that work for them, they thrive. This starts with being invited to come into the space (often more than once) to share in their tamariki learning; building positive relationships - which usually involves sharing kai and manaakitanga; over time this leads to the building of trust and confidence - which lays the foundation for a positive and enduring relationship with the kura centred on learning and aspirations. Both approaches have helped target and accelerate the learning needs of tamariki and whānau.

Overall, tamariki and whānau engagement rates were high. Over the three-year period, the average number of tamariki and whānau engaged annually in the programme in Te Waipounamu exceeded the contracted targets. Similarly, in Te Ika a Māui, over 600 tamariki were engaged.

Whakatupu: An independent evaluation report

KEY FINDINGS (CONTINUED)

Implementation across both programmes was significantly impacted by the COVID-19 pandemic. In 2020, programme delivery was put on hold to ensure partners and kura could focus on supporting the needs of tamariki, whānau and community. However, the impacts of the pandemic have a long tail and are acutely felt to this day impacting negatively on student attendance rates, whānau re-engagement, and staff wellbeing and retention. Kura and partners continue to work hard to reengage tamariki and whānau.

Despite these unprecedented challenges, the outcomes and impacts achieved as a result of both approaches were significant.

The shared tamariki outcomes achieved by both approaches were:

» increased attendance (pre-COVID-19)

» increased confidence and motivation to achieve

» increased engagement and positive attitude and enjoyment of learning,

» increased willingness to interact with the teacher

» increased numeracy and literacy skills

The shared whānau outcomes included:

» greater value placed on education

» improved communications in the home

» increased understanding of numeracy and literacy and child’s progress

» improved engagement and interactions with kura

» increased interest in education and child’s learning

In addition, Te Reo Matatini me te Pāngarau reported on the following tamariki and whānau outcomes:

» increase in learning new reo Māori words

• increased te reo use including domains outside of kura

• increased digital literacy

• increased whānau confidence and motivation to learn, read and use te reo

WHAT ARE THE SUCCESS FACTORS?

The critical success factors identified in this evaluation are:

» Whānau need to be fully engaged to support tamariki success

» Whānau Ora and whānau centred approaches boosted engagement and outcomes

» The kaupapa was agile – kura and partners valued the high trust relational approach

» Tamariki enjoyed fun and diverse approaches to learning numeracy and literacy skills

“Overall, tamariki and whānau engagement rates were high. Over the three-year period, the average number of tamariki and whānau engaged annually in the programme in Te Waipounamu exceeded the contracted targets”

CRITICAL SUCCESS FACTOR ONE:

Whānau need to be fully engaged to support tamariki success

The level of whānau engagement in these programmes was essential to tamariki success and had positive impact on the wider whānau. Mainstream schools have struggled for decades to engage whānau in learning, yet this programme shows what can be done when whānau are invited to the table in ways and at a pacea that works for them.

The whānau we spoke to all wanted the very best for their tamariki in education, health and wellbeing. They talked about their tamariki being confident and proficient or fluent readers, writers and speakers of te reo Māori and English. They want their tamariki to flourish and be proud of who they are as Māori.

In Te Waipounamu, a strong cohort of kaumātua were identified as raising their mokopuna across all of the kura. Their aspirations for their mokopuna are strong, particularly in regard to learning and speaking te reo Māori. Many of the kaumātua were not speakers of te reo themselves and some spoke openly about their language trauma. All of the kaumātua we spoke to valued te reo for their mokopuna.

Some whānau have had negative experiences of education and the schooling system which meant they struggled to feel confident about engaging in school activities, however they were drawn to this kaupapa because it was positive (i.e., not focused on poor behaviour), and tamariki, kaiako and pouako kept asking and encouraging them to attend.

For other whānau, the barriers preventing strong engagement are significant. Kura and partners talked about whānau that experience language trauma; are working multiple jobs; supporting other whānau members as caregivers or financially; suffering the hardships created by poverty; experiencing isolation or mental health and wellbeing challenges.

“Proud of my girl [daughter] and what she’s done in her reading and writing. Not just in reo, English too. She likes to read now, enjoys it. We’ve seen the change in our whānau really. We read too. Was always hard for me but, I’ll try. I’ll give it a go.”

- Whānau, 2020

“I didn’t used to come to kura things. I kinda just felt like it wasn’t my place. I didn’t feel comfortable, knew I needed to come for my girls,to support them. I’ve been coming for two years, learning the reo, helping out. It’s the aroha I think, kai, manaaki. Everyone is friendly and welcoming. My girls are happy I’m here and I get to meet other whānau. Learn about my reo, my whakapapa to support the girls.”

- Whānau, 2020

independent evaluation report

CRITICAL SUCCESS FACTOR TWO:

Whānau Ora and whānau centred approaches boosted engagement and outcomes

Whānau Ora and whānau centred approaches were critical to engaging tamariki and whānau and achieving the intended outcomes. Whānau ora is focused on supporting whānau to thrive, the data showed how this was not simply a literacy and numeracy programme, it was holistic, expansive, and focused on whānau and tamariki education and wellbeing outcomes through literacy and numeracy.

Te Pūtahitanga o Te Waipounamu leveraged its Whānau Ora approach to work collaboratively with kura to develop Te Reo Matatini me te Pāngarau, creating space for each kura to determine their needs and priorities, and express ‘mana motuhake’.

For example, kura pouako and kaiako started by asking whānau what they want to achieve for themselves. Kura provided flexible support; focused on relationships, self-agency and building skills for the long term; recognised the complex challenges relevant to whānau wellbeing and wellness; acknowledged each whānau has different needs and there is no one-size-fits all approach. Most importantly, they celebrated and trusted that whānau have the skills, knowledge, and experiences to become self-managing and independent.14

The inter-relationship between tamariki success and the level of whānau engagement is well understood by these providers. Kura and partners experienced similar challenges with whānau engagement but on different scales depending on the kura location (urban and rural) and size. Some of the challenges included: a transient whānau population; hardships associated with living in poverty; a lack of access to affordable housing; a mistrust of government; whānau who have had negative experiences of schooling; whānau feeling whakamā about not being able to speak te reo, not knowing their whakapapa or not feeling strong in their identity as Māori.

Despite these challenges, Te Reo Matatini me te Pāngarau whānau engagement prior to the COVID-19 pandemic was strong – and kura worked hard to engage whānau during and post the pandemic. Some of the kura (4) had waitlists for their programmes.

Whānau Ora Commissioning Agency utilised its network of Whānau Ora partners to provide targeted support in the four regions. The partners are experienced in delivering holistic support to whānau using the Whānau Ora approach.

Tutors and kaiako worked with tamariki and whānau to develop individual tamariki plans and whānau plans to assess needs and identify goals, aspirations, and barriers to learning success. Whānau were provided with access to Whānau Ora navigators who worked with them to access social services and financial supports like Whānau Direct to help remove any barriers to learning success and activate whānau wellbeing.

Puni Kōkiri.(2021). Whānau Ora and Whānau Centred approaches Retrieved from: https://www tpk govt nz/en/nga-putea-me-nga-ratonga/whanau-ora/whanau-ora-kaupapa

“Whānau Ora forces us to look at the whānau unit, not just at the tamariki. The focus is on the whole, the holistic, barriers to tamariki learning and how whānau can draw on their strengths and how we can support them to move these.”

- Pouako, 2020

“Each kura has taken a home-grown approach to Te Reo Matatini me te Pāngarau. The unlimited opportunities this initiative has provided across Māori -medium is mind-blowing. Each initiative highlights the kaupapa that has taken many shapes based on tamariki and whānau needs and interests.”

- Coordinator, 2019

An independent evaluation report

CRITICAL SUCCESS FACTOR THREE:

The kaupapa was agile – kura and partners valued the high trust relational approach

Te Reo Matatini me te Pāngarau kura strongly valued the Te Pūtahitanga o Te Waipounamu approach to investing in and working with them. This approach was characterised as being ‘high trust, relational and empowering’ creating space for each kura to express ‘rangatiratanga and mana motuhake’. It is important to note that this approach was considered innovative because it was not the norm in terms of how kura are traditionally treated by funding partners.

Kura appreciated being able to design and implement at a pace that worked for tamariki and whānau and felt safe providing feedback to the coordinator and trusted that this would be acted on. Similarly, Te Kete Aronui partners valued the opportunity to build new relationships with kura and education providers in their local communities, to deliver a targeted and validated kaupapa Māori numeracy and literacy approach.

The 2020 decision made by Te Pūtea Whakatupu Trust, to put the kaupapa on hold during the COVID-19 pandemic was well supported by all stakeholders. Kura and partners expressed sincere thanks and gratitude for the funding that enabled them to pivot and provide care and learning packages directly to whānau. This gratitude was echoed by whānau who warmly received the care and learning packages.

“It’s not usual that kura like ours are given space and pūtea to express rangatiratanga and that of our whānau and ākonga. To be given the time to create and learn, adapt and make mistakes all along the way. You know, to be trusted, to be valued. Having [programme coordinator name] was hugely important for us. And to be resourced to support whānau with care packages and focus on them, was much appreciated.”

- Pouako Waihopai

CRITICAL SUCCESS FACTOR FOUR:

Tamariki enjoyed fun and diverse numeracy and literacy approaches

In Te Waipounamu, all the tamariki we spoke to enjoyed participating in this kaupapa. They described the activities as being fun because they got to spend more time with friends; got to learn new things, new reo, new kupu, waiata; play with and learn about new technology; see their whānau learning at kura; and help teach te reo to whānau at home (when they were in the mood). In particular, they all enjoyed doing these activities and sharing kai with whānau.

Kura utilised games and competitions (online and face-to-face), technology (coding, robotics, ipapa), celebrations and the sharing of kai with whānau, physical activities, and fieldtrips. This diverse range of activities kept tamariki and whānau interested and engaged.

In Te Ika a Māui, partners reported positive shifts in tamariki attitudes, self-esteem, and confidence as a direct result of improved numeracy, reading and writing comprehension. With boosted confidence this made learning fun for tamariki who had previously struggled in the classroom setting.

“ What I enjoy is………My pāpā and mum are learning te reo ……. Te pūrei ki ōku hoa me taku kaiako. He pai taku kaiako ki te whakaatu āna mahi hei tauira mōku ………Ko ngā kēmu me ngā whānau e haere mai me te reo Māori me ngā karetao me ngā īpapa.“

- Tamariki – Waihopai, Ōtautahi, Whakatū

“All kura and whānau were hugely supportive of the programme continuing in order to build on the positive momentum created as a result of the approaches – as well as reclaim the ground lost as a result of the pandemic.“

WHAT WERE THE BARRIERS TO SUCCESS?

COVID-19 exacerbating whānau vulnerabilities

The COVID-19 pandemic had a profoundly negative impact on tamariki and whānau learning, education and wellbeing outcomes, particularly those living in Tāmaki Makaurau and Waikato who experienced multiple regional lockdowns.15 In short, the pandemic exacerbated existing whānau vulnerabilities and inequities.

The kura, partners and whānau we interviewed all talked about how incredibly tough and challenging the past two years have been for everyone. The impact on programme delivery was significant. Kura and partners pivoted (and rightly so) to support whānau and communities and the programme was put on hold. However, encouraging whānau to return to kura and reengage has been challenging.

In particular, the vaccine mandate created significant tensions and challenges for staff, whānau and tamariki. This resulted in some whānau disengaging or leaving kura due to fears about the virus, vaccination or underlying health issues.

During this time kura were confronted with the realities of whānau household challenges in the context of a pandemic. Many whānau households are big and multi-generational which meant the average whānau bubble was much larger than the average household, many whānau experienced underlying health issues and suffer significant financial pressures. The majority of whānau struggle with online learning and multi-level teaching is difficult for kaiako and pouako. During this time, five kura were forced to close during the pandemic due to staff and tamariki illness. This perfect storm of factors impacted tamariki and whānau engagement and learning from the time the pandemic hit in 2020 through to today (June 2022).

15 Aiko Consultants Limited (August (2021). The effects of COVID-19 on Māori education outcomes Te Pūtea Whakatupu Trust

“ Protect the pā became the overriding kaupapa. It set the tone for all future decisions and the Board did not resile from its responsibility to protect the pā no matter what everyone else was doing.“

-

Tumuaki, Ōtautahi

evaluation report

WHAT WERE THE BARRIERS TO SUCCESS?

Building momentum post-COVID-19 a challenge

The long tail of the pandemic is acute and felt by kura, partners, whānau and tamariki. Kura worked hard to re-engage whānau and tamariki and rebuild momentum. While some kura were more successful than others, the findings showed how much of a struggle this was. For example, when the evaluation team visited three kura during May and June 2022, kura attendance levels at each kura were recorded at their lowest in decades. Moreover, whānau attendance and engagement at whānau hui had reduced significantly. Most kura were revisiting their whānau engagement plans and exploring different ways to support, reengage and reenergise whānau and staff.

Te Reo Matatini me te Pāngarau coordinator reported on how inventive kura were, juggling whānau contracting COVID-19, close contacts, supporting those isolating, running kura with staffing shortages, some kura trying to operate from home and one rural kura closing at one stage due to no staff to run the kura. In summation:

» Two out of ten kura had to put their kaupapa on hold due to limited staffing

» Five out of ten kura closed for a period to break the chain of transmission of COVID-19

» Most whānau still choose to opt for kanohi ki te kanohi learning rather than online

Te Kete Aronui reporting also highlighted how challenging it was for partners to reengage with tamariki and whānau. When whānau are struggling to make ends meet and having to work multiple jobs to put kai on the table, their capacity to engage with kura and the learning needs of their tamariki is significantly reduced.

“ Return to kura post lockdown has been difficult and stressful for kaiako, whānau and ākonga.“ - Coordinator, 2021

independent evaluation report

WHAT WERE THE BARRIERS TO SUCCESS?

Lack of whānau access to devices and the internet

The digital divide across all communities was further exacerbated by the pandemic. The assumption that every household has access to affordable and stable internet, and a robust device proved to be a falsity for many of the target tamariki and whānau.

In Te Waipounamu, some kura had to provide devices and pay for internet access directly to whānau. The additional resource provided by Te Pūtahitanga o Te Waipounamu presented three kura with an opportunity to purchase devices for tamariki and another three kura to purchase internet services for whānau.

However, even when whānau already possessed or were provided with access to devices and the internet, all kura faced different degrees of digital challenges with whānau. These included access to internet services, access to devices, and lack of proficiency with digital devices. Although most tamariki were digitally savvy, most whānau were not. The antidote to this was that all kura included a digital component to their delivery to build capability amongst whānau.

Te Kete Aronui partners also reported on the significant challenges they faced maintaining contact and engagement with tamariki and whānau in an online learning environment. This disconnect was amplified for the three partners not operating within a school context because they had less direct interaction with tamariki and whānau once schools switched to online learning.

“ The lockdowns hit our whānau hard…. Exposed our digital inequities. Like, access to devices and internet was a struggle, those ones in multi-generation homes, with lots of tamariki, whānau living on the breadline. This assumption by government that we’d all just shift to online learning. Well, that didn’t happen, and it’s still not happening for many of our whānau.“

- Pouako, Te Waipounamu, 2022

WHAT WERE THE BARRIERS TO SUCCESS?

Growing Māori numeracy and literacy theories, tools and pedagogies

Te Reo Matatini me te Pāngarau kura identified the need to develop a better understanding of numeracy and literacy theories and pedagogies through a te ao Māori lens. They commented on the importance of looking beyond a western view of what a literacy and numeracy programme should look like. Kura were at different stages of their journey in exploring what this could mean through a kaupapa Māori and mātauranga Māori lens.

The coordinator acknowledged that more work is needed to decolonise traditional approaches to literacy and numeracy in kura and provide access to relevant kaupapa Māori and mātauranga Māori professional development and learning to support this capability build across all kura. This relates to a broader issue around kura kaupapa Māori kaiako having equitable access to professional learning and development opportunities, like their mainstream education counterparts.

Te Kete Aronui partners also highlighted the need to support te reo matatini me te pāngarau for tamariki attending kura kaupapa Māori and Māori medium education settings, noting the increased demand from reo speaking tamariki and whānau and the inadequacy of English medium assessment tools and resources to accelerate te reo Māori numeracy and literacy achievement.

Some of the evaluation findings were specific to Te Reo Matatini me te Pāngarau's approach and delivery model.

“ Growing Māori numeracy and literacy theories, tools and pedagogies. We have a large cohort of tamariki that come from Māori medium settings. We found that the entry assessment tools do not provide a true account of the actual level of learning of tamariki – as they assess knowledge of English language. This impacts negatively on tamariki confidence. Our 25 kaiako are working with tamariki to overcome this, and we are also looking at more holistic assessment tools.“

- Kaiako, West Auckland, 2021

TE REO MATATINI ME TE PĀNGARAU FINDINGS

Supporting the design and development of innovative whānau led numeracy and literacy initiatives in ten Māori medium and bilingual kura, located in diverse regions and locations across Te Waipounamu was a significant achievement. Whānau have engaged well and there is great aroha for this kaupapa, with consistently high positive feedback from whānau and tamariki across the three years.

Delivery occurs through diverse and innovative activities based on whānau and tamariki strengths and interests, but the kaupapa has remained constant– whānau learning together, encouraging each other in their learning journeys, and growing in mātauranga Māori.

Since kura believe in supporting whānau in holistic ways, the kaupapa has broadened since the start of the pandemic to include ways that support whānau to get through including learning resources and wellbeing packages to keep whānau safe from the impact of the pandemic

TABLE 7: TE REO MATATINI ME TE PĀNGARAU FINDINGS

Areas Covers

Key findings

Success factors

Barriers to success

» High rates of engagement and positive education and wellbeing outcomes for tamariki and whānau

» Kaupapa Māori approaches designed by kura, tamariki and whānau

» Project coordinator role pivotal

» Building whanaungatanga between kura

» Kaiako retention and kura resourcing an ongoing challenge

» Generic monitoring and reporting tools do not align with innovative approaches

Impacts

» Positive results in established larger kura and positive indications in most of the other kura. A couple of kura are building their kaupapa.

Over the past three years, 1,324 tamariki, 639 whānau, 1,645 whānau members have engaged in te reo matatini me te pāngarau.16 Each year the number of tamariki and whānau members engaged in the kaupapa has increased with a small cohort of new tamariki and whānau engaged each year alongside the established cohort, across all kura.

» In year one, from July 2019 to June 2020, the data available showed on average a total of 259 tamariki, 259 whānau, 518 whānau members engaged in the programme.

» In year two, from July 2020 to June 2021, a total of 498 tamariki, 172 whānau and 450 whānau members were engaged.

» In year three, from July 2021 to June 2022, a total of 567 tamariki, 208 whānau and 677 whānau members were engaged.

Whānau survey completion rates were consistent across the three years and showed a high level of whānau satisfaction about the initiatives (over 90%). The survey data also indicated that many tamariki learnt new things about words, numbers and te reo, mātua observed an improvement in their tamariki attitude to learning and understood more about how their tamariki learn. Perhaps most importantly, the data showed how the majority of mātua engaged in the programme became more involved in the kura.

WHAT ARE THE SUCCESS FACTORS?

The critical success factors identified in Te Reo Matatini me te Pāngarau are:

» Kaupapa Māori approaches designed by kura, tamariki and whānau

» Project coordinator role pivotal

» Building whanaungatanga between kura

An independent evaluation report

TE REO MATATINI ME TE PANGARAU CRITICAL SUCCESS FACTOR ONE:

Kaupapa

Māori approaches

designed by kura, tamariki and whānau

All kura commented on how liberating, exciting and scary it was to work with the coordinator to lead and design bespoke kaupapa Māori literacy and numeracy programmes that best meets the needs and priorities of tamariki and whānau. The approaches were diverse, designed and led by Māori, for Māori. Kura appreciated the opportunity to exercise rangatiratanga and agency in this kaupapa while also acknowledging the critical role the coordinator played in terms of supporting them throughout the journey to design, develop, deliver, monitor and evaluate their respective kaupapa.

The kaupapa Māori elements of the approaches were fit for purpose and included delivery in te reo in most kura and for kura kaupapa Māori, alignment to Te Aho Matua. The kaupapa Māori features included: The kaupapa Māori features included te reo me ōna tikanga, whakawhanaungatanga, manaakitanga (kai and koha for whānau), ngā kōrero ā kui mā ā koro mā (drawing on grandparents’ stories and knowledge), whakapapa, ngā mahi toi (visual arts), and ngā mahi a Hinerēhia (waiata, kapa haka, haka).

The diverse range of activities included:

» Whānau hāngarau challenges

» Pāngarau whānau evenings

» Community guest speakers speaking about their te reo journeys

» Integrating coding and robotics in te reo into the classroom

» Integrating digital skills to build literacy skills

» Supporting whānau to learn about maramataka to grow maara kai

» Run whānau cooking classes at kura and home-based challenges

» Holding whānau te reo lessons, including games and resource making nights and shared kai

» Extending reo domains outside of kura to practice using te reo at local café and library

»

Providing one to one reading lessons at kura, library visits and shared kai

»

Visiting local sites of significant e.g., pā sites, maunga (Aoraki), awa, marae

» Hikoi in the local ngāhere

Whānau commented on how much fun they had sharing in activities with their tamariki and other whānau at kura and taking these into the home. For many whānau Te Reo Matatini me te Pāngarau kaupapa was the first time they had engaged with the kura in a sustained way. They enjoyed the flexibility of the kaupapa in terms of being able to do activities at home, join in kura activities whenever they could, participating in games and friendly competitions, receiving and creating resources and games, and sharing kai. The kaupapa helped bring them closer to their tamariki, the kura and better understand the learning needs and aspirations of their tamariki. It also helped bolster whānau confidence to learn and use te reo, no matter how big or small their te reo Māori vernacular.

- Project coordinator 2022

“

I’m seeing people speak te reo for the first time, whānau speak to their kids in te reo. It’s a big thing for the kids to see their parents speaking te reo. Te tino hiakai ō ngā whānau, kite ana rātou i ngā hua i roto i ngā tamariki. Tino ū ana ngā whānau ki te kaupapa nei “

- Pouako 2022

“ I have been blown away by the creative initiatives each kura have carefully created and designed to achieve quality language and education outcomes for their whānau.“

An independent evaluation report

TE REO MATATINI ME TE PANGARAU CRITICAL SUCCESS FACTOR TWO:

Project coordinator role pivotal

Kura acknowledged how pivotal the coordinator role was to the success of the project. The diverse mix of kura size, type, location, depth of leadership experience and levels of whānau engagement meant a central role was critical to provide support at the chalk face and facilitate cohesion and collaboration between kura.

Kura described the coordinator as ‘highly skilled critical friend…a sounding board, matatau i te reo - pono ki te kaupapa, hardcase, understanding and empathetic, a weaver of people, there whenever we needed, critical to our success, a huge help because she’s been where we are and understands the realities, the headaches and challenges”. The coordinator brought a deep knowledge to the kaupapa, grounded in kaupapa Māori education passionate about supporting tamariki, kaiako and whānau.

The three pilot kura were based in Ōtautahi, had experienced tumuaki, and benefited from having a year to test and trial what worked, what didn’t work, and shared lessons learnt to help build their local community of practice.

The expansion of the programme in year two included the introduction of five new kura and four new regions including Nelson, Blenheim, Dunedin and Invercargill. The coordinator was appointed and worked closely with these kura to provide intense support, particularly for the three kura with first-time principals. The role required intensive guidance and mentoring to the new kura. Smaller kura with a small staffing also required extra support to get things up and running.

The coordinator noted the need for guidelines to better support new kura that are designing and implementing literacy and numeracy initiatives. For many kura, this was the first time they have had the freedom and funding to design a programme from scratch.

The inclusion of two rurally based kura in year three added another layer of complexity to the kaupapa. Both of these kura were more geographically isolated and significantly impacted by staff retention, staff capacity and the COVID pandemic. The coordinator worked with both kura to provide support and leverage the lessons learned from the past two years of implementation. Progress in both kura happened at the pace at which whānau were ready to engage.

“ He poutautoko a [ingoa] ki ngā mahi katoa o te hōtaka. Ko te whāinga mō te wāhanga kei te tū mai e āhei ana ia ki tae atu ki te mātakitaki me te arotake hei hoatu ngā whakaaro me ta tautoko. He taura hono ia. He kanohi kitea ki te whānau.“

- Pouako, Ōtautahi

“Always checking in, communicates frequently, informs us on expectations and timeframes, her willingness to support us in all ways she can, she is awesome.“

independent evaluation report

TE REO MATATINI ME TE PANGARAU CRITICAL SUCCESS FACTOR THREE:

Building whanaungatanga between kura

At the end of 2019, Te Pūtahitanga o Te Waipounamu facilitated a workshop that brought together the eight kura involved in the kaupapa at that time. Feedback on the workshop was overwhelmingly positive. Kura appreciated the opportunity to come together, share learnings, challenges, opportunities and to meet each other ā kanohi (face-to-face). It was also an opportunity to celebrate te reo me ōna tikanga and the brilliance of tamariki and whānau – as evidenced by the presentations shared by each kura.

All agreed that coming together was essential for the kaupapa moving forward because it reenergised, inspired and motivated kaiako, pouako and tumuaki to continue their Te Reo Matatini me te Pāngarau journey.

While the COVID-pandemic resulted in the cancellation of planned workshops in 2020 and 2021, the coordinator worked hard to build relationships between kura, particularly between the tuakana and teina kura to foster relationships and the sharing of learning and expertise in particular subject areas. For example, two kura in Ōtautahi joined a wānanga in Murihiku to support the delivery of hāngarau based activities with tamariki and whānau.

“The wānanga, yeah. Harikoa ana taku ngākau. To realise I wasn’t alone in my wero. Hearing others talk about their challenges, the āwangawanga, the hapa, how they’d do things different and better –you know, learnings. I came away refreshed, well invigorated even, because you know, you don’t really know till you start talking with others and hearing how they do things. And then you think, oh yep, we’re not doing too bad after all. [laughter]“

- Pouako, Te Waipounamu

independent evaluation report

BARRIERS TO SUCCESS FOR TE REO MATATINI ME TE PANGARAU

Kaiako retention and kura resourcing creates sustainability challenges

Kaiako retention and kura resourcing was a challenge prior to COVID-19, however the pandemic reinforced this as kura kaupapa Māori, Wharekura and kura Māori struggled to support tamariki, whānau and staff through the pandemic.

All of the kaiako and pouako leading Te Reo Matatini me te Pāngarau do so on top of their day-to-day mahi - with no extra benefits or remuneration. Often, this work is delivered after school hours in the evenings as well as during the weekends and holidays. The workload also requires kaiako and pouako to set aside time for preparation, planning, review and evaluation. While kaiako and pouako enjoy the work and can see the fruits of the mahi, they expressed varying levels of fatigue and stress. The funding goes directly towards supporting the numeracy and literacy learning activities for tamariki and whānau helping to pay for resources (digital, paper, kai, fieldtrips, guest speakers.

In a few kura the programme is led and driven by the tumuaki due to a lack of staff capacity, three kura realised this was not sustainable and new kaiako or pouako picked up the leadership. All kura are balancing resources to ensure the programme is sustainable. Some kura reported how kaiako and pouako were fatigued due to workload demands and the long tail of Covid. As a result, some kura were challenged by keeping this kaupapa fresh and fun for tamariki and whānau.

The issue of ongoing sustainability was a concern voiced strongly by all kura. While they understood funding for this initiative was time-limited, all kura and whānau were hugely supportive of the programme continuing in order to build on the positive momentum created as a result of the approaches – as well as reclaim the ground lost as a result of the pandemic.

“Our reality. We are a small kura. We can’t get relievers, hard for us to attract kaiako Māori. They get snapped up and paid more to teach in mainstream kura. So I teach classes and lead our literacy programme. That’s the only workable solution we have.“

- Tumuaki, 2022

Generic monitoring and reporting tools do not align with bespoke innovative approaches

The project coordinator and kura have consistently reported on the challenges associated with administering the quarterly tamariki and whānau online surveys. Feedback from kura is that whānau don’t engage well in surveys, let alone online surveys. It has been a constant challenge to encourage whānau to complete the surveys consistently. While there have been overall improvements in the survey completion rates, the response rate to the majority of survey questions is relatively low. As a result, the survey data provides an indicative data set rather than robust and definitive data.

Moreover, the survey questions are generic which does not align with the unique and innovative features of each approach. This has made it difficult for kura to capture feedback on the innovation, opportunities and challenges unique to their approach.

The surveys also require someone to administer and analyse the results which is a challenge when the kaiako and pouako driving the initiative are already pressured for time. The frequency of reporting was also an issue for some kura due to resourcing constraints. These kura would like less frequent written reporting and electronic surveys, and more opportunities to orally report on the progress made in their respective kura.

BARRIERS TO SUCCESS FOR TE REO MATATINI ME TE PANGARAU

“Administering the survey monkey can be hōhā. It comes around fast, and you know when you’ve got a million and one other things to do, it can create pressure and stress. Our whānau prefer face to face kōrero rather than an online surveys, so it’s been difficult for them to engage with.“

- Kaiako and Pouako, Te Waipounamu

OVERALL IMPACT

The findings showed how Te Kete Aronui and Te Reo Matatini me te Pāngarau have contributed to the achievement of the intended numeracy and literacy outcomes and impacts for tamariki and whānau in Te Ika a Māui and Te Waipounamu. Table 8 outlines the intended social impacts, measures and indicators and the results as reported by partners and kura.

TABLE

8: NGĀ URI WHAKATUPU INVESTMENT: INTENDED SOCIAL IMPACT MEASURES AND ACTUAL RESULTS

Measures & Indicators Results17

Social Impact Results

Outcome: Improve and promote the educational achievement of tamariki Y1-Y8 in South Auckland, Hamilton, Wellington and Christchurch

#Tamariki participants

#Whānau participants

% Increased positive engagement

% Increased confidence to read and write

% Increase confidence to speak

% Improved reading and writing levels

% Improved reo-āwaha and reo-ātuhi

More than 1,600 tamariki and 1,000 whānau engaged

Over 80% positive engagement

Over 80% tamariki reported increased confidence to read, write and speak in te reo and English

Over 85% whānau reported tamariki enjoyed learning more now

Te Kete Aronui18 - 88% (104/118) showed positive changes in their literacy - 86% (74/87) showed positive changes in their numeracy skills

Whānau Māori engaging with their tamariki mokopuna

Tamariki are confident in reading their world, critically thinking and problem-solving.

Tamariki are achieving high standards of literacy and numeracy outcomes in kura and schools

Whānau are engaging with and learning alongside tamariki mokopuna in te reo Māori and English

Tamariki are building confidence in reading their world, thinking critically and problemsolving.

Tamariki are achieving success based on their own measures of success, including numeracy, literacy and whānau wellbeing outcomes.

Whakatupu: An independent evaluation report

Notwithstanding the significant impact the COVID-19 pandemic had on programme delivery, the findings demonstrate how both approaches have contributed to the achievement of the intended numeracy and literacy outcomes and impacts and made a positive difference to the lives of the tamariki, whānau, kaiako, pouako and kaiāwhina in Te Ika a Māui and Te Waipounamu.

Whānau are engaging with their tamariki and mokopuna in learning, many for the first time as a result of the programmes. Tamariki are building confidence or confident about numeracy and literacy; they are building numeracy and literacy skills, engaging and enjoying learning.

17 Based on aggregated Te Kete Aronui and Te Reo Matatini me te Pāngarau tamariki and whānau survey results 2019/20-2021/22 See also, Lakhotia, S., Mehr, E S (2021). Te Kete Aronui Report: Social Return of Investment Whānau Ora Commissioning Agency Unpublished Report

18 Based on results from tamariki numeracy and literacy assessments and kaiako observations.

An independent evaluation report

In Te Waipounamu, there has been a demonstrable improvement in the achievement of tamariki engaged in Te reo matatini me te pāngarau. Kura were resourced and empowered to design their own fit for purpose approaches to support the numeracy and literacy needs of tamariki and whānau. The activities are diverse, fun, interactive, engaging and draw on the strengths of tamariki and whānau in te reo.

Tamariki and whānau have participated in the programmes at unprecedented levels. Kura reported increased positive engagement by whānau and tamariki across the three years, increased confidence among tamariki to read and write (reo-a-tuhi) and to speak (reo-a-waha) and there are indications of improved reading and writing levels in te reo and english in some kura that use standardised English literacy and numeracy assessments and te reo assessments to measure progress.

Moreover, whānau are engaging in learning with tamariki, they are building confidence to learn and speak te reo matatini and te reo pāngarau, they are building more positive relationships with kura and are valuing education and learning for themselves and their tamariki.

The findings showed how positive results were achieved in established kura with bigger student rolls in urban areas. This was largely attributed to economies of scale e.g., access to more resources, a bigger pool of kaiako, kaiāwhina and pouako (although it can still be a struggle for urban Māori medium kura to attract staff); and having more experienced principals and senior leaders.

In smaller urban or smaller rural kura, the resourcing challenges are ever present which means progress can be slow or rather progress happens at the pace at which staff, whānau and tamariki are comfortable or have capacity to support and engage in.

“I really like reading books, and I’m good at it. I read heaps now.“

- Tamariki, Te Ika a Māui

“Mum can understand what I’m saying in te reo now.“

- Tamariki, Te Waipounamu

“This has been really effective in bringing our whole whānau together and learning for fun.“

- Whānau, Te Waipounamu

- Coordinator, Te Waipounamu

“I have witnessed kaiako and whānau who emulate language and tikanga aspirations and invest so much time in growing powerful connections to produce better language and educational outcomes for our tamariki mokopuna, Ka mau te wehi.“

Whakatupu: An independent evaluation report

The social return on investment analysis reveals the story of change and value created by Te Kete Aronui (Lakhotia, Mehr, 2021). The most significant changes identified in the report include:

» Tamariki are more confident and motivated

» Tamariki enjoy learning

» Tamariki have improved numeracy and literacy skills

» Tamariki have improved comprehension and language skills

» Tamariki have a greater sense of achievement

» Tamariki have improved behaviour

» Tamariki and whānau feel supported

» Whānau understand and engage in tamariki learning more

» Whānau are happy and proud of tamariki progress.

The report identified five key tamariki outcomes and five key whānau outcomes that were achieved and contribute to increased education attainment, performance and attendance. These outcomes align with our monitoring and reporting analysis.

The five tamariki outcomes are: » Improved attendance » Increased confidence and motivation to achieve » Increased engagement and enjoyment in learnings » Increased willingness to interact with the teacher » Increased numeracy and literacy skills

The five key whānau outcomes are:

» Greater value placed on education » Improved communications in the home » Increased understanding of numeracy and literacy and childrens progress » Improved interactions with school

» Increase interest in education and child’s learning

The Te Kete Aronui approach and its focus on providing comprehensive and holistic support has reduced many of the barriers whānau face in supporting their tamariki with learning.

Overall, the findings show how both approaches have made a difference to the lives of tamariki and whānau beyond the intended numeracy and literacy outcomes. This kaupapa Māori investment has supported te reo matatini me te reo pāngarau to flourish in holistic ways supporting tamariki and whānau health and wellbeing.

Tamariki who once struggled to read and complete basic numeracy equations, can now read and engage in numeracy and literacy with confidence. They sit proudly in class and enjoy reading and learning activities. They are now reading books regularly, a few visit the library with their whānau outside of school hours or ask the kaiako for more books to read.

As a result of this programme, tamariki are learning more about whakapapa, pepeha, connection to whenua, awa and maunga, and building their identity and pride in who they are alongside whānau members. They are teaching te reo to whānau in the home and extending the domains (spaces) where they use te reo outside of the kura and the home. Tamariki are using technology to learn code in te reo, to build robotics, to create social enterprises. Whānau are learning about financial literacy, growing their own kai and participating in physical activities. The social, cultural and educational impacts of this programme are immense. In the absence of this programme, kura would not have the capacity and resource to provide this level of targeted support to tamariki and whānau.

Whakatupu: An independent evaluation report

“The Te Kete Aronui approach and its focus on providing comprehensive and holistic support has reduced many of the barriers whānau face in supporting their tamariki with learning.“

CONCLUSION

The evaluation findings point towards the positive outcomes and impact that can be achieved with targeted investment in two innovative, kaupapa Māori, Whānau Ora and whānau centred numeracy and literacy approaches.

The strong partnership developed with Te Pūtahitanga o Te Waipounamu and Whānau Ora Commissioning Agency over a number of years is important to the overall success of the kaupapa. Both commissioning agencies draw on their trusted and deep networks (kura and partners) to deliver the kaupapa using kaupapa Māori, Whānau Ora and whānau centred approaches. Te Reo Matatini me te Pāngarau – designed by kura and delivered for kura; and Te Kete Aronui – a validated kaupapa Māori numeracy and literacy programme delivered by Whānau Ora partners to four communities have supported a previously unmet numeracy and literacy need for tamariki and whānau. What struck the evaluation team was that without this investment, many tamariki and whānau would still be struggling to access targeted numeracy and literacy support that meets their social, cultural and educational needs and aspirations.

The high levels of engagement in both approaches speaks to the high level of need and whānau interest. As reported by kura, whānau have engaged in this numeracy and literacy kaupapa at unprecedented levels challenging the notion that whānau do not care or do not prioritise the numeracy and literacy learning needs of tamariki and mokopuna.

“Without this investment, many tamariki and whānau would still be struggling to access targeted numeracy and literacy support that meets their social, cultural and educational needs and aspirations.“

In terms of what works for tamariki and whānau:

» Being engaged in ways and at a pace that works for them and not giving up if whānau drop off the radar for a period

» Whānau Ora and whānau centred holistic approaches that support numeracy and literacy outcomes for the whānau and seek to remove barriers to learning

» Validated approaches like Te Kete Aronui

» Kura designed approaches like Te Reo Matatini me te Pāngarau

» Kaupapa Māori activities that support numeracy and literacy teaching and learning inside and outside the kura

» Fun activities that utilise games, competition, technology, creation of resources, incentives (prizes) and regular celebrations.

What makes an effective programme:

» Delivery flexibility, consistency and adaptability

» The right assessment and learning tools to measure progress and accelerate numeracy and literacy outcomes in English and te reo Māori

» A group of pouako or kaiako driving the programme so that it is not on the shoulders of one person

» Resource to support sustainable delivery

» Fit for purpose monitoring and reporting

What investors should think about when investing in numeracy and literacy in Māori medium and kaupapa Māori settings includes:

» The potential for positively impacting whānau members beyond the tamariki enrolled

» The greater likelihood of long-term educational success (and other impacts like earning potential) of Māori when they have literacy and numeracy confidence from an early age

» Likely resourcing differences when investing in urban versus rural settings (to address staff retention and capacity)

The potential for this programme to be building tamariki skill with digital technologies.

Whakatupu: An independent evaluation report

“Ko te whakaaro kia āio ngā whānau i roto i tā rātou ake reo.”

- Pouako, Tuia Te Mātangi

APPENDICES

KURA CASE STUDIES

The case studies provide insight to the experiences of tamariki, whānau, tumuaki, pouako and kaiāwhina. The evaluation team were privileged to speak directly with tamariki and whānau at one kura, their quotes are included in the case study.

TE WHAREKURA O AROWHENUA – HE KURA HEI KĀINGA

“Whānau who have had a negative experience with kura are bringing their kids back to kura and realising kua tino huri te ao. Ia hui kī ana te whare i te whānau, kua tino hono te whānau.”

- Tumuaki

Te Wharekura o Arowhenua is the only Te Aho Matua kura kaupapa in Southland. The kura provides Māori medium education for Years 1 – 13. The school’s vision, ‘Tū pakari, tū rangatira hei raukura mō tō iwi’ guides educational initiatives to support their students and graduates. ‘Arowhenuatanga’ is the school’s acknowledgement of their cultural uniqueness as Māori living as Māori in Invercargill. Whānau create and maintain kura-iwi relationships with mana whenua and Waihōpai, Invercargill.19

Approach

“I te timatanga i tino ngako mātou ki te reo me te pāngarau, then we widened it, we looked at what our kids and whānau wanted, their apirations. Whakatinana ō moemoeā. Ngākaunui ana ngā tamariki ki te mahi hangarau matahiko.”

- Pouako

He kura hei kāinga is based on the concept of extending the learning in the akomanga into the home. The programme is run on a weekly basis and has utilised digital technologies to communicate and collaborate with whānau and to practise and consolidate specific skills learnt to develop their pāngarau skills and te reo. Tamariki and their whānau are empowered to demonstrate and share their learning through the digital portfolio, Seesaw.

Whānau have been able to explore a variety of powerful tools to generate multimedia work such as images and animations and voice clips to transfer their new learnings of te reo Māori me te pāngarau and share these. Weekly reo classes and kai are held with whānau to support their reo journey in order to better support their tamariki.

It has taken two years to evolve and refine the kaupapa which is driven by school leadership, including the tumuaki, and supported by pouako, kaiāwhina and whānau. Te Aho Matua at the heart of all teaching and learning.

Whakatupu: An independent evaluation report

In 2021, the kaupapa included a specific focus on a cohort of year 5-6 akomanga using digital technologies.

In 2022, tamariki moved towards literacy and numeracy through enterprise, producing products that involved graphic design (procreate app) and promotion of te reo. This kaupapa was named, Te Paewai o te Rangi which is the place where the ocean meets the sky, and where the sun sets and rises. This is the transition point where Tāne travelled on his quest to ascend into the heavens and to the place ‘Te Whānau Mārama’ in the heavens.

The evolution of the programme to build on and extend tamariki interests while maintaining reo classes for whānau is a key feature of He kura hei kāinga.

What worked well

The findings showed that what worked well for tamariki and whānau included:

» Full kura buy in to support the programme from the tumuaki, pouako and kaiāwhina

» Kura listened to whānau who said they do not like online learning and adapted to provide hybrid learning options

» Evolution of student enterprise kaupapa - ‘Te Paewai o Te Rangi’

» Shared kai is enjoyed by all – tamariki, whānau and staff

» A number of whānau have also pushed themselves out of their comfort zone to challenge their reo in an advanced group.

Challenges and improvements

Kura closed for two weeks to conduct online learning, then two weeks of hybrid learning. A mix of online and onsite learning for students of critical and essential workers and to relieve those whānau that could not cope with online learning for all their tamariki (these are families of more than five tamariki).

The biggest challenge has been the impact of COVID-19:

» not being able to continue onsite reo sessions

» whānau not requesting online session

» some pouako not confident to conduct their sessions on-line and delivering to multi-levels

» COVID impact, reducing the opportunity for new whānau to attend sessions.

The suggested improvements include:

» more training for lead pouako to keep them inspired and energised

» adding a grammar class for fluent students

» adding a tikanga, karanga, whaikōrero class for adults

» providing devices to some whānau that still do not have access to these.

Outcomes achieved

“Ko ngā whānau kāore he reo, kua whakatō he kakano, kua puawai te reo i roto i te kāinga. Whāngai i te reo ki ngā matua, ngā whānau, he reo kōrero ki te kāinga. Ko te kāinga te kura tuatahi... He wāhi haumaru mō ngā whānau ki te whakahua i te reo, there’s no language police here “

The outcomes achieved as a result of ‘he kura hei kāinga’ include:

» Consistently high and positive whānau and tamariki engagement. There is a waitlist to enter he kura kei kāinga.

» Whanaungatanga and the strengthening of relationships between the kura and whānau. Some whānau have been involved in the programme since 2019. They are tuakana encouraging and support new whānau to use te reo.

» Students are developing confidence using digital technology i.e., robots for coding; using circuit boards for engineering; using ipad apps for graphic design and production.

» Increased whānau te reo use and proficiency at kura and the home

» Increased whānau te reo use in domains outside of the kura e.g., whānau café visit to order coffee

» Increased whānau demand for extra te reo classes.

Tamariki and whānau journeys

Pseudonyms are used in these narratives. They are designed to highlight some of the shifts that have occurred in the lives of tamariki that have engaged in the kaupapa.

Context Positive shifts Future aspirations

Whānau

» Kāpua, 8 years old attended kōhanga reo prior to coming to wharekura. She has been coming to He kura hei kāinga for two years. Her māmā is Māori and learning reo, Pāpā is Pākehā and has no reo. She has two older sisters who attend mainstream schools.

Tamariki

» Tehuia, 9 years old has been coming to He kura hei kāinga for 2 years with her pāpā and siblings.

» Matariki, 9 years old has been coming to He kura hei kāinga for 2 years with his pāpā.

“I wanted to learn and support her. Now I help out as a kaiāwhina. My older girls come, and they enjoy it. My partner, he doesn’t come to He kura hei kāinga, but He kura hei kāinga is coming to his home. He is not participating but he understands why and is supportive. At home it’s not just us speaking te reo, it might only be one word or a few but our whānau use te reo, from no reo to some reo. My father is Māori and the generation that don’t have te reo. He is so proud to see his mokopuna speaking te reo”.

“We want to keep growing our reo no matter how small. Our daughter. She is thriving. Her older siblings see it too and they want a part of that”.

“Te mahi kaitahi – there’s always yum kai. I mahi mātou i te mahi karetao (robotics) me te coding. Its heaps of fun. Matua makes it fun. You know, with kēmu and activities”.

“We started our own pākihi to raise money for our trip. And my dad he has his reo class. So, I help him out too”.

“E hiahia ana au ki te mahi karetao and having fun”.

“I wanna be rich with my dad”.

Whakatupu: An independent evaluation report

TE KURA KAUPAPA MĀORI O TUIA

TE MATANGI

“Te hiahia o ngā tamariki ki te kōrero, kia ū ki te reo, te ū ki te kōrero ki o rātou matua. Ko rātou e arahi ana i ngā mahi, kapa haka mai, akonga mai. Te toenga o ngā tauira ka whai i ēnei ahuatanga“

“I’m seeing people speak te reo for the first time, I’m seeing whānau speak to their kids in te reo. It’s a big thing for the kids to see their parents speaking te reo” - Pouako, Tuia Te Mātangi

Te Kura Kaupapa Māori o Tuia Te Matangi is in Richmond, Nelson. It is the only kura kaupapa Māori Te Aho Matua in this rohe. Te Aho Matua philosophy guides school wide practices. The kura provides an education for Years 1 – 13, that has impacted on an increase in the roll.

The establishment of the kura was a passionate desire for Te Tau Ihu Tangata Whenua - Ngāti Kuia, Ngāti Koata, Ngāti Rārua, Te Ātiawa, Ngāti Tama, Ngāti Apa ki te Rā Tō, Rangitane o Wairau, and Ngāti Toa Rangatira for more than 30 years. Cloaked by those aspirations, in 2004 a working group was re-formed to progress this moemoeā. As the journey took on momentum, a name was gifted to the kaupapa, Tuia te Matangi.

As guided by those who worked so hard to bring this kaupapa to fruition, the kura strives to develop young people in their inherent potential to be leaders, exemplars, and contributors within their communities, and to their iwi, hapū and whānau.20

Approach

Putaki Mai Kia Puawai

The original programme was led by the tumuaki and focused on increasing whānau engagement and fostering the use of te reo in the home through hāngarau. The programme set whānau challenges / activities to do in the home. These challenges were complemented with wānanga. The programme has given the opportunity for the kura to open an akomanga hāngarau and invite experts from their community to tautoko the running of the akomanga and teaching technology capabilities through the delivery of te reo.

Te Reo Āio

In line with whānau and tamariki feedback, the programme shifted to focus on te reo hākinakina, because that is what the kids loved. It also empowered the parents to speak a bit of te reo when their kids played sport, even small kupu like ‘karawhiua’.

Through this, the pouako realised some of the whānau and kids did not know their whakapapa. The assumption being that because the whānau and kids that come to kura kaupapa they already had this. However, many of the kids came into the kura from mainstream schooling. This led to learning about whakapapa and creating reo resources for whānau. The programme then became Te Reo Āio,

Te Kura Kaupapa Māori o Tuia Te Mātangi Review

from: https://ero govt nz/institution/628/tkkm-o-tuia-te-matangi

In 2022, a total of six whānau and 12 tamariki engage in Te Reo Āio. The 2022 focus is building confidence to be active learners and practitioners of reo and tikanga focused on the importance of whakapapa. Each fortnight the whānau meet, begin with pepeha. Whānau learn simple sentence structures and the negation of these sentences. A pukamahi is used to support whānau in learning and to consolidate what they have learnt. This is also a little taonga as they continue on their reo journeys.

What worked well?

Whānau engagement using technology

• Zoom was easier for whānau to engage, many live a distance from the kura and have young children.

• A facebook group is where whānau post notices, videos, mahi kāinga.

• Technology supported whānau to reengage post COVID because they didn’t have to worry about masks, vaccination status etc. All of the whānau had a device and internet access.

Increased te reo use

• Whānau are speaking te reo in the home – kia ūkaipō anō te reo.

• Tamariki that were too shy to speak in front of people are now leading sessions. They are tuakana

• Increased confidence and pride in te reo.

Me aro ki ngā wawata o te whānau – listening to whānau aspirations

• Going back to the basics of te reo and whakapapa was a success. Whānau learned pepeha which gave them more confidence in who they are as Māori.

• Building confidence and self-belief was critical starting with whānau knowing who they are in the language, and having the confidence to share that

• Tamariki feeling supported to learn and speak te reo outside of kura.

Whakawhanaungatanga – relationship building

• He mea nui te whakawhanaunga – this is when the real learning happens, in that safe space, starting with aroha and empathy

• Gaining trust with tamariki and whānau is critical and takes time

• Modelling what is expected of tamariki and whānau

• Whānau enjoyed the shared kai and fieldtrips at the end of every term to celebrate each other and the kaupapa.

Whakatupu: An independent evaluation report

Whakatupu: An independent evaluation report

What were the challenges and improvements?

The kaupapa took a backseat in 2021 because of COVID19. The kura was able to provide care packages to whānau - ‘Ko te mea nui ki a noho haumaru te whānau’. But there were also positives, like whānau proactively using facebook and zoom to engage.

Some tamariki wanted to join Te Reo Āio but their whānau never reached out or could not support the kaupapa for a number of reasons.

Engaging with whānau when the programme started was hard. The kura noted some hesitancy and apprehension from the tamariki because the tumuaki was originally the kaiwhakahaere. The shift in tamariki and whānau engagement was noted when the kaiako took on the role.

An ongoing challenge for small kura is that the mahi falls on the kaiwhakahaere, ‘i te ata, i te pō’. There is no 9am-5pm role, it is demanding because the kaiwhakahaere is invested and wants the tamariki to succeed in their own terms.

Outcomes achieved

Led by a dynamic kaiwhakahaere who is also a pouako at the kura, Te Reo Āio has supported tamariki and whānau to learn about their whakapapa and identity in a safe and supportive space. In connecting to who they are as Māori they have increased their knowledge and use of te reo inside the kura and at home.

The confidence and self-esteem of tamariki and whānau has increased. Tamariki who were once too shy to stand and speak in te reo are now leading classroom activities. Whānau who lacked the confidence to engage in kura activities have been attending Te Reo Āio classes every fortnight either online or in person.

Future aspirations

“My dreams are that my tamariki know who they are, they’re proud of it, not afraid to be able to share it. Understand that they can succeed in a Māori world, not just a pākehā world. Succeed in themselves being Māori. I just want to feed them what I had to offer and be better and want more - Kia ū tonu ki te reo Māori i waho i te kura.“

The kaiwhakahaere wants to see the tamariki succeed, whatever that looks like for them and work with whānau to determine the next steps based on their needs. Additional resource in the form of a kaiāwhina or another kaiwhakahaere would help expand the programme so that more whānau and tamariki can participate.

AROWHENUA MĀORI SCHOOL

“One of our biggest challenges is how do we get whānau to engage? We still don’t have the answer, and still struggle with whānau engagement. Before we can teach numeracy and literacy, we need to make sure the whānau are well.”

- Tumuaki

Located in Temuka, Arowhenua Māori School was established 127 years ago beside the Kati Huirapa marae. The school's vision is ‘He anga onamata - he aronga anamata (Ancestrally driven - future focused)’. This vision is underpinned by the core values of manaakitaka, whanaukataka, kaitiakitaka and rakatirataka. The school’s intention is for ākonga to leave with a strong sense of identity as Māori, confidently and competently able to use this lens to guide their future learning, connections and success in the global world.21

In 2017 the school adopted Te Marautanga o Aotearoa, using Level 2 bilingual/Māori language instruction. The school’s key priorities focus on ongoing development of the new curriculum and school vision, including a strong emphasis on establishing fluency in te reo Māori across the school. To support this, teachers have undertaken professional learning of te reo matatini (te reo Māori literacy, in te-reo-a-waha and tuhituhi) and collaborative teacher practice.

Context

Temuka has a strong colonial history with prevailing colonial attitudes about Māori and cultural diversity. This has brough many challenges in terms of creating a supportive environment for te reo me ōna tikanga to thrive. Many whānau Māori are early in their journey of connecting to their identity as Kai Tahu and other iwi.

The kura is a new build. Previously, the kura was housed in prefabricated building for years because the old school buildings were not in great shape. The kura has a strong, proud and long history as a native school. The kura provides cooked lunch to the kids every day.

Launched in 2021, ‘Nōku Te Ao’ is led and run by the tumuaki due to a lack of staff capacity. This has been a constant juggle but a rewarding journey because the tamariki are given opportunities to learn that they would not otherwise have access to.

As of June 2022, the roll is low which means the tumuaki is also a teaching principal. Managing and attracting staff is always a challenge, particularly when competing against mainstream schools for teachers that have te reo me ōna tikanga capabilities.

Challenges

“COVID-19 has had a far-reaching negative effect on maintaining a solid community. We experienced whānau non-engagement, effects of stress and anxiety and a sense of negativity. We reviewed this and for the latter part of term one decided to change our focus and prioritise well-being. We were set to hold a whānau hui to get ‘nōku te ao’ back on track and to review our goals with our whānau but again, we were set back by a lack of whānau engagement. We will keep trying. Three weeks ago, all of our teaching staff and one of two kaiāwhina were away due to COVID-19. Eventually resulting in closing the kura due to no staff to run the kura.”

- Tumuaki

Whakatupu: An independent evaluation report

Whānau engagement with the kura was low throughout the pandemic as whānau had bigger things to worry about. Engaging with the school was not their number one priority and the progress of Nōku Te Ao reflects this. The kura has learnt that programmes like this must be delivered at the pace and cadence at which the whānau are able and ready to engage. The pandemic created big divisions within the whānau between the vaxxed, non-vaxxed and anti-mandate whānau. This divided the kura and made tough times even harder. The net result was the kura lost students and their whānau. In addition, whānau did not engage well in online learning and planned activities and a fieldtrip were cancelled. This created a perfect storm of illness, tension, anxiety, fear and apathy that meant the majority of whānau disengaged from kura and ‘Nōku Te Ao’.

What have been your successes?

The kura reported on how the opportunities provided by the programme have been invaluable. They were able to provide whānau with kai, rauemi, and kēmu, particularly during COVID. Without these opportunities and resources, tamariki would not have made positive shifts in their learning and wellbeing.

Future aspirations

A local fieldtrip to show the tamariki their whenua, whakapapa, to connect with the land in Tekapo and Aoraki. To continue to provide opportunities to whānau and tamariki to learn together in a positive space. The kura want to give tamariki everything they deserve.

MINISTRY OF EDUCATION HEI RAUKURA MŌ TE MOKOPUNA STRATEGY

Hei Raukura mō te Mokopuna seeks to:

I. Reclaim the meaning of te reo matatini to reflect its original intent and clarify its contribution to Te Tamaiti Hei Raukura (kia rangatira te mokopuna).

II. Emphasise the importance of te reo matatini for meeting iwi aspirations for language revitalisation and regeneration (kia rangatira te iwi).

III. Imagine the wāhanga ako of Te Marautanga o Aotearoa as tīrewa ako in Te Tīrewa Marautanga from understandings gained from the development of tīrewa ako for reo-ā-waha, pānui, tuhituhi, and pāngarau (kia rangatira te reo).

IV. Emphasise the critical role of kaiako, both as learners and teachers of ngā reo matatini to maximise mokopuna success in the classroom and in their everyday lives (kia rangatira te kaiako).

V. Promote opportunities for mokopuna to make sense of their experiences and to make connections with their stories, cultures and communities, through increased access to and understanding of the role of te reo matatini and āngarau.

VI. Propose te reo matatini as an important leverage to social inclusion, and to take concrete measures to ensure that everybody has the opportunity to obtain the skills required for them to participate fully in society.

VII. Ensure that whānau and kaiako have a primary role in the educational development of mokopuna through te reo matatini and pāngarau and that the support services in schools complement the work of the classroom teacher.

It is envisaged that from these goals, a set of actions will be developed and implemented as part of the work program in transitioning into Te Tīrewa Marautanga, Te Tīrewa Ako and Te Tīrewa Mātai from Te Marautanga o Aotearoa.

An independent evaluation report

EVALUATION INFORMATION SHEET

What is the purpose of the evaluation?

The purpose of the evaluation is to show the impacts of the investments made in Te Reo Matatini me Te Pāngarau

The evaluation will provide:

• an understanding of how well the programmes are working and identify critical success factors

• an assessment of the impacts of the investments made for tamariki, rangatahi and their whānau through the programmes.

Who is conducting the evaluation?

The evaluation is being conducted by AIKO on behalf of Te Pūtea Whakatupu Trust.

Evaluation Team members:

• Timoti Brown - Ngāti Kahungunu ki te Wairoa, Rongowhakaata. timoti.brown@gmail.com. 027 911 8000.

• Dr Chelsea Grootveld - Ngaitai, Ngāti Porou, Whānau-a-Apanui, Whakatōhea. chelsea.grootveld@gmail.com. 021 911 854.

• Roimata Hanchard – Ngāpuhi. Roimata@litmus.co.nz

How is information being gathered?

Information will be gathered in three ways:

• 45-60-minute interviews or focus groups with staff, whānau, tamariki, mokopuna

• data analaysis of administrative data and other relevant information.

What are the benefits of this evaluation to whānau?

The final evaluation report will demonstrate how the programme has impacted tamariki, mokopuna and whānau and the difference made to their lives.

What are my rights? It is your choice whether or not you wish to take part in the evaluation. If you do not wish to participate, you do not have to give a reason. Your relationship with Te Pūtahitanga will not be harmed.

If you agree to take part, you will be asked to read and complete a consent form. The evaluation team will also verbally outline your rights.

Who do I contact with questions or concerns?

If you have any questions or concerns, please contact one of the evaluation team members.

EVALUATION CONSENT FORM

I understand that:

• My participation is voluntary, and I can withdraw at any time.

• My name or identifying information will not be included in the evaluation without my permission. If I work for an organisation, the type of organisation I work for may be identified.

• My relationship with Te Pūtahitanga o Te Waipounamu will not be affected through my participation.

• With my permission, the discussion will be audio recorded, and may be transcribed.

• I have the right to request a copy of the audio or transcript of my discussion.

• Digital recordings, notes, and summaries will be stored securely at AIKO. Hard copies of stored information will be destroyed after three years.

I have read the information sheet and consent form and been given the opportunity to ask questions. I give my consent to participate in this evaluation.

Name: Signature: Date:

Whakatupu: An independent evaluation report

An independent evaluation report

INTERVIEW SCHEDULE: STAFF

Set up, rapport building

• Introductions, explain evaluation, informed consent, survey, answer questions.

• Invite general discussion about role and responsibilities.

Assessing implementation

• Tell me about your Te Reo Matatini me Te Pāngarau kaupapa/programme?

o How has it developed in the last three years?

o What are you trying to achieve?

o What were the enablers of success for your kura/community/campus?

o What have been the biggest challenges and how did you overcome these? (i.e., COVID-19, lockdowns, school closures).

• How was your literacy and numeracy programme implemented in your kura/ community/campus?

• How was success and quality of implementation measured and to what extent were these measures achieved?

• What are you most proud of about this kaupapa?

Assessing early impact

• In what ways did tamariki and whānau experience positive literacy/numeracy outcomes?

o How did you evidence these changes and outcomes?

o What is required to make this successful and sustainable?

Assessing the counterfactual

• What would happen to your tamariki, whānau, community, if this literacy and numeracy programme was not available?

• Invite other comments, thank participation, and close.

INTERVIEW SCHEDULE: WHĀNAU,

TAMARIKI, MOKOPUNA

Set up, rapport building

Assessing implementation

• Introductions, explain evaluation, informed consent, survey, answer questions.

• Mihimihi

• Tell me about yourself

• How long have you been involved in Te Reo Matatini me Te Pāngarau programme?

• o Why did you get involved?

• o What did you want to get out of it?

• What do you like about the programme? Why?

• What do you think could be improved?

Assessing early impact

• What difference has the programme made to you, your whānau, your tamariki, particularly in regard to literacy and numeracy?

• o Have you achieved what you wanted to achieve?

• o How do you know?

• Do you think this programme could be used to help whānau and tamariki in other kura/communities? Why? Why not?

• What do you think is needed to make this programme successful and sustainable?

Whakatupu: An independent evaluation report

• What would happen to your tamariki, whānau, community, if this literacy and numeracy programme was not available?

• Invite other comments, thank participation and close.

Assessing the counterfactual

REFERENCES

END NOTES

1 Lakhotia, S., Mehr, E.S. (2021). Te Kete Aronui Report: Social Return of Investment. Whānau Ora Commissioning Agency. Unpublished Report.

2 Te Pūtea Whakatupu Trust. (2017). Te Rautaki. Rua mano tekau mā whitu ki Rua mano rua tekau mā rua. Unpublished paper.

3 KPMG. (2018). Te Pūtea Whakatupu Trust Business Case Support. Unpublished report.

4 Ministry of Education. (2021). Strengthening literacy and te reo Matatini. Briefing Note. March 7 April 2021.

5 McNaughton, S. (2020). The literacy landscape in Aotearoa New Zealand. Office of the Prime Minister’s Chief Science Advisor

6 Sutherland, D. (2019). Literacy Proficiency among Students in Aotearoa-New Zealand: Why the Gap between Māori and Pākehā?”. In Leiden, Perspectives on Indigenous Writing and Literacies. The Netherlands: Brill.

7 Ministry of Education. (2018). Achievement and progress in mathematics, reading and writing in primary schooling, Analysis of e-asTTle assessment data, 2011 to 2016. Ministry of Education, New Zealand

8 Measured by the proportion of students from deciles 1–3 and deciles 8–10 schools who left without NCEA Level 1 or better

9 Tanielu, R., Barber, P., & Wijeysingha, V. (February 2020). Tangata Whenua, Tangata Tiriti, Huia Tangata Kotahi: State of the Nation Report. The Salvation Army, Social Policy & Parliamentary Unit.

10 Ministry of Education. (2022). Hei Raukura Mō Te Mokopuna Strategy.A Te Reo Matatini and Pāngarau Strategy. Ministry of Education, Wellington.

11 Te Pūtahitanga o Te Waipounamu. (2019). First Report to Te Pūtea Whakatupu Trust. May 2019.

12 Whānau Ora Commissioning Agency. (2021). Te Kete Aronui. Return on Social Investment Report. May 2021.

13 Te Maro, P. (2013). Ko Te Kete Aronui Report: Evaluative Research of the Te Kete Aronui/Kip McGrath Literacy and Numeracy Programme. Unpublished Report.

14 Te Puni Kōkiri. (2022). Whānau Ora and Whānau Centred approaches. Retrieved from: https://www.tpk.govt.nz/en/ngaputea-me-nga-ratonga/whanau-ora/whanau-ora-kaupapa

15 Aiko Consultants Limited. (August 2020). The effects of COVID-19 on Māori education outcomes. Te Pūtea Whakatupu Trust.

16 Te Pūtahitanga O Te Waipounamu (2022). Te Reo Matatini me te Pāngarau tamariki and whānau statistics 2019-2022. Email correspondence provided on 8 Sept 2022. Unpublished figures.

17 Education Review Office. (2020). Te Wharekura o Arowhenua Review Report. Accessed 01/08/22 from: https://ero.govt. nz/institution/4217/te-wharekura-o-arowhenua

18 Education Review Office. (2019). Te Kura Kaupapa Māori o Tuia Te Mātangi Review Report. Accessed 01/08/22 from: https://ero.govt.nz/institution/628/tkkm-o-tuia-te-matangi

19 Education Review Office. (2019). Arowhenua Māori School Review Report. Accessed 01/08/22 from: https://ero.govt.nz/ institution/3280/arowhenua-maori-school

20 Education Review Office. (2019). Te Kura Kaupapa Māori o Tuia Te Mātangi Review Report. Accessed 01/08/22 from: https://ero.govt.nz/institution/628/tkkm-o-tuia-te-matangi

21 Education Review Office (2019) Arowhenua Māori School Review Report Accessed 01/08/22 from:https://ero govt nz/ institution/3280/arowhenua-maori-school

Whakatupu: An independent evaluation report

Whakatupu: An independent evaluation report

MILESTONE REPORTS REVIEWED

» Te Pūtahitanga o Te Waipounamu. (2019). Te Reo Matatini me Te Pāngarau. May 2019.

» Te Pūtahitanga o Te Waipounamu. (2019). Te Reo Matatini me Te Pāngarau. July 2019.

» Te Pūtahitanga o Te Waipounamu. (2019). Te Reo Matatini me Te Pāngarau. October 2019.

» Te Pūtahitanga o Te Waipounamu. (2020). Te Reo Matatini me Te Pāngarau. Akuhata 2020.

» Te Pūtahitanga o Te Waipounamu. (2021). Te Reo Matatini me Te Pāngarau. Āpereira 2021.

» Te Pūtahitanga o Te Waipounamu. (2021). Te Reo Matatini me Te Pāngarau. Hurae 2021.

» Te Pūtahitanga o Te Waipounamu. (2022). Te Reo Matatini me Te Pāngarau. Āpereira 2022.

» Te Pūtea Whakatupu Trust (2020). Variation to Contract sent to Whānau Ora Commissioning Agency. 9 April 2020.

» Whānau Ora Commissioning Agency (2019). Te Kete Aronui quarterly report, Jan to March 2019.

» Whānau Ora Commissioning Agency (2019). Te Kete Aronui quarterly report, April to June 2019.

» Whānau Ora Commissioning Agency (2020). Te Kete Aronui quarterly report, Oct to Dec 2020.

» Whānau Ora Commissioning Agency (2021). Te Kete Aronui quarterly report, Jan to March 2021.

» Whānau Ora Commissioning Agency (2021). Te Kete Aronui quarterly report, 1 April to 30 June 2021.

» Whānau Ora Commissioning Agency (2021). Te Kete Aronui COVID-19 Response

Uri Whakatupu: An independent evaluation report