Report includes findings from Year 3 (of 3) - focus on student and whānau experiences

MartinJenkins has now completed a multi-year programme of evaluation and research of the PSKH policy for the Ministry. The evaluation was conducted over a number of years (beginning in 2014) and focused on the eight schools/kura that opened in 2014 and 2015 (not the PSKH sector as a whole).

Author(s): MartinJenkins (Martin, Jenkins & Associates Limited). Report prepared for the Ministry of Education.

Date Published: April 2018

Summary

Partnership Schools | Kura Hourua policy intent

The Ministry of Education (the Ministry) has a clear focus on improving student achievement, and employs a range of approaches to support the sector's efforts, including provision of strategic leadership, resources, and targeted interventions. Partnership Schools | Kura Hourua (PSKH) was a policy designed as an innovative addition to this mix.

A fuller description of the policy features and policy intent is available in previous evaluation reports. Key points include:

the policy was intended to 'operate in areas of significant educational challenge, and for those groups of students that the system has not served well. These were Māori and Pasifika students, students with special educational needs, and students from low socio-economic areas' 1

greater flexibility than state schools over aspects of governance and management, including staffing, approaches to teaching and learning, curriculum and qualifications

a clear and ambitious mission that distinguished the PSKH from surrounding state and state integrated schools

a Sponsor who was responsible for ensuring the PSKH meets learner achievement goals

accountability for outcomes was a strong focus

emphasis was also placed on the need for PSKH to be strongly engaged with parents/family/whānau and community.

High-level theory of change

A concise, high-level theory of change for the PSKH policy is:

IF schools have clear outcome-focused accountability, freedom to manage and govern, and support that is equitable to state schools

schools have clear outcome-focused accountability, freedom to manage and govern, and support that is equitable to state schools THEN they will develop innovative solutions that match local needs while still meeting high quality standards

they will develop innovative solutions that match local needs while still meeting high quality standards WHICH WILL attract students who have previously not been well served by the education system and will lead to equitable achievement outcomes for them.

This high-level theory of change is unpacked through a detailed intervention logic included as Appendix 3. The intervention logic acted as an important guide for the evaluation.

Evaluation design and focus

MartinJenkins has now completed a multi-year programme of evaluation and research of the PSKH policy for the Ministry. The evaluation was conducted over a number of years (beginning in 2014) and focused on the eight schools/kura that opened in 2014 and 2015 (not the PSKH sector as a whole).2

The original purpose of the multi-year evaluation was to assess the extent to which the PSKH policy delivered on its intended outcomes with regard to flexibility, innovation and student outcomes.3 In practice, each year of the evaluation had a separate focus, designed to explore key policy questions of interest to the Ministry of Education (the Ministry).

The specific focus of each year was determined through an annual scoping phase and was guided by a detailed intervention logic (Appendix 3). Each year's scope was determined by the Ministry following discussion with an Evaluation Working Group (EWG), comprising representatives from the Ministry, the PSKH Authorisation Board, and the MartinJenkins evaluation team.4 The Ministry and the EWG provided feedback on draft reports, as did schools/kura, however analysis and editorial decisions were made independently by MartinJenkins within the project scope.

Year 1 (reported in 2015) focused on innovation, in particular the lower levels of the intervention logic - the structural framework and the delivery component.

Year 2 (reported in 2016) focused on the approaches taken by PSKH to meet the needs of priority students, further examining the delivery component level of the intervention logic.

Year 3 (presented in this report) aimed to build insight into the outputs and short- and medium-term outcomes of the intervention logic from the perspectives of students and whānau.

Caveats

It is important to note:

the evaluation was focused on eight Round 1 and Round 2 kura only

the evaluation did not seek to compare the outcomes achieved by PSKH with outcomes achieved by other types of schools

the Year 1 and 2 findings draw heavily on the views of sponsors

the Year 3 findings incorporate the perspectives of students and whānau but low and uneven response rates mean the feedback is not representative (noting an excellent response rate was achieved for the Middle School survey of students).

Specific caveats for Year 3

MartinJenkins worked with the Ministry to refocus the final year of the evaluation (away from a primary focus on outcomes) because:

it was still too early to determine 'success': schools/kura were still becoming established, numbers of students that had received a 'full dose' of the PSKH intervention were low, and efforts were ongoing by the Ministry to define and agree contracted outcomes

emerging policy questions remained focused on understanding implementation.

Appendix 2 provides full details of the methods for Year 3, including data limitations and analytical decisions.

The Year 3 findings paint a positive picture but are based on data that has significant caveats. Low response rates to surveys and selection bias meant we were not able to examine student and whānau perspectives from all angles or across all schools/kura.

The overall response rate to the student survey was 47% (373 responses), but nearly all responses were from the two Middle Schools (336 responses or 90% of all student responses, giving a 90% response rate for the Middle Schools students). As a result, we only present feedback from the Middle School students (as a case study)

Whānau feedback was limited (33% response rate overall, 249 responses) and not evenly distributed across all schools/kura (of the 249 responses received from whānau, 124 or 50% were from Middle School whānau, this gives an indicative response rate for the Middle School whānau of 36%).

Almost no early exiter-parents (7 responses, 4% response rate) or graduates (15 responses, 4% response rate) responded to the surveys; feedback is not reported for these groups. 5

This meant we were not able to find out where students go after leaving PSKH, or why they leave.

In addition, outcomes are only discussed from the perspective of parents. This is because:

available achievement data was not reported due to the minimal time PSKH students at NCEA level have been in PSKH

no comparison cohort was identified to compare outcomes of the PSKH cohort

limitations in the administrative data:

attendance data was not sufficiently robust to be included



we were unable to compare quality of outcomes with outcomes that had been achieved at previous schools or to accurately identify where students went after exiting.

About the report

Part 1 of this report provides a brief overview of the evaluation's key messages and presents summary findings for each year of the evaluation. The differing focuses for each year of the evaluation limited our ability to draw overall conclusions. Full details of findings for Years 1 and 2 are contained in previous reports.6

This is the first time that Year 3 findings have been reported - the summary for Year 3 is therefore more detailed than the summaries for Years 1 and 2.

Parts 2, 3 and 4 provide further detail on Year 3.

Part 2 introduces the Year 3 findings and includes the evaluation questions and the methodology.

Part 3 gives the detailed findings of the Year 3 administrative data analysis.

Part 4 gives the detailed findings from the Year 3 surveys.

Footnotes