After Jihad Dib became principal of Punchbowl Boys High in Sydney’s south-west, he was widely celebrated for helping to transform a once-notoriously unruly public school into a source of pride for students and the local community.

But several years ago, Dib told me about the effect on his students of seeing a Naplan league table.

The tables, created by media organisations using publicly available test data, rank schools across the state and country.

Educational inequality widening Australia’s rich-poor gap, report finds Read more

“‘Sir, are we really that bad?’” Dib recalled his students, who overwhelmingly came from low socioeconomic, non-English speaking backgrounds, asking him.

“‘You told us we were a good school, and this is saying we’re one of the worst schools’,” Dib recalled. “It was just this collapse.”

School league tables or rankings don’t actually show which schools are the “worst” or “best”.

They don’t necessarily show you how good a particular school is at teaching students and improving their educational outcomes.

Essentially, what they show you is the rate at which a group of students, on average, reached a particular benchmark. But that nuance is naturally often lost – not only on students, but on the wider community.

The annual rankings of year 12 results, which are published by major newspapers at this time every year, powerfully shape how students and parents think about schools.

The top ranking school is often feted as the state’s “best”.

Even small shifts in a school’s rank over time – rising 10 places one year, sliding back another – is often interpreted as a sign of whether the school itself is getting better or worse.

This is a real shame, because that’s not what they tell us at all.

I understand the appeal of the exercise - I remember poring over them as a high school kid to compare my school to others in the state

With HSC school rankings in New South Wales, the tables are created by adding up the number of HSC course scores of more than 90, divided by the number of courses attempted by students, to give you a single school-wide figure – often dubbed the “success rate”. That rate is then ranked against the equivalent rate for every other school.

I understand the appeal of the exercise - I remember poring over them as a high school kid to compare my school to others in the state, like we were competing in a fantasy football league. And I was formerly a newspaper education editor who wrote about league tables and witnessed the voracious interest from readers.

There’s much pleasure to be taken too if your school does well. Ranking highly is a huge source of pride for some schools - especially public selective ones, where students who study furiously often relish trouncing their counterparts at expensive private schools.

But there’s no denying it’s a hugely simplistic exercise - and one from which the wrong conclusions are often drawn.

These rankings don’t take into account the different types of students different schools might admit or the types of subjects attempted by the students.

They don’t tell us about student wellbeing and happiness, about achievement in areas beyond the classroom, and crucially, about whether a school actually helped a young person improve.

Hypothetically, a student who may have only recently arrived in Australia with no English, who is nurtured by skilled and caring teachers and goes on to get 89 in every subject won’t make a dent in their school’s rank. Results lower than a band six – or 90 – don’t count.

Jenny Allum, principal of Sydney’s SCEGGS, wrote in the Sydney Morning Herald in 2015 that she was “sickened” by the “drive to analyse school results from meaningless league tables” after her school was praised for jumping several spots. She pointed out the obvious limitations in the success rate formula – how it could reflect students doing really well, but it could also signify students taking easier subjects, or fewer subjects, in an attempt to maximise their chances of getting above 90.

It is obvious to many too that the schools that dominate these rankings also draw on student populations that aren’t at all representative of the wider community.

They don’t take all comers. They get to choose.

Consistently, in NSW, these rankings are topped by the most competitive public selective schools, as well as some of the most expensive private schools (which are of course, selective in their own way – charging $30,000 a year is a pretty selective barrier to entry).

Demographic data available on the MySchool website shows the schools in this year’s top five in NSW had no enrolments across the entire school from the lowest socio-educational quartile of the population (the official measure of advantage devised by the Australian Curriculum, Assessment and Reporting Authority, which takes into account parents’ occupation and levels of education). At each of the schools, at least 70% – and as many as 97% – of students came from the highest quartile.

James Ruse Agricultural High School, a public selective school, has topped the NSW rankings every year since 1995, a fact that is often remarked on with awe.

But it’s not only a selective school, but frequently the most selective school in the state. The minimum entry test score to get into the school in the first place is usually higher than for any other selective school.

“James Ruse isn’t the ‘best’ school – it’s a school that takes all the ‘best’ students,” is how Adrian Piccoli, the former NSW education minister, Nationals MP, and now the director of UNSW’s Gonski Institute for Education, puts it to me.

As he and many others see it, these rankings don’t “really tell us anything about how effective the schools are.”

He believes looking for effectiveness, not outcomes, is more useful way of thinking about good schools.

“I want to send my kids to a school where they’ll take them – wherever they are, and improve them,” he says.

There are concerns too that school rankings are one small factor feeding into a broader trend in Australian education: the drift away from comprehensive public high schools.

The concern is that some parents, who tend to come from more advantaged backgrounds, respond to school rankings by shopping around for a higher ranking school – often a private or a selective one – abandoning their local comprehensive.

“The problem is what it’s increasingly doing is concentrating high SES [socio-economic] kids in these schools – the concentration of advantage in some schools is growing and that also means the concentration of disadvantage in other schools is growing,” Piccoli said. “[Those schools] become difficult schools to manage, they’re less attractive for staff.”

Or as Dib, who is now NSW Labor’s education spokesman, puts it, they lead to “picking the eyes out from schools that can’t afford to lose them”.

'Warped and elitist': are Australia’s selective schools failing the fairness test? Read more

The problem with how we value schools of course isn’t the fault of these rankings, and our obsession with them doesn’t come from a bad place. Parents want to know if their kids are receiving the best education possible.

Australia’s education system is riddled with inequality, and attempts to make resourcing more fair are frequently stymied.

But the focus on rankings, and the emphasis on education as a competitive exercise between schools, contradicts more important messages about striving for excellence – wherever a student might start from – and the broader role of education in becoming a healthy, well-rounded person.

Dib said after his students became crestfallen at their poor showing in the Naplan rankings he got up at assembly and gave an impassioned speech. “I said all that’s important is the effort you make.”

It can’t always be an easy line to sell when contrary messages are everywhere.

• Josephine Tovey is associate news editor at Guardian Australia