Teaching has always been a demanding job. Performing for five hours a day in front of a class is tiring, but add to that lesson preparation, marking, meetings and admin and most teachers clock up 55-60 hours a week – and have been doing so for decades.

But over the past 15 years, there has been one significant change. Today, teaching is no longer a private endeavour that takes place in a classroom. Now teachers are required to create a paper trail that proves learning has happened, for people who were not present in the room at the time.

This audit culture means that, in many schools, the teacher no longer is able to decide how to prepare and deliver lessons, mark pupils’ work, and assess and record learning. This is dictated by school policy.

Latest government figures show that teachers are spending, on average, an extra hour a day on work compared with a decade ago. According to Teacher Tapp, an app that polls teachers on their experiences, about 60% say they are unhappy with the balance between their professional and personal commitments. It’s making them feeling drained and exhausted.

Research from the National Foundation for Educational Research (NFER) shows the number of teachers considering leaving the profession has increased in the past year, from 17% to about 23%. And government figures show the numbers leaving before the official retirement age has also risen, from 6% a year in 2011 to just over 8% in 2016.

With more than 20,000 schools now operating independently, owing to the government’s drive to turn the majority of schools into academies, trying to understand what has gone wrong and why the role of the teacher has changed is complicated.

What is clear is that the pace of change has been exhausting. Over the past two decades, there have been hundreds of curriculum reforms and initiatives, including major reforms to the inspection process.

In the past, when inspectors paid a visit, it was to learn about a school. Now they check the headteacher knows their school, which means they must look at planning, monitoring and other evidence teachers submit to their leadership team. When schools were given several months’ notice of an Ofsted visit, they could gather this evidence ahead of an inspection. But with the notice period now as little as 24 hours, this kind of evidence can no longer be manufactured when needed. While this is a good thing, it does mean inspection-compliant paperwork must be ready at all times, which puts pressure on teachers and school leaders.

While onerous, the increase in paperwork has had one important use in helping headteachers put their weakest staff through capabilities procedures. We all remember from our school days teachers whose inadequate teaching was left unchecked. I’m pleased the audit culture has helped remove them, but fear that in doing so, we’ve damaged the morale of teachers we love and value.

There is no evidence that constant teacher paperwork improves pupils’ performance. Photograph: Rex Features

Yet despite all teachers’ extra work educational standards have not improved since the mid-1990s, according to a recent report [pdf] from the Centre for Evaluation and Monitoring (CEM).

Just as it took multiple initiatives and institutions to increase teacher workload to current levels, it will take a plethora of initiatives to undo this mess. But there are things the government can do to ease the journey, for example looking at replacing “directed time” contracts (which set out the number of the hours a teacher can be asked to work over the academic year) with ones that specify teachers’ daily hours. The government could also introduce four-year lead times for curriculum and assessment reforms, as countries such as Singapore, Finland and Wales have. Ofsted could be required to monitor workload and teacher turnover, using readily available data such as the school workforce census.

All this will help at the margins. But the main impetus for change has to come from school leaders, since they have created the audit culture. Expecting headteachers to reverse years of learned behaviours, without any guarantees of protection from rogue inspectors demanding audits, requires considerable bravery on their part.

School leaders need to be made aware that auditing teaching isn’t actually possible. The links between what they observe through these activities and the quality of learning is simply unproven. A headteacher cannot know what is going on in a classroom unless they are there. School leaders need to learn to live with this uncomfortable truth and stop asking for lesson plans, performing book scrutiny, reviewing marking and collecting tracking data. All of which means learning to trust teachers again.

They also need to be brave enough to say no to the endless “silver bullets” in education: the new initiatives that appear each month from people who claim they know how to fix educational underachievement.

All of this requires financial investment. But the government won’t fund reduced contact time for teachers. Nor will it raise teacher salaries so pay at least competes with other professions with a long-hours culture, such as law.

Once upon a time teachers had spare time to spend with their families, friends and hobbies. Parents became teachers because the job allowed them to see their family – the long term-time working hours seemed like a fair trade-off for the longer holidays. Now new parents give up teaching because they perceive the hours to be irreconcilable with raising children.

Teaching is suffering from a workload crisis that nobody seems to want or will take responsibility for. It is in everyone’s interest that we sort it out, for if we fail, then schools will become overwhelmed by chronic teacher shortages and educational standards will fall.

And while no single person, government or establishment should take the blame for the current crisis, we all have a responsibility to step up and make teaching a job worth doing again.

This article is an edited version of the Caroline Benn memorial lecture, which Dr Allen will give this evening, 7 November, at the Commons. The lecture is an annual event in memory of the comprehensive schools campaigner, who died in 2000