Peter Preston, editor of this newspaper for almost 20 years from 1975 – then the longest service of any editor since CP Scott; a member of its controlling Scott Trust for almost 30; and overwhelmingly the most decisive and influential figure in its reshaping and development through the later years of the 20th century, has died aged 79.

The Guardian – not long since the Manchester Guardian when he joined it in 1963 – was his first and only national newspaper until it acquired the Observer, to which he also became and remained deeply committed. This was not for want of offers to go elsewhere – there were always plenty of those – but because it was always where he wanted to be. In Peter’s case this was not so much for its politics as for its commitment to good writing and good reading, and the sharp, inquisitive eye that it turned on life.

Few inside or outside the office knew him well; even fewer who worked for him were ever in doubt about his iron commitment, his versatility, his deep journalistic instincts, his matchless talent for innovation or his courage.

What you first noticed on meeting Peter was his polio-affected arms, the legacy of a disease contracted in childhood. After that, you forgot them. He would have hated to see his disability mentioned here, just as he hated people trying to help as he struggled to haul his coat on. No fuss was his rule, and no concessions. Yet the complexity of his nature; the reticence, the habits of secrecy; the unsparing way he drove himself; his hatred of whingeing complaint – none of these was wholly explicable except in terms of the illness that restricted his movement, and almost did for him, in childhood.

He was 10 when polio struck first his father, who was dead within days, and then Peter himself. For months he was kept alive in an iron lung. It was almost two years before he returned to normal life: but this was a life dependent on a whole new vocabulary of movement. Before he was into his teenage years, he had learned that life was struggle. “You had,” he said later, “to claw your way back.”

Born in Barrow upon Soar, Leicestershire, Peter was the son of John Preston, a manager in the greengrocery trade, and his wife, Kathlyn (nee Chell). He grew up in the village of Quorn, two miles south of Loughborough. He was bright enough to get to the local grammar school, but his illness had knocked two years out of his education and the school thought he ought to leave at 15 and train as a jewel cutter.

Nonetheless, he clawed his way back. He won a place at St John’s College, Oxford, reading English, and even more to the point, beginning his lifelong affair with newspapers. He joined the university newspaper, Cherwell, where he swiftly acquired all the skills of the operation and ended as its editor.

From there, in 1960 he moved to the Liverpool Daily Post as an editorial trainee. For some, that meant a painstaking process of learning the skills of reporting and subediting. But Peter was soon at the heart of the operation, writing leaders in the afternoon, then moving to the subs’ desk, and before long even to designing page one.

Three years later he moved down the road to Manchester, and the Guardian. He wanted, above all, to write, and he swiftly landed what some reporters saw then as the best writing job of the lot: coverage of political news away from Westminster. He was also, however, sent off to hotspots: Ghana; Cyprus; the war between India and Pakistan – places that continued to preoccupy him throughout his writing days. He landed a plum specialist job: education correspondent.

But that did not last for long. In 1966 he was posted at one day’s notice to begin a new column, succeeding a staid institution of the Manchester Guardian called the London Letter. If he had a brief, it was largely one he invented himself. The Miscellany column, a melange of news, gossip, entertainment and often bitchy comment, was soon the talk of the town, even if, with its insider jokes and the convoluted delivery which was Peter’s trademark, it sometimes baffled the country.

Facebook Twitter Pinterest Peter Preston, December 1973 Photograph: E Hamilton West/The Guardian

Its breathless irreverence was something new for the Guardian. “Many milestones marked the track along which the bier of the old Guardian was carried,” wrote the newspaper’s historian of the period 1956-88, Geoffrey Taylor, in his book Changing Faces. “This was one of them.”

Though dependent on a menage of specialist correspondents, dissident politicians and compulsive leakers, Peter wrote it all himself. Throughout his first year he hardly took a day off, and eschewed proper holidays. He was never that keen on leisure. Who’s Who year by year listed his hobbies as four children (later plus eight grandchildren), films and football (sometimes Charlton, sometimes Millwall), but his real addiction was newspapers. He hated bank holidays, most of all when they meant that the Guardian did not publish.

The fraught but freewheeling life of Miscellany was exactly what Peter enjoyed, but that, too, could not last. He wanted to write, but he also increasingly wanted to organise, and as one who often sounded off about the paper’s frailties, above all at what he saw as its continuing lack of professionalism, he increasingly yearned to get his hands on the levers.

In 1968 he took over and swiftly transformed the features department, introducing a page of news features across from the leader page, and moving on through the arts and fashion to revolutionise the rest of his new domain. He knew from the start exactly what he wanted to do. At the outset, he took his new team to the pub across the road and described it so exhilaratingly that we all but floated back to the office, bursting to make a start.

Peter was 30 now, and already a clear contender to succeed Alastair Hetherington as editor when that vacancy came. In the meantime, though, he accumulated all kinds of experience to fit him for an editing role. From features he moved on to the role of production editor – the journalist most closely responsible for the practical job of getting the paper out at night. The characteristic modernising, professionalising techniques that had marked his writing and his redesign of the feature pages were now applied to news pages, and above all to the nightly showpiece, page one. Pipes were smoked, and defenceless biro tops chewed into a mush in moments of stress. He was also deeply involved in preparing the paper for a move into new technology, travelling to the US to see how the future worked.

The experience gave Peter a dimension that was missing from the CV of the other potential editor-elect, John Cole. Successively labour correspondent, a reforming news editor and deputy editor, John lacked Peter’s experience of design and production.

Facebook Twitter Pinterest Peter Preston with a newly printed copy of the first Guardian front page. Photograph: Denis Thorpe/The Guardian

That Peter won reflected in part his broader experience, but the outcome turned above all on the contrasting of John’s solidity and embodiment of Guardian tradition against Peter’s innovation and flair. The main charge against which Peter was required to defend himself was his lack of any obvious political base. People knew what John’s politics were: he did not disguise his conviction that the Guardian ought, though never uncritically, to be on Labour’s side. What Peter thought was a mystery.

The journalists were divided: the younger ones wanted Peter and change, while the older tended to opt for John and continuity.

The management representatives involved in the process were keen to get Peter, whom they saw as more likely to broaden the paper’s appeal and maximise its revenue. The selectors picked Peter. In June 1975, at the age of 37, he became the paper’s ninth editor.

It was not a propitious moment. The financial pressures were excruciating. Within two years of his accession, the Guardian’s cover price was up by 50% to 15p, with drastic effects on the circulation, which fell from 306,000 in 1976 to 260,000. Even so, as the Guardian ended its uneasy tenancy in the Thomson building in Gray’s Inn Road and took over brand new premises half a mile away in Farringdon Road in 1976, there was a cheerful sense of new beginnings. Peter was creating across the paper an air of excited expectancy that recalled the first days of his revolution in features.

The Guardian that Peter inherited was a very different paper from the one you are reading today. The average number of broadsheet pages in the week he took over was 24, in a single section. The old standard Century typeface – dignified to some and over-solemn to others – was swept away in the revamp that Peter engineered with the designer David Hillman in 1988. The pictures when he moved in were mean and meagre, and reproduced in an often bleary black and white. The tone of the writing and presentation, too, was notably more cautious, restrained and “respectable”.

In 1975, the editor’s life was made miserable by constant production problems and battles with union power. Peter’s early days and especially nights in the job were full of such confrontations: as editor, he was in the office when crises blew up, and as a board member was pressured to intervene. Guardian pay rates then were about the worst in Fleet Street, which was one source of print room friction. That applied to the journalists too. When Rupert Murdoch took over the Times and sought to poach a good many Guardian stars, the editor of the Times, Harold Evans, confessed himself shocked by Guardian salaries. The traditions of the newspaper said that those who worked for the Guardian should not expect great rewards. Having done you the honour of hiring you, it expected, and usually got, your gratitude and loyalty.

The immediate problem was to claw his way out of the sharp decline brought on by price pressures over the first two years. It was Peter’s mission, along with allies of whom the most notable was Gerry Taylor, managing director in the early days of his editorship, to rescue the paper from the mildly amateur, not-quite-the-top-league status that made it part loved legend and part Fleet Street joke.

In particular, they wanted to break down its client status as an institution sustained by Manchester money (it had traditionally lived off the profits of the Manchester Evening News). That meant rescuing the paper from the climate of financial instability that had twice nearly killed it, and re-establishing it on a strong and buoyant basis.

It meant updating the paper’s appeal, broadening its horizons, bringing in readers who until now might have shied away from it, preferring something simpler and more accessible.

One key element was the strategy, devised in partnership with Taylor in his days as advertising director, of developing specialist areas of the paper – first education, then the social services, and later the media – where strong editorial content would help attract classified advertising. Education Guardian became the places that people in universities, colleges and schools turned to, both for the coverage and because that was where they would find the best job offers. The Society pages brought in precious new revenue from local government.

The formula worked. Helped by a period of almost a year in 1978-79 when, because of industrial conflicts far more serious even than ours, the Times did not print at all, the circulation soared. That brought satisfaction and awards, but it also brought complacency. The Guardian’s hold on its readers was less secure than it seemed. That was proved with the launch in 1986 of the Independent; a paper founded by three refugees from the Telegraph, which was initially expected to hurt the Telegraph and the Times, but which was soon making disturbing inroads into the Guardian’s territory. There had been too many Guardian readers – most Guardian journalists could number some of their friends among them – who had taken it less because it gave them what they wanted than because the alternatives were even less to their taste.

As ever, Peter was determined to claw his way back. The difficult business of coming to terms with technological change had long inhibited other changes he wanted to make in the paper. In 1987 he recruited Hillman, whom he had met as a fellow judge on an awards panel, and told him what he wanted. When Hillman’s designs were unveiled to heads of department the initial reaction was favourable.

Facebook Twitter Pinterest Preston in the newsroom in 1984. Photograph: Ken Saunders/The Guardian

Within weeks of the relaunch early the following year, as the anguished, often angry letters poured in from discontented customers, sentiment turned against it. Week after week, Peter had to defend it in editorial meetings. He would not concede.

It was sometimes a lonely enterprise, but the mood switched again as reader response swung around and the evidence grew that the change had worked: that the new Guardian was happily gaining new readers. The even more radical change to a G2 tabloid section four years later – a notion Peter had toyed with a decade before, only to be balked by production considerations – brought no such turmoil. This time the acclaim was immediate.

The tone of the paper shifted again with its remodelled appearance. There was new wine in Peter’s new bottles, not old. In a parody of Pass Notes published when he gave up the editorship, the Evening Standard said: “He has transformed the Grauniad from a misprint-ridden in-house asexual organ of the Do-gooding Dungaree classes into the sassiest, most fashionable and strongest daily broadsheet on the Street.” A travesty of the paper he had inherited; but those final words were worth having. You would certainly never have heard them in 1975.

Not everybody was happy. Some complained the paper had lost its authority. Some of that had to do with the diminished clout of the leader columns, where Peter changed the tone. The days of thunder from Mount Olympus were done. No sensible newspaper now expected people to jump to attention when they lectured it, as had been the case in Scott’s day. Why address the world from a pulpit instead of attempting a conversation?

Thus the tone became more informal, not least when he wrote himself, drawing on the special vocabulary that ensured you could always tell it was Peter who was addressing you. Words like “wambled” and “mired” and “pother” cropped up, though none quite as much as “intrinsic”. Sometimes the approach misfired. “Couched in an embarrassed (and embarrassing) jokey and vulgar prose which was a disgrace to a quality newspaper,” said the Times of the Guardian’s final leader in the 1987 election campaign – a verdict endorsed by the Nuffield study of that election and by quite a few of his colleagues.

But he could be deeply and convincingly serious. The paper’s outstanding campaigns during his editorship – the battle against establishment secrecy over the Spycatcher affair of the late 1980s, concerning publication of Peter Wright’s memoirs, for instance, and above all its lonely, often bitterly disparaged, dissent from the Falklands adventure, were largely conducted by him.

There were dreadful moments, too, above all the mishandling of the case of Sarah Tisdall, who in 1983 sent to the paper, anonymously, secret documents about the stationing of nuclear weapons at Greenham Common. Through the collective failure of a half a dozen senior people, of whom I was one, a crisis was created in which the Guardian had either to return the documents, thus enabling the authorities to trace the source, or to risk, not just the jailing of its editor, which Peter was ready to contemplate, but unlimited fines as long as his refusal lasted, which the editor was advised could destroy the paper.

A meeting open to all editorial staff was called on the morning when he had to decide whether or not to return the document. Though it was almost entirely supportive, he felt he must have lost the respect that had so far sustained his editorship. He delivered the document. He then submitted his resignation to the Scott Trust, which refused it.

That evening, colleagues persuaded him to go across to the pub: with difficulty, since he was not at any time a great one for social occasions, usually consulting his watch within 15 minutes or so and fidgeting to get back to his desk. But he went; and the respect and affection he found there ensured that he would survive. One saw in that sequence, though other issues had demonstrated it too, the loneliness of Peter’s editorship. “As an editor,” he once told an interviewer, “you can’t have friends in a normal sense.”

That partly reflected the nature of his office, affecting all who occupy it; the need to visit sometimes on cherished colleagues decisions they are bound to resent, or even to switch them out of jobs they would like to do for ever; the need to veto exciting but extravagant projects; the removal on prudent legal advice of stories for which their authors have worked their hearts out, are wholly convinced are true, and love as they love their own children. All this was made worse in Peter’s case by his reticence, his incurable habit – infuriating and sometimes near insupportable to all his deputy editors – of leaving you in the dark about impending events.

As he got drawn more and more into the paper’s management, finally severing the arm’s length relationship between editorial and management enshrined in the Scott tradition, his attitudes hardened. Sometimes when the leader writers met to discuss what the paper should be saying, he would take a near-Thatcherite line. “Let me play devil’s advocate,” he would say, but it wasn’t just playing devil’s advocate: it reflected a feeling that he lived every day with the need to take tough commercial decisions on tough commercial grounds, under pressures from which the rest of those assembled were cosily shielded.

Facebook Twitter Pinterest Peter Preston at the House of Lords in 1988 during the Spycatcher affair. Photograph: Ken Saunders/The Guardian

His experiences with the unions had cost them much of the paper’s traditional sympathy (which Cole would have continued). His doubts about the Labour party often boiled up into exasperation, though the fears of old Labour supporters that the paper might come out in 1980-81 with full-blooded endorsement of the fledgling SDP proved unfounded. People said there were SDP posters in the windows of his home in Camberwell, south London. So there were: but his wife, Jean, had put them there.

In February 1995 he resigned the editorship. In a piece of mendacity seasoned with malice, the Spectator columnist Paul Johnson said he had been kicked upstairs for his handling of another Guardian crisis, over the use of what Peter called a “cod fax” to establish otherwise unobtainable facts for a story about the Conservative cabinet minister Jonathan Aitken. Preston’s conduct, Johnson fantasised, had provoked near civil war on the Guardian. Peter’s then deputy, Alan Rusbridger, called this allegation “deranged”. Peter’s refusal to yield to Aitken’s pressure was another demonstration of that conspicuous courage that every great editor needs. The eventual outcome was Aitken’s conviction for perjury. There was no civil war and no kicking upstairs. Those close to Peter had known for some time he thought it was time to move on. The fact that he had recently moved ahead of Hetherington in the long-serving editors’ league may just have had something to do with the timing.

Rusbridger, who some had feared might move elsewhere if Peter stayed put, was his barely challenged successor. Peter was appointed editor-in-chief of the Guardian and the Observer, whose purchase he had been crucial in engineering. On his final night, having said he wanted no formal departure, he entertained his staff to a party in a wine bar across the road. He had never seemed so serene. The editors he wanted – Rusbridger at the Guardian, Andrew Jaspan at the Observer – were safely installed. He planned, he told friends, gradually to fade from the scene. He was hugely looking forward to resuming the calling the job had removed from him: that of human being.

Yet the period that followed was difficult, as tensions rose over the Observer’s future, and old colleagues feared that in the interests of shoring up their new acquisition, the management, Peter conspicuously among them, was contemplating drastic measures that could badly damage the Guardian. The loneliness of the editor, it now seemed, was nothing to the loneliness of the editor-in-chief.

The manoeuvres that resolved the issue made for an unhappy end to Peter’s time of pre-eminence, costing him both the new role of editor-in-chief and the chairmanship of the Guardian board. This whole process, conducted in his absence abroad, left Peter aggrieved; it left his wife, Jean, considerably angry. What Peter achieved at the Guardian could scarcely have been possible without the calm, tenacious, unswerving support of the former Jean Burrell, whom he married in 1962. For years, she had to accept a degree of commitment that must have felt draining on their life together. Her judicious use of kindly ultimatums even persuaded him in the end to take proper holidays, and not to ring the office while away (in the early years, two or three times daily) to ask if all was OK.

A slippered retirement was the last thing that Peter aspired to, however. There were columns for the Guardian and the Observer, assessments and analyses on big occasions, book reviews for both papers, occasional features especially on the press, and film reviews for the Observer when Philip French was away. His long-running weekly column in the Observer, Peter Preston on press and broadcasting, the last of which appeared on New Year’s Eve, cast a sharp and often disapproving eye on the radically changing role of the media, not sparing that of the paper he had once edited. He persistently lamented an excessive readiness to accept that text papers must be doomed as screens took over.

More recently he took to tweeting, pursuing other favourite themes – the madness of Brexit (he had been all his life an enthusiast for Britain as part of Europe); a BBC commitment to balance that drove it to give as much space to wrongheaded causes (like Brexit) as good ones; the irresponsibility of Catalan separatists. After part of his family settled in Spain he had a base in Catalonia, where he worked on two novels, 51st State (1998) and Bess (1999). Then, too, there was work for the Guardian Foundation and chairmanships of other organisations working for journalists, especially where they were threatened – notably as British executive chairman and for a time world chairman of the International Press Institute. In 2012 he was a founder of the European Press prize.

Yet though he still pumped up the pressure, there was also by now a new, relaxed, more human Peter than most people outside his family had previously been permitted to see. One day to my astonishment he spoke of a “lovely long lunch” on the previous day. This was a man who, at the height of his editorship, had seemed to regard long lunches as a wretched, if not immoral, distraction from the job that had to be done.

Newspapers evolve, sometimes in ways that old adherents find unsettling. The difference between Peter’s final edition and this morning’s paper, or its online equivalent at any time of the day, reflects not only the passage of more than 20 years but a whole new confluence of preoccupations and pressures – especially, financial constraints even more harsh than they were in the hardest time of Peter’s leadership.

Yet that does not deny his legacy. No editor could have been braver in times of struggle. No editor in the paper’s history had displayed greater commitment and diligence. For all the changes that have inevitably ensued, he left an indelible mark on the newspaper he served for so long.

He is survived by Jean and their twin daughters, Alex and Kate, two sons, Ben and Rupert, three granddaughters and five grandsons.

• Peter John Preston, journalist, born 23 May 1938; died 6 January 2018.