We all know how it ended – phone hacking, cover-up, corruption and "the humblest day of my life". To many it was just what Rupert Murdoch deserved. Dragging Britain's press into the gutter, showing contempt for the law and contaminating politics and public life had finally caught up with him. The Doctor Evil of global media finally brought to heel. But is that media caricature of Murdoch – and it's one that many people of my vintage and liberal leanings were thoroughly soaked in – right?

I've never met Murdoch – been in the same room a couple of times maybe, but never spoken to him. But having made two programmes about him – one for Radio 4 two years ago at the time of his 80th birthday and one that airs on Sunday night on BBC2 – I'm beginning to think that caricature is most definitely wrong. Actually I'd go further and say that almost everything I thought I knew about Murdoch and what he did is at the very least more complex than it appears, and in some cases just plain wrong. What's more, the Murdoch story tells us as much about Britain and us as it does about him.

I knew Murdoch came from Australia but hadn't appreciated the importance of his background or his father. Sir Keith Murdoch was the biggest name in Australian journalism after exposing the Gallipoli scandal during the first world war. The Murdochs were Australian aristocracy – wealthy and respected – with a pronounced dose of antipathy to the "old country". But the other key thing about Murdoch senior was that, for all his public profile as a national hero (and media baron) when he died, the only thing he actually ended up being able to pass on to his son was one small newspaper – the Adelaide Evening News. And that was one lesson the young Rupert learned from his father – ownership and control were far more important than public profile. And so it has been with Murdoch businesses ever since – own and control or dispose.

I knew Rupert Murdoch had been to Oxford University but I didn't know he was a dyed-in-the-wool socialist. He had a bust of Lenin on his Worcester College mantelpiece and, according to one contemporary, was even close to the Oxford Communist party.

Having attended a Daily Worker rally in London, young Murdoch and his friend were late back and locked out of college. Murdoch apparently produced the key to the Oxford party HQ, where the pair spent the night. Early next morning, once the first London train had arrived, in came a man in a Lenin-style cap with a red star on the front bearing that day's Daily Workers, addressing Murdoch thus: "Morning, comrade."

Academically, Murdoch was pretty undistinguished, but on account of his political activities, his gambling and the fact that he had a car, was a bit of a name around town. The university newspaper, Cherwell, described him as "Rupert Murdoch, cataclysmic chauffeur from the outback … he is known as a brilliant betting man with that individual 'Billingsgate' touch." Ouch! In any event it's pretty clear that many people saw Murdoch even then pretty much as he saw himself – a rebel and an outsider from the start. Definitely not one of "us".

After a brief spell at the market-leading Daily Express (his father knew Beaverbrook), under the tutelage of tabloid genius (Sir) Edward Pickering, Murdoch returned to Australia to take up his meagre inheritance. Within 15 years he had risen to become a major force in Australian media, buying failing papers and turning them round with a mixture of sensation and irreverence, and starting that country's only national newspaper – the Australian. The Murdoch approach to business was already evident – an inveterate deal-maker, gambling on opportunities others regarded as crazy and with a remarkable appetite for risk.

I knew Murdoch was routinely referred to as the "dirty digger". What I didn't know was that the title came from his very first big story in his very first British newspaper. He had bought the News of the World from the Carr family, who had owned it since 1891, in 1968. In truth they sold it to him in order to avoid having to sell it to Robert Maxwell. In the words of one insider from the time, the Carrs had run it like a family grocery store. Chairman Sir William Carr – never less than two bottles a day (of scotch, that is) – used the Savoy Grill as the company canteen. In business terms, Murdoch made mincemeat of them.

But it was his first big story – the Christine Keeler memoirs – that would get him noticed more widely. Murdoch saw it as classic tabloid fare – anti-establishment with a decidedly racy twist and a great commercial opportunity. But the liberal establishment reacted with fury; all they could see was an old story rehashed purely for commercial gain by a grasping colonial for whom no gutter was too deep. And thus the "dirty digger" was born and the essential fault line between Murdoch and liberal Britain was set. Murdoch had gone from near unknown to beneath contempt in just six months.

Murdoch and his Australian wife Anna found themselves virtually ostracised. Veteran journalist Chapman Pincher recalled Murdoch turning up to a society shooting party wearing a "brand new, brown, knickerbocker shooting suit and with a gun that hadn't been fired". Murdoch was trying to fit in. But what he talked to Pincher about in the course of the day was "English snobbery".

I knew Murdoch had bought the Sun but I hadn't appreciated how and why it had become so successful. By setting out to give readers what they wanted – as opposed to what someone else thought they should have – Murdoch rode the wave of an enormous social transformation. The sexual revolution was playing out, a new generation of younger, better educated, less deferential, more aspirational skilled workers was emerging, and, critically, the liberal consensus that had dominated postwar politics was breaking down. And, unlike its competitors, in political terms the Sun was always a floating voter. In fact it had its own manifesto published on the front page in Murdoch's first week: "The Sun has no party politics. The Sun is a radical newspaper. We are not going to bow to the establishment in any of its privileged enclaves. Ever."

When the paper backed Margaret Thatcher in 1979, it was the first time it had backed an election winner. But when the then editor, Larry Lamb, was given a knighthood in Thatcher's first honours list, Murdoch sacked him for accepting it. Ever since Murdoch had bought the paper, it had campaigned against the honours system.

Murdoch's purchase of the Times and Sunday Times in 1981 has gone down in history as one of the clearest examples of the dirty digger's malign influence in the corridors of power – payback even for helping Thatcher into power. Now I can't say there was no secret deal to avoid a reference to the monopolies and mergers commission, but if there was it was very secret indeed. Because I've looked high and low – cabinet minutes, notes of the secret meeting between Thatcher and Murdoch at Chequers and even John Biffen's unpublished memoirs (he was the minister responsible) and there is not the faintest whiff of such a deal. There didn't really need to be one. The figures showed both papers as loss-makers and as such Biffen was perfectly entitled to let the deal go through.

But it also appears that people who I had assumed – and history records – as taking the opposite position supported the Murdoch bid at the time. In a letter to Gordon Brunton, then MD of Thomson and responsible for finding a buyer for the papers, Sunday Times editor Harry Evans says that his top choice among the potential bidders was Murdoch! And 30 years on, the Times is still going and still losing money. With the moonlight flit to Wapping, Murdoch moved to take on the print unions who had supported his purchase of the Sun. And difficult and brutal as it was, there is now a consensus, more or less, that change had to happen and that, by acting in a way no other press owner dared to do, Murdoch has extended the commercial life of Britain's press by at least 20 years.

And then there is Sky TV, started in 1989 and derided as "Sun TV" by the broadcasting establishment. With the "best TV in the world" why would anyone want to pay for more? It was the biggest business risk of Murdoch's career – threatening to bankrupt the whole of News Corporation. Well, there are now 10 million reasons (aka Sky-subscribing households) to think that Murdoch was right and the traditional broadcasters were wrong. The fact is that people did want the choice Murdoch offered and the "brilliant betting man with the individual Billingsgate touch" had beaten the British establishment again.

Of course, there is an almost Shakespearean twist to the end of the story. Having built his business on aggressive, no holds barred, sensational journalism, focused on the "democracy of the marketplace" – giving the people what they want – it was the phone hackers' version of that that came to undo him. That and the fact that through the New Labour years – contrary to common belief Murdoch was never personally close to Thatcher – Murdoch's organisation became part of a new establishment every bit as self-serving and corruptible as the old-boy networks of old he had battled against.

In the post-phone-hacking world, Murdoch has lost effective control of his company – splitting News Corp in two was forced on him by impatient investors and ambitious executives who seized on his moment of weakness. He has been forced to drop his bid to regain full ownership of his beloved BSkyB, which might now end up being sold elsewhere. And his fondest wish – to be succeeded at the head of the family business by one of his children – looks like a pipe dream after heir apparent James Murdoch's fall from grace at News International.

For Britain, the Murdoch era is almost over – not a moment too soon, I hear some of you say. But is Britain really worse off for having had 40 years of Rupert Murdoch?

@steve_hewlett