We have come to the end of privacy; our private lives, as our grandparents would have recognised them, have been winnowed away to the realm of the shameful and secret. To quote ex-tabloid hack Paul McMullan, "privacy is for paedos". Insidiously, through small concessions that only mounted up over time, we have signed away rights and privileges that other generations fought for, undermining the very cornerstones of our personalities in the process. While outposts of civilisation fight pyrrhic battles, unplugging themselves from the web – "going dark" – the rest of us have come to accept that the majority of our social, financial and even sexual interactions take place over the internet and that someone, somewhere, whether state, press or corporation, is watching.

The past few years have brought an avalanche of news about the extent to which our communications are being monitored: WikiLeaks, the phone-hacking scandal, the Snowden files. Uproar greeted revelations about Facebook's "emotional contagion" experiment (where it tweaked mathematical formulae driving the news feeds of 700,000 of its members in order to prompt different emotional responses). Cesar A Hidalgo of the Massachusetts Institute of Technology described the Facebook news feed as "like a sausage… Everyone eats it, even though nobody knows how it is made".

Sitting behind the outrage was a particularly modern form of disquiet – the knowledge that we are being manipulated, surveyed, rendered and that the intelligence behind this is artificial as well as human. Everything we do on the web, from our social media interactions to our shopping on Amazon, to our Netflix selections, is driven by complex mathematical formulae that are invisible and arcane.

Most recently, campaigners' anger has turned upon the so-called Drip (Data Retention and Investigatory Powers) bill in the UK, which will see internet and telephone companies forced to retain and store their customers' communications (and provide access to this data to police, government and up to 600 public bodies). Every week, it seems, brings a new furore over corporations – Apple, Google, Facebook – sidling into the private sphere. Often, it's unclear whether the companies act brazenly because our governments play so fast and loose with their citizens' privacy ("If you have nothing to hide, you've nothing to fear," William Hague famously intoned); or if governments see corporations feasting upon the private lives of their users and have taken this as a licence to snoop, pry, survey.

We, the public, have looked on, at first horrified, then cynical, then bored by the revelations, by the well-meaning but seemingly useless protests. But what is the personal and psychological impact of this loss of privacy? What legal protection is afforded to those wishing to defend themselves against intrusion? Is it too late to stem the tide now that scenes from science fiction have become part of the fabric of our everyday world?

Novels have long been the province of the great What If?, allowing us to see the ramifications from present events extending into the murky future. As long ago as 1921, Yevgeny Zamyatin imagined One State, the transparent society of his dystopian novel, We. For Orwell, Huxley, Bradbury, Atwood and many others, the loss of privacy was one of the establishing nightmares of the totalitarian future. Dave Eggers's 2013 novel The Circle paints a portrait of an America without privacy, where a vast, internet-based, multimedia empire surveys and controls the lives of its people, relying on strict adherence to its motto: "Secrets are lies, sharing is caring, and privacy is theft." We watch as the heroine, Mae, disintegrates under the pressure of scrutiny, finally becoming one of the faceless, obedient hordes. A contemporary (and because of this, even more chilling) account of life lived in the glare of the privacy-free internet is Nikesh Shukla's Meatspace, which charts the existence of a lonely writer whose only escape is into the shallows of the web. "The first and last thing I do every day," the book begins, "is see what strangers are saying about me."

Our age has seen an almost complete conflation of the previously separate spheres of the private and the secret. A taint of shame has crept over from the secret into the private so that anything that is kept from the public gaze is perceived as suspect. This, I think, is why defecation is so often used as an example of the private sphere. Sex and shitting were the only actions that the authorities in Zamyatin's One State permitted to take place in private, and these remain the battlegrounds of the privacy debate almost a century later. A rather prim leaked memo from a GCHQ operative monitoring Yahoo webcams notes that "a surprising number of people use webcam conversations to show intimate parts of their body to the other person".

Max Mosley: 'Your private life belongs to you. If anyone takes it from you it's theft and it's the same as theft of property.' Photograph: Tom Jenkins for the Guardian

It is to the bathroom that Max Mosley turns when we speak about his own campaign for privacy. "The need for a private life is something that is completely subjective," he tells me. "You either would mind somebody publishing a film of you doing your ablutions in the morning or you wouldn't. Personally I would and I think most people would." In 2008, Mosley's "sick Nazi orgy", as the News of the World glossed it, featured in photographs published first in the pages of the tabloid and then across the internet. Mosley's defence argued, successfully, that the romp involved nothing more than a "standard S&M prison scenario" and the former president of the FIA won £60,000 damages under Article 8 of the European Convention on Human Rights. Now he has rounded on Google and the continued presence of both photographs and allegations on websites accessed via the company's search engine. If you type "Max Mosley" into Google, the eager autocomplete presents you with "video," "case", "scandal" and "with prostitutes". Half-way down the first page of the search we find a link to a professional-looking YouTube video montage of the NotW story, with no acknowledgment that the claims were later disproved. I watch it several times. I feel a bit grubby.

"The moment the Nazi element of the case fell apart," Mosley tells me, "which it did immediately, because it was a lie, any claim for public interest also fell apart."

Here we have a clear example of the blurred lines between secrecy and privacy. Mosley believed that what he chose to do in his private life, even if it included whips and nipple-clamps, should remain just that – private. The News of the World, on the other hand, thought it had uncovered a shameful secret that, given Mosley's professional position, justified publication. There is a momentary tremor in Mosley's otherwise fluid delivery as he speaks about the sense of invasion. "Your privacy or your private life belongs to you. Some of it you may choose to make available, some of it should be made available, because it's in the public interest to make it known. The rest should be yours alone. And if anyone takes it from you, that's theft and it's the same as the theft of property."

Mosley has scored some recent successes, notably in continental Europe, where he has found a culture more suspicious of Google's sweeping powers than in Britain or, particularly, the US. Courts in France and then, interestingly, Germany, ordered Google to remove pictures of the orgy permanently, with far-reaching consequences for the company. Google is appealing against the rulings, seeing it as absurd that "providers are required to monitor even the smallest components of content they transmit or store for their users". But Mosley last week extended his action to the UK, filing a claim in the high court in London.

Mosley's willingness to continue fighting, even when he knows that it means keeping alive the image of his white, septuagenarian buttocks in the minds (if not on the computers) of the public, seems impressively principled. He has fallen victim to what is known as the Streisand Effect, where his very attempt to hide information about himself has led to its proliferation (in 2003 Barbra Streisand tried to stop people taking pictures of her Malibu home, ensuring photos were posted far and wide). Despite this, he continues to battle – both in court, in the media and by directly confronting the websites that continue to display the pictures. It is as if he is using that initial stab of shame, turning it against those who sought to humiliate him. It is noticeable that, having been accused of fetishising one dark period of German history, he uses another to attack Google. "I think, because of the Stasi," he says, "the Germans can understand that there isn't a huge difference between the state watching everything you do and Google watching everything you do. Except that, in most European countries, the state tends to be an elected body, whereas Google isn't. There's not a lot of difference between the actions of the government of East Germany and the actions of Google."

All this brings us to some fundamental questions about the role of search engines. Is Google the de facto librarian of the internet, given that it is estimated to handle 40% of all traffic? Is it something more than a librarian, since its algorithms carefully (and with increasing use of your personal data) select the sites it wants you to view? To what extent can Google be held responsible for the content it puts before us?

It isn't Mosley who has pushed European courts into giving a definitive answer to these questions, but, rather, an unknown lawyer from northwest Spain. In 2009, Mario Costeja González found that a Google search of his name brought up a 36-word document concerning a case from the late 90s in which banks threatened to seize his home. The information was factually incorrect – he'd actually paid off the debts in question. More than this, it was, he argued, irrelevant. He is now a lawyer with a successful practice and any former money worries ought not to feature on the internet record of his life. Google fought the case within Spain and then all the way to the European Court of Justice. Costeja González won, the article was taken down, the victory labelled "the right to be forgotten".

Google's response to the ruling has been swift and sweeping, with 70,000 requests to remove information processed in the weeks following the judgment. A message now appears at the bottom of every search carried out on Google, warning: "Some results may have been removed under data protection law in Europe." It seems that Google has not been judging the quality of these requests and relying on others to highlight content that has been taken down erroneously. Search results to newspaper articles on Dougie McDonald, a Scottish football referee accused of lying, were taken down. And a Robert Peston piece for the BBC about Merrill Lynch CEO Stan O'Neal's role in the 2008 financial crisis was removed; as of August 2014, it is still missing from Google searches.

To understand how much protection the law offers those wishing to defend their privacy against the triumvirate of state, press and data-harvesting corporations, I turn to one of the country's top privacy lawyers, Ruth Collard at Carter Ruck. We meet in Carter Ruck's offices near St Paul's Cathedral. I sit under the beady gaze of the firm's late founder – the target of much Private Eye bile – until Collard, more genial than I'd expected, with short, neat hair, comes to meet me. We stroll to a coffee shop and I ask her about the Costeja González case.

"I think it's a very surprising decision, I really do," she says, "but it was Google's collection of data, the arrangement and prioritisation of it that influenced the judgment, as the court found they were a 'controller' of the information."

I ask about the freedom of speech implications of the judgment – surely every politician will be applying to Google, trying to scrub away the traces of the sordid affair, the expenses scandal? She nods emphatically. "Almost immediately after the judgment, there were reports of a politician trying to clean up his past. We have been contacted by clients who have read about the judgment and are interested. So far, the ones that have contacted me, I have not thought they had a case to make. It was clear from the judgment that a balance has to be struck – between the interest of the subject in keeping information private and the interest of internet users in having access to it. The effect will be removing information that may have been once completely justified in being there, but is now outdated or irrelevant."

We go on to discuss the changing face of privacy in the internet age. She firstly notes how relatively young privacy law is in this country – only since 1998 have there been laws in place to protect privacy. Cases brought before this, such as that of 'Allo 'Allo star Gordon Kaye, who in 1990 was photographed in his hospital bed badly battered following a car crash, had to prove that a breach of confidence had taken place. Collard points me to the recent Paul Weller case, in which the Modfather sued after being pictured in the Daily Mail out walking with his 16-year-old daughter and 10-month-old twins.

"A lot of these cases are Mail cases," she says with a twinkle. "It was the fact that the photos showed the children's faces that was found to be significant by the court. Weller brought the claim on behalf of all three children and it succeeded [although the Mail is appealing]." In his judgment, Justice Dingemans argued that the children's faces represented "one of the chief attributes of their respective personalities… These were photographs showing the expressions on faces of children, on a family afternoon out with their father. Publishing photographs of the children's faces, and the range of emotions that were displayed, and identifying them by surname, was an important engagement of their Article 8 rights."

Collard tells me: "The Mail argued very hard on a number of grounds, one of which was that the children didn't have a reasonable expectation of privacy, given that the teenager a couple of years earlier had done some modelling for Teen Vogue and the babies had had photos of them tweeted by their mother. However, she had been careful when tweeting not to show their faces. This thing about facial expressions is new and it will be interesting to see where it goes."

This year has not only seen a spate of news stories surrounding the issue of privacy, but also a series of literary and artistic approaches to the subject. One of the most interesting was Privacy, a play at the Donmar Warehouse that came out of a close collaboration between director Josie Rourke and playwright James Graham. It was a groundbreaking production, from the moment the audience were urged to leave their telephones on as they entered the theatre. Over the course of the play, by turn intrusively intimate and angrily political, the audience (and their smartphones) found themselves at the centre of an attempt to chart the status of contemporary privacy – and the seemingly muted public reaction to its loss.

Within minutes of entering the theatre, I was squirming in my seat. An actor on stage was publicly analysing the results of a study designed by the psychometrics department of Cambridge University that used my Facebook profile to reveal my innermost secrets, questioning almost every aspect of my personality, from my political views to my sexuality. It was terrifying and compelling all at once. As the play ratcheted up to a coruscating finale, we, the audience, were made to see the enormous value of the rights we'd handed over as the mere cost of life in the 21st century (who, we were asked, had read the iTunes privacy policy? It's word for word as long as The Tempest).

Privacy director Josie Rourke and playwright James Graham: 'A lot of normal people, even if they did have the knowledge of how much they're giving away, would just go: that's fine, because I'm not interesting enough to be spied on.' Photograph: Linda Nylind for the Guardian

I meet Rourke and Graham in Donmar's offices on Dryden Street in Covent Garden. We discuss the play, how charged tremors ran around the audience after each subsequent revelation and, above all, the sense of outrage that propels the narrative as it turns its fire on Facebook, Apple and finally on the American government. Are they worried, I ask, that there seems to be so little anger on behalf of the public? That it is beginning to feel (to me at least) as if we have given up our right to a private life with barely a whimper? Graham nods. "I think a lot of normal people, even if they did have the knowledge of how much they're publishing and how much they're giving away, and how much a corporate or government agency might know about them, would just go, 'That's fine, because I'm not interesting enough to be spied on.'"

There were moments in the play that were profoundly uncomfortable: for the audience and, I would imagine, for Rourke and Graham. Both director and playwright featured on stage and we delved deeply into their private lives, into their secret selves (and into our own). I ask about the dynamic between the two of them during the writing of the play and where they decided to draw the line between probing and invasive.

"We thought it through," Graham answers. "From the very beginning, we decided that we never wanted any of these interactions to be exposing or judgmental. We wanted them to be kind and benevolent and ask people to examine their data streams."

Rourke nods in agreement. "In order to research the show, we kept pushing at the boundaries of each other's privacy, really to see what that felt like. I don't want to make it sound like a piece of performance art, but we kept asking ourselves – oh look, we've found out that it's possible to do this. Are we prepared to do that to each other? The distinction between secret and private has been the guiding philosophical principle. We're not looking to get your secrets. We're asking you to test this thing which is less tangible and less transactable, which is your privacy."

A few days after our meeting, Rourke puts me in touch with Michal Kosinksi, the Cambridge academic who, with David Stillwell, has designed youarewhatyoulike.com, the psychometric algorithm that produces from your Facebook "likes" a map of your soul. I think of Ruskin, who in 1864 said: "Tell me what you like and I'll tell you what you are." I also think of Orwell's thought police and Philip K Dick's The Minority Report, where criminals are identified and arrested before they commit crimes. This is one of the problems about advances in technology: we are preconditioned to view them through a dystopian lens, with Orwell, Ballard, Burgess and others staring over our shoulders, marvelling, but fearful.

While I wait for my results, I ask Kosinski whether the potential for misuse worries him. "Most technologies have their bright and dark side," he replies, buoyantly. "My personal opinion is that a machine's ability to better understand us would lead to improved consumer experience, products, etc… But imagine that we published a clone of youarewhatyoulike.com that simply predicted which of your friends was gay (or Christian or liberal or HIV-positive, etc); lynches are not unlikely to follow…"

I'm left baffled by my results. According to the algorithm, I'm 26 (I'm actually 35). The program is unsure whether I'm male or female (I'm male). I'm borderline gay or, as Kosinski puts it in his analysis: "You are not yet gay, but very close." I'm most likely to be single, extremely unlikely to be married (I'm not sure what my wife will say to all this). The algorithm correctly predicts my professional life (art, journalism, psychology) and my politics (liberal) but claims that I exhibit low neuroticism. It should sit next to me on a turbulent flight. I realise that I'm viewing the results of a kind of double of myself, the public persona I present through social media (and over which I presume some sort of control), nothing like the real me. For that, I need a psychiatrist.

Alongside Rourke, Graham, William Hague and director of Liberty, Shami Chakrabarti, one of the real-world figures represented in the play Privacy is academic and psychoanalyst Josh Cohen. Cohen's book, The Private Life (2013), is an intelligent and highly literary exploration of the changing nature of privacy in the age of Facebook and Celebrity Big Brother. Skipping from Katie Price to Freud to Booker-winning author Lydia Davis, Cohen paints a convincing picture of a culture fighting a desperate psychological battle over the private self. He argues that both our ravenous hunger for celebrity gossip and the relentless attempts by the wealthy to protect their privacy have recast the private life as "a source of shame and disgust". The tabloid exposé and the superinjunction both "tacitly accede to the reduction of private life to the dirty secrets hidden behind the door".

Psychoanalyst Josh Cohen: 'Privacy, precisely because it ensures that we are never fully known to others, provides a shelter for imaginative freedom, curiosity and self-reflection.' Photograph: readmesomethingyoulove.com

And yet what neither the press nor the lawyers recognise when they treat privacy as they would secrecy – as something that can be revealed, possessed, passed on – is that the truly private has a habit of staying that way. Cohen argues that the private self is by definition unknowable, what George Eliot calls "the unmapped country within us". In an email conversation, Cohen gives me a condensation of this thesis: "When we seek to intrude on the other's privacy, whether with a telephoto lens, a hacking device or our own two eyes, we're gripped by the fantasy of seeing their most concealed, invisible self. But the frustration and disappointment is that we only ever get a photograph of the other, an image of their visible self – a mere shadow of the true substance we really wanted to see. The most private self is like the photographic negative that's erased when exposed to the light."

There is something strangely uplifting in this idea – that no matter how deep they delve, the organs of surveillance will never know my true self, for it is hidden even from me.

I ask Cohen about the differences between our "real" selves and those we project online. I think of the younger, gayer, less neurotic incarnation of myself that appears on Facebook. "I agree that the online persona has become a kind of double," he says. "But where in Dostoevsky or Poe the protagonist experiences his double as a terrifying embodiment of his own otherness (and especially his own voraciousness and destructiveness), we barely notice the difference between ourselves and our online double. I think most users of social media and YouTube would simply see themselves as creating a partial, perhaps preferred version of themselves."

I remember something James Graham had said to me: that early on in the writing of Privacy, he'd sat at Josie Rourke's kitchen table and read his tweets out loud to her. "I had to stop it was so awful… Just seeing the person there, listening as I said what I thought about my food, about politics, about what was on the television, it felt very exposing, like I'd sold myself badly."

This is the horror of social media – that it gives us the impression we are in control of our virtual identities, putting out messages that chime with our "real" selves (or some idealised version of them). In fact, there is always slippage and leakage, the subconscious asserting its obscure power. The internet can, as Cohen tells me, "provide a way of exploring and playing the multiplicity and complexity of the self". It can also prove to us just how little control we have over how we appear. As William Boyd put it in Brazzaville Beach: "The last thing we discover in life is our effect."

There is, of course, a flipside to the dystopian view of profit-hungry corporations and totalitarian governments relentlessly reaping our private selves. Josh Cohen describes the lifelogging movement as bearing "an overriding tone of utopian enthusiasm". Lifelogging involves the minute-by-minute transmission of data about one's life, whether by photographs, web-journals or the sort of Quantified Self technologies – wearable watches, data-gathering smartphone apps – developed by German firm Datarella (and many others, Google Glass not least among them). An early proponent of lifelogging was conceptual artist Alberto Frigo, who in 2003 decided to record every object he would hold with his right hand for the next 36 years. He put the pictures on his website, 2004-2040.com.

Frigo's project started with photographs but has developed into a labyrinthine mapping of his thoughts and dreams, the music he is listening to and the world around him. The website is now a wormhole, a place in which it is possible to lose yourself in the beautiful but useless ephemera of a single existence. Frigo tells me that his aim is to create for a future audience "some kind of a Rosetta Stone of this time, where different aspects of a person's life, recorded with different media, can be compared and interpreted."

2004-2040.com has come to dominate Frigo's life. He has insisted on radical honesty, writing down detailed dreams, a sophisticated map of his mental state. For his wife, this was problematic. "She did not accept that I dreamed of other girls," he tells me. "This slowly led to our separation." It also led to a hiatus in Frigo's lifelogging, when he carried on in private, recording but not publishing his material. "My ex-wife did not wish me to publish my dreams and I did not feel like presenting my project without one part. After the divorce, it took me sometime before I went online again."

Josh Cohen told me about the psychic risks of lifelogging. For some, he said, "shadowing your transient, irretrievable life is a permanent digital life, and the really frightening spectre here is that the digital recording becomes more 'real', more authoritative than your memory." I asked Frigo about this. How would he look back on it once it had all finished? How will he feel to have this stunning, often baffling pictorial record of his life? "At the end of the project," he says, "I think I would feel that it is time to retire. I would love to go back to the Italian alps, when I am originally from, and have not so much to do with technology anymore, just have a little garden and look back, write a book perhaps, but mostly take care of a little land which I unfortunately never had the possibility to own as my father went off to Canada when I was little… I might well end up in a hospital connected to a machine for the rest of my life, who knows? Perhaps, in 2040 I might not reach my 60th birthday and the 1,000,000 photos of all the objects my right hand has used, the DNA code of my life, will remain incomplete, and no one will care about them or me."

Your privacy has a value. There are even companies such as RapLeaf.com that will tell you what your personal information is worth. The basic facts? Very little. More detailed information – for example, you own a smartphone, are trying to lose weight or planning a baby – are worth much more. Big life changes – marriage, moving home, divorce – bring with them fundamental changes in our buying patterns as we seek, through the brands with which we associate ourselves, to recast the narratives of our lives. Through analysis of buying patterns, US retailer Target predicted one of its customers was pregnant (and sent her coupons for maternity wear) before the teenager had broken the news to her disapproving parents.

Perhaps the reason people don't seem to mind that so much of their information is leaking from the private to the public sphere is not, as some would have it, that we are blind and docile, unable to see the complex web of commercial interests that surround us. Maybe it's that we understand very clearly the transaction. The internet is free and we wish to keep it that way, so corporations have worked out how to make money out of something we are willing to give them in return – our privacy. We have traded our privacy for the wealth of information the web delivers to us, the convenience of online shopping, the global village of social media.

Let me take you back to August 2006, the Chesterfield Hotel, Mayfair. My little brother (Preston from the Ordinary Boys) was marrying a girl he'd met on the telly (Chantelle Houghton). Celebrity Big Brother was still in its Channel 4 pomp, attracting 6 million viewers and upwards, the focus of water-cooler debate and gossip mag intrigue. The wedding was, briefly, an event. I remember a moment between the ceremony and the reception when we were queuing up in our gladrags to have our pictures taken for the OK! magazine spread. I felt a sudden, instinctive lurch – the thought of my phiz besmirching every hairdresser's salon and dentist's waiting room. I wasn't on Facebook, Twitter hadn't been invented, Friends Reunited? No thanks. I ran – to the bemusement of my family and the photographer.

Now, though, I post pictures of my breakfast on Instagram, unspool my soul in 140-character soundbites on Twitter, allow – even encourage – Facebook and Google and Apple to track my every move through the smartphone that has become less a piece of technology and more an extension of (and often a replacement for) my brain. I write articles on subjects I'd previously kept secret from my nearest and dearest. I let a Sunday newspaper take a (relatively tasteful) picture of me and my children when I was promoting my last novel. We have all – to a greater or lesser extent – made this same transaction and made it willingly (although my children didn't have much say in the matter).

We weren't private creatures in centuries past, either. In a 1968 talk on privacy in the electronic age, sociologist Marshall McLuhan argued that it was the coming of a new technology – books – and the "closed-off architecture" needed to read and study that had forged the sense of the private self. It may be that another new technology – the internet – is radically altering our sense of what (if anything) should remain private. We live in a liberal democracy, but, with recent lurches to the right, here and abroad, you don't need to be Philip K Dick to imagine the information you gave up so glibly being used against you by a Farage-led dictatorship.

More immediately, there is the normalising effect of surveillance. There is a barrier or check on our behaviour when we know we are being watched: deviancy needs privacy. This was the thinking behind Jeremy Bentham's Panopticon, a model for a jail where a single watching guard could survey a whole prison of inmates (the model, by the way, for Zamyatin's One State). Soon, it didn't matter whether the guard was on duty or not, the mere possibility of surveillance was enough to ensure compliance. This is where we find ourselves now, under surveillance that may seem benign enough but which nonetheless asserts a dark, controlling power over us, the watched.

The message seems to be that if you really want to keep something private, treat it as a secret, and in the age of algorithmic analysis and big data, perhaps best to follow Winston Smith's bitter lesson from Nineteen Eighty-Four: "If you want to keep a secret, you must also hide it from yourself."

Here lies our greatest risk, one insufficiently appreciated by those who so blithely accept the tentacles of corporation, press and state insinuating their way into the private sphere. As Don DeLillo says in Point Omega: "You need to know things the others don't know. It's what no one knows about you that allows you to know yourself." By denying ourselves access to our own inner worlds, we are stopping up the well of our imagination, that which raises us above the drudge and grind of mere survival, that which makes us human.

I asked Josh Cohen why we needed private lives. His answer was a rallying cry and a warning. "Privacy," he said, "precisely because it ensures we're never fully known to others or to ourselves, provides a shelter for imaginative freedom, curiosity and self-reflection. So to defend the private self is to defend the very possibility of creative and meaningful life."

Alex Preston's most recent novel is In Love and War, published by Faber at £14.99. To buy a copy for £11.99 with free UK p&p call 0330 333 6846 or go to guardianbookshop.co.uk