The worldwide web has made critics of us all. But with commenters able to hide behind a cloak of anonymity, the blog and chatroom have become forums for hatred and bile

For a while after his first TV series was broadcast in 2009, comedian Stewart Lee was in the habit of collecting and filing some of the comments that people made about him on web pages and social media sites. He did a 10-minute Google trawl most days for about six months and the resultant collected observations soon ran to dozens of pages. If you read those comments now as a cumulative narrative, you begin to fear for Stewart Lee. A good third of the posts fantasised about violence being done to the comic, most of the rest could barely contain the extent of their loathing.

This is a small, representative selection:

"I hate Stewart Lee with a passion. He's like Ian Huntley to me." Wharto15, Twitter

"I saw him at a gig once, and even offstage he was exuding an aura of creepy molesty smugness." Yukio Mishima, dontstartmeoff.com

"One man I would love to beat with a shit-covered cricket bat." Joycey, readytogo.net

"He's got one of those faces I just want to burn." Coxy, dontstartmeoff.com

"I hope stewart lee dies." Idrie, Youtube

"WHAT THE HELL! If i ever find you, lee, i promise i will, I WILL, kick the crap out of you." Carcrazychica, YouTube

"Stewart Lee is a cynical man, who has been able to build an entire carrer [sic] out of his own smugness. I hope the fucking chrones disease [sic] kills him." Maninabananasuit, Guardian.co.uk

"I spent the entire time thinking of how much I want to punch Stewart Lee in the face instead of laughing. He does have an incredibly punchable face, doesn't he? (I could just close my eyes, but fantasizing about punching Stewart Lee is still more fun than sitting in complete, stony silence.)" Pudabaya, beexcellenttoeachother.com

Lee, a standup comedian who does not shy away from the more grotesque aspects of human behaviour, or always resist dishing out some bile of his own, does not think of himself as naive. But the sheer volume of the vitriol, its apparent absence of irony, set him back. For a few months, knowing the worst that people thought of him became a kind of weird compulsion, though he distanced himself from it slightly with the belief that he was doing his obsessive collating "in character". "Collecting all these up isn't something I would do," he suggests to me. "It is something the made-up comedian Stewart Lee would do, but I have to do it for him, because he is me…"

Distanced or not, Lee couldn't help but be somewhat unsettled by the rage he seemed to provoke by telling stories and jokes: "When I first realised the extent of this stuff I was shocked," he says. "Then it appeared to me that a lot of the things I was hated for were things I was actually trying to do; a lot of what people considered failings were to me successes. I sort of wrote a lot of series two of Stewart Lee's Comedy Vehicle with these comments in mind, trying to do more of the things people hadn't liked."

The "40,000 words of hate" have now become "anthropologically amusing" to him, he insists. "You can see a lot of them seem to be the same people posting the same stuff under different names in different places, and it is strange to see people you have known personally, whom you thought you had got on fine with at the time, abusing you under barely effective pseudonyms."

He's stopped looking these days, and never really tried to identify or confront any of his detractors. "I am slightly worried that some of them might be a bit insane and hope I haven't made myself or my family a target."

Lee is, of course, not alone in having this anonymous violent hatred directed toward him. On parts of the internet it has become pretty much common parlance. Do a quick trawl on the blog sites and comment sections about most celebrities and entertainers – not to mention politicians – and you will quickly discover comparable virtual rage and fantasised violence. Comedians seem to come in for more than most, as if taboo-breaking was taken as read, or the mood of the harshest baying club audience had become a kind of universal rhetoric. It's not quite heckling this, though, is it? A heckle requires a bit of courage and risk; the audience can see who is doing the shouting. Lee's detractors were all anonymous. How should we understand it then: harmless banter? Robust criticism? Vicious bullying?

The psychologists call it "deindividuation". It's what happens when social norms are withdrawn because identities are concealed. The classic deindividuation experiment concerned American children at Halloween. Trick-or-treaters were invited to take sweets left in the hall of a house on a table on which there was also a sum of money. When children arrived singly, and not wearing masks, only 8% of them stole any of the money. When they were in larger groups, with their identities concealed by fancy dress, that number rose to 80%. The combination of a faceless crowd and personal anonymity provoked individuals into breaking rules that under "normal" circumstances they would not have considered.

Deindividuation is what happens when we get behind the wheel of a car and feel moved to scream abuse at the woman in front who is slow in turning right. It is what motivates a responsible father in a football crowd to yell crude sexual hatred at the opposition or the referee. And it's why under the cover of an alias or an avatar on a website or a blog – surrounded by virtual strangers – conventionally restrained individuals might be moved to suggest a comedian should suffer all manner of violent torture because they don't like his jokes, or his face. Digital media allow almost unlimited opportunity for wilful deindividuation. They almost require it. The implications of those liberties, of the ubiquity of anonymity and the language of the crowd, are only beginning to be felt.

You can trace those implications right back to the genesis of social media, to pioneering Californian utopias, and their fall. The earliest network-groups had a sort of Edenic cast. One representative group was CommuniTree, which was set up as an open-access forum on a series of modem-linked computers in the 1970s when computers were just humming into life. For a while the group of like-minded enthusiasts ran on perfectly harmonious lines, respecting others, having positive and informed discussions about matters of shared relevance. At some point, however, some high school teenagers armed with modems accessed the open-access space and used it to trash and abuse the CommuniTree, taking free speech to uninhibited extremes that the pioneers had never wanted. The pioneers were suitably horrified. And eventually, after deciding that they could neither control the students through censorship, nor tolerate the space with them in it, they shut CommuniTree down.

This story has become almost folkloric among new media prophets, a sort of founding myth. It was one of the first moments when the possibilities of the new collective potential was tainted by anonymous lowest-common-denominator humanity, a pattern that has subsequently been repeated in pretty much all virtual communication. Barbarians, or "trolls" as they became known, had entered the community, ignoring the rules, shouting loudly, encouraging violence, spoiling it for everybody. Thereafter, anyone who has established a website or forum with high, or medium-high ideals, has had to decide how to deal with such anonymous destructive posters, those who got in the way of constructive debate.

Tom Postmes, a professor of social and organisational psychology at the universities of Exeter and Groningen in his native Netherlands, and author of Individuality and the Group, has been researching these issues for 20 years. "In the early years," he says, "this online behaviour was called flaming. And then that became institutionalised. Among friends, the people who engaged in this activity were actually quite jocular in intent but they were accountable to standards and norms that are radically different to those of most of their audience. Trolls aspire to violence, to the level of trouble they can cause in an environment. They want it to kick off. They want to promote antipathetic emotions of disgust and outrage, which morbidly gives them a sense of pleasure."

Postmes compares online aliases to the tags of graffiti artists: "Trolls want people to identify their style, to recognise them, or at least their online identity. But they will only be successful in this if an authority doesn't clamp down on them. So anonymity helps that. It's essentially risk-free."

There is no particular type of person drawn to this kind of covert bullying, he suggests: "Like football hooligans, they have family and live at home but when they go to a match the enjoyment comes from finding a context in which you can let go, or to use the familiar phrase 'take a moral vacation'. Doing this online has a similar characteristic. You would expect it is just normal people, the bloke you know at the corner shop or a woman from the office. They are the people typically doing this…"

Some trolls have become nearly as famous as the blogs to which they attach themselves, in a curious, parasitical kind of relationship. Jeffrey Wells, author of Hollywood Elsewhere, is a former columnist on the LA Times who has been blogging inside stories about movies for 15 years. For the last couple of years his gossip and commentary has been dogged by the invective of a character called LexG, whose 200-odd self-loathing and wildly negative posts recently moved Wells to address him directly: "The coarseness, the self-pity and the occasional eye-pokes and cruel dismissiveness have to be turned down. Way down. Arguments and genuine disdain for certain debaters can be entertaining, mind. I'm not trying to be Ms Manners. But there finally has to be an emphasis on perception and love and passion and the glories of good writing. There has to be an emphasis on letting in the light rather than damning the darkness of the trolls and vomiting on the floor and kicking this or that Hollywood Elsewhere contributor in the balls…"

When I spoke to Wells about LexG, he was philosophical. "Everybody on the site writes anonymously, except me," he says. "If they didn't I think it would cause them to dry up. This place is like a bubble in which you can explode, let the inner lava out. And, boy, is there a lot of lava."

He has resisted insisting that people write under their own name because that would kill the comments instantly. "Why would you take that one in 100 chance that your mother or a future employer will read what you were thinking late one night a dozen years ago if you didn't have to?" For haters, Wells believes, anonymity makes for livelier writing. "It's a trick, really – the less you feel you will be identified, the more uninhibited you can be. At his best LexG really knows how to write well and hold a thought and keep it going. He is relatively sane but certainly not a happy guy. He's been doing this a couple of years now and he really has become a presence; he does it on all the Hollywood sites."

Have they ever met?

"Just once," Wells says. "I asked him to write a column of his own, give him a corner of the site, bring him out in the open." LexG didn't want to do it, he seemed horrified at the prospect. "He just wanted to comment on my stuff," Wells suggests. "He is a counter-puncher, I guess. The rules on my site remain simple, though. No ugly rancid personal comments directed against me. And no Tea Party bullshit."

The big problem he finds running the blog is that his anonymous commenters get a kind of pack mentality. And the comments quickly become a one-note invective. As a writer Wells feels he needs a range of emotion: "I also do personal confession or I can be really enthusiastic about something. But the comments tend to be one colour, and that becomes drab. It's tougher, I guess, to be enthusiastic, to really set out honestly why something means something to you. It takes maybe twice as long. I can run with disdain and nastiness for a while but you don't want to always be the guy banging a shoe on the table. Like LexG. I mean it's not healthy, for a start…"

Wells does his own marshalling of the debate, somewhat like the bartender of a western saloon. Other sites – including our own Comment is Free – employ moderators to try to keep trolls in line, and move the debate on. A young journalist called Sarah Bee was for three years the moderator on seminal techie news and chat forum the Register. She started as a sub-editor but increasingly devoted her time to looking after the "very boisterous" chat on the site. She has no doubt that "anonymity makes people bolder and more arsey, of course it does. And it was quite a politically libertarian crowd, so you get people expressing things extremely stridently, people would disagree and there would often be a lot of real nastiness." She was very liberal as far as moderating went, she thinks, with no real hard and fast rules, except, perhaps, for "a ban on prison-rape jokes, which came up extremely often".

Every once in a while, however, the mood would get "very ugly" and she would try to calm things down and remonstrate with people. "I would occasionally email them – they had to give their email addresses when registering for the site – to say, 'Even though you are not writing under your real name, people can hear you.'" In those instances, strangely, she suggests, most people were incredibly contrite when contacted. It was like they had forgotten who they were. "They would send messages back saying, 'Oh, I'm so sorry', not even using the excuse of having a bad day or anything like that. It is so much to do with anonymity…"

Bee became known as the Moderatrix – "all moderators have an implicit sub-dom relationship with their site" – though she was just about the only person in the comment section who used her own name. "There was a lot of misogyny and casual sexism, some pretty off-colour stuff. I would get a few horrible emails calling me a cunt or whatever," she says, "but that didn't bother me as much as the day-to-day stuff, really."

The day-to-day stuff was, though, "like being in another world. It got really wearying. I would go home sometimes and just sigh and wonder about it all."

She is keen to say that the Register itself she thought a great thing, and loved the idea of working there, but being Moderatrix eventually got her down. "A hive mind sets in," she suggests. "Just occasionally good sense would prevail but then there is that fact that arguments on the internet are literally never over. You moderate a few hundred comments a day, and then you come in the next morning and there are a few hundred more waiting for you. It's Sisyphean."

In the end she needed a change. She's in another "community management" job now, dealing through Facebook, which is a relief because "it removes anonymity so people are a lot more polite". When she retired Moderatrix she did a goodbye and got 250 comments wishing her well. She doesn't miss it, though. "Just occasionally I would let a stream of the most offensive things through, just to let people know how those things looked in the world… People would realise for a bit. But then the old behaviours would immediately set in. The thing any moderator will tell you is that every day is a new day and everything repeats itself every day. It is not about progress or continuity…"

There are many places, of course, on the internet where a utopian ideal of "here comes everybody" prevails, where the anonymous hive mind is fantastically curious and productive. A while ago I talked to Jimmy Wales, the founder of Wikipedia, about some of this, and asked him who his perfect contributor was. "The ideal Wikipedian, in my mind, is someone who is really smart and really kind," he said, without irony. "Those are the people who are drawn into the centre of the group. When people get power in these communities, it is not through shouting loudest, it is through diplomacy and conflict resolution."

Within this "wikitopia" there were, too, though, plenty of Lord of the Flies moments. The benevolent Wiki community is plagued by "Wikitrolls" – vandals who set out to insert slander and nonsense into pages. A policing system has grown up to root out troll elements; there are well over 1,000 official volunteer "admins", working round the clock; they are supported in this work by the eyes and ears of the moral majority of "virtuous" Wikipedians.

"When we think about difficult users there are two kinds," Wales said, with the same kind of weariness as Moderatrix. "The easy kind is someone who comes in, calls everyone Nazis, starts wrecking articles. That is easy to deal with: you block them, and everyone moves on. The hard ones are people who are doing good work in some respects but are also really difficult characters and they annoy other people, so we end up with these long intractable situations where a community can't come to a decision. But I think that is probably true of any human community."

Wales, who has conducted perhaps the most hopeful experiment in human collective knowledge of all time, appears to have no doubt that the libertarian goals of the internet would benefit from some similar voluntary restraining authority. It was the case of the blogger Kathy Sierra that caused Wales and others to propose in 2007 an unofficial code of conduct on blog sites, part of which would outlaw anonymity. Kathy Sierra is a programming instructor based in California; after an online spat on a tech-site she was apparently randomly targeted by an anonymous mob that posted images of her as a sexually mutilated corpse on various websites and issued death threats. She wrote on her own blog: "I'm at home, with the doors locked, terrified. I am afraid to leave my yard, I will never feel the same. I will never be the same."

Among Wales's suggestions in response to this and other comparable horror stories of virtual bullying was that bloggers consider banning anonymous comments altogether, and that they be able to delete comments deemed abusive without facing accusations of censorship. Wales's proposals were quickly shot down by the libertarians, and the traffic-hungry, as unworkable and against the prevailing spirit of free-speech.

Other pioneering idealists of virtual reality have lately come to question some of those norms, though. Jaron Lanier is credited with being the inventor of virtual worlds. His was the first company to sell virtual reality gloves and goggles. He was a key adviser in the creation of avatar universe Second Life. His recent book, You Are Not a Gadget, is, in this sense, something of a mea culpa, an argument for the sanctity of the breathing human individual against the increasingly anonymous virtual crowd. "Trolling is not a string of isolated incidents," Lanier argued, "but the status quo in the online world." He suggested "drive-by anonymity", in which posters create a pseudonym in order to promote a particularly violent point of view, threatened to undermine human communication in general. "To have substantial exchange, you need to be fully present. That is why facing one's accuser is a fundamental right of the accused."

We rightly hear a great deal about the potential of social media and websites to spread individual freedom, as evidenced during the Arab spring and elsewhere. Less is written about their capacity to reinforce pack identities and mob rule, though clearly that is also part of that potential.

Social psychologist Tom Postmes has been disturbed by the coarsening of debate around issues such as racial integration in his native Netherlands, a polarisation that he suggests has grown directly from the fashionable political incorrectness of particular websites where anonymity is guaranteed. "There is some evidence to suggest that the mainstream conservative media even cuts politically correct or moderate posts from websites in favour of the extremes," he says. "The tone of the public debate around immigration has diminished enormously in these forums."

One effect of "deindividuation" is a polarisation within groups in which like-minded people typically end up in more extreme positions because they gain credibility by exaggerating loosely held prejudices. You can see that in the bloggers trying to outdo one another with pejoratives about Stewart Lee. This has the effect of shifting norms: extremism becomes acceptable. As Lanier argues: "I worry about the next generation of young people around the world growing up with internet-based technology that emphasises crowd aggregation… will they be more likely to succumb to pack dynamics when they come of age?" The utopian tendency is to believe that social media pluralises and diversifies opinion; most of the evidence suggests that it is just as likely, when combined with anonymity, to reinforce groupthink and extremism.

A lot of this comes down to the politics of anonymity, a subject likely to greatly exercise the minds of legislators as our media becomes increasingly digitised, and we rely more and more on mostly unaccountable and easily manipulated sources – from TripAdvisor to Twitter feeds to blog gossip – for our information.

One simple antidote to this seems to rest in the very old-fashioned idea of standing by your good name. Adopt a pseudonym and you are not putting much of yourself on the line. Put your name to something and your words are freighted with responsibility. Arthur Schoepenhauer wrote well on the subject 160 years ago: "Anonymity is the refuge for all literary and journalistic rascality," he suggested. "It is a practice which must be completely stopped. Every article, even in a newspaper, should be accompanied by the name of its author; and the editor should be made strictly responsible for the accuracy of the signature. The freedom of the press should be thus far restricted; so that when a man publicly proclaims through the far-sounding trumpet of the newspaper, he should be answerable for it, at any rate with his honour, if he has any; and if he has none, let his name neutralise the effect of his words. And since even the most insignificant person is known in his own circle, the result of such a measure would be to put an end to two-thirds of the newspaper lies, and to restrain the audacity of many a poisonous tongue."

The internet amplifies Schopenhauer's trumpet many times over. Though there are repressive regimes when anonymity is a prerequisite of freedom, and occasions in democracies when anonymity must be preserved, it is clear when those reservations might apply. Generally, though, who should be afraid to stand up and put their name to their words? And why should anyone listen if they don't?