If you are reading this article on a printed copy of the Guardian, what you have in your hand will, just 15 years from now, look as archaic as a Western Union telegram does today. In less than 50 years, according to Clay Shirky, it won't exist at all. The reason, he says, is very simple, and very obvious: if you are 25 or younger, you're probably already reading this on your computer screen. "And to put it in one bleak sentence, no medium has ever survived the indifference of 25-year-olds."

You have probably never even heard of Shirky, and until this interview I hadn't either. When I ask him to define what he does, he laughs, and admits that often when he's leaving a party someone will say to him, "What exactly is it you do?" His standard reply – "I work on the theory and practice of social media"– is not just wilfully opaque, but crushingly dreary, which is funny, because he is one of the most illuminating people I've ever met.

The people who know about Shirky call him an "internet guru". He winces when I say so – "Oh, I hate that!" – and it's easy to see why, for he is the very opposite of the techie stereotype. Now 46, his first career was in the theatre in New York, and he didn't even own a computer until the age of 28, when he had to be introduced to the internet by his mother. Arrestingly self-assured and charismatic, his conversation is warm and discursive, intently engaged yet relaxed – but it's his rhetorical fluency which bowls you over. The architecture of his argument is a Malcolm Gladwell-esque structure of psychological and sociological insight, analysing contemporary technology with the clarity of a historian's perspective and such authority that were he to tell you the sun actually sets in the east, you might almost believe him. At the very least, you'd probably want to – and if a guru is defined by the credulous deference he commands from others, then Shirky unquestionably qualifies. Within minutes I found myself hanging on his every word – despite being temperamentally hostile to almost everything he believes.

Shirky has been writing about the internet since 1996. As the chief technological officer for several web design companies during the 90s, he was quickly hired as a consultant by major media companies – News Corporation, Time Warner, Hearst – all curious about this new thing called the world wide web. In 2000, following "an intuition that the internet was turning social", Shirky turned to the fledgling phenomenon of online social networking – an obscure concept back then, but which has since evolved into MySpace, Facebook and Twitter, to become the web's primary purpose for billions of people all over the world. Shirky now teaches new media at New York University, and in 2008 published his first book, Here Comes Everybody: How Change Happens When People Come Together, which celebrated individuals' new power to communicate, organise and change the world via the web.

His predictions for the fate of print media organisations have proved unnervingly accurate; 2009 would be a bloodbath for newspapers, he warned – and so it came to pass. Dozens of American newspapers closed last year, while several others, such as the Christian Science Monitor, moved their entire operation online. The business model of the traditional print newspaper, according to Shirky, is doomed; the monopoly on news it has enjoyed ever since the invention of the printing press has become an industrial dodo. Rupert Murdoch has just begun charging for online access to the Times – and Shirky is confident the experiment will fail.

"Everyone's waiting to see what will happen with the paywall – it's the big question. But I think it will underperform. On a purely financial calculation, I don't think the numbers add up." But then, interestingly, he goes on, "Here's what worries me about the paywall. When we talk about newspapers, we talk about them being critical for informing the public; we never say they're critical for informing their customers. We assume that the value of the news ramifies outwards from the readership to society as a whole. OK, I buy that. But what Murdoch is signing up to do is to prevent that value from escaping. He wants to only inform his customers, he doesn't want his stories to be shared and circulated widely. In fact, his ability to charge for the paywall is going to come down to his ability to lock the public out of the conversation convened by the Times."

This criticism echoes the sentiment of Shirky's new book, Cognitive Surplus; Creativity and Generosity in a Connected Age. The book argues that the popularity of online social media trumps all our old assumptions about the superiority of professional content, and the primacy of financial motivation. It proves, Shirky argues, that people are more creative and generous than we had ever imagined, and would rather use their free time participating in amateur online activities such as Wikipedia – for no financial reward – because they satisfy the primal human urge for creativity and connectedness. Just as the invention of the printing press transformed society, the internet's capacity for "an unlimited amount of zero-cost reproduction of any digital item by anyone who owns a computer" has removed the barrier to universal participation, and revealed that human beings would rather be creating and sharing than passively consuming what a privileged elite think they should watch. Instead of lamenting the silliness of a lot of social online media, we should be thrilled by the spontaneous collective campaigns and social activism also emerging. The potential civic value of all this hitherto untapped energy is nothing less, Shirky concludes, than revolutionary.

Unfortunately, I am precisely the sort of cynic Shirky's new book scorns – a techno-luddite bewildered by the exhibitionism of online social networking (why does anyone feel the need to tweet that they've just had a bath, and might get a kebab later?), troubled by its juvenile vacuity (who joins a Facebook group dedicated to liking toast?), and baffled by the amount of time devoted to posting photos of cats that look amusingly like Hitler. I do, however, recognise that what I like to think of as my opinions are really emotional prejudices. But equally, Shirky's prediction for Murdoch's paywall sounds suspiciously like an emotional objection, rather than a financial calculation. How, then, can he be certain his entire analysis of the internet isn't just as subjective as my kneejerk cynicism?

"I'd say first of all that the notion that any expression of the world can be a value-neutral description of what life is really like is a fantasy, right?" he agrees readily. "We're all postmodern enough to recognise that any writer on any subject is operating within those constraints. And I have the amiably simple-minded view of this stuff you would expect from an American, which is that I think freedom is good, full stop. So therefore I think I'm probably constitutionally incapable of seeing a massive spread in those freedoms as being anything other than salutary for society.

"But ultimately, over the long haul I'm vetted on accuracy, not on enthusiasm. So if I'm wrong about paywall, I've got no place to hide. I will have been flamingly, publicly wrong for 15 years. There will be no way I can weasel out of it." He laughs, looking sublimely untroubled by this possibility.

"The final thing I'd say about optimism is this. If we took the loopiest, most moonbeam-addled Californian utopian internet bullshit, and held it up against the most cynical, realpolitik-inflected scepticism, the Californian bullshit would still be a better predictor of the future. Which is to say that, if in 1994 you'd wanted to understand what our lives would be like right now, you'd still be better off reading a single copy of Wired magazine published in that year than all of the sceptical literature published ever since."

The one point of agreement between internet utopians and sceptics has been their techno-deterministic assumption that the web has fundamentally changed human behaviour. Both sides, Shirky says, are wrong. "Techies were making the syllogism, if you put new technology into an existing situation, and new behaviour happens, then that technology caused the behaviour. But I'm saying if the new technology creates a new behaviour, it's because it was allowing motivations that were previously locked out. These tools we now have allow for new behaviours – but they don't cause them." Had Facebook been around when he was in his 20s, he cheerfully admits, he too would have spent his youth emailing photos of himself to everyone he knew.

But even if he's right, and the internet has merely unveiled ancient truths about human behaviour, isn't it still legitimate to feel a little bit dismayed by Facebook's revelation of almost infinite narcissism? Shirky lets out a polite but weary sigh. "Would the world really be better off if we were to hide from ourselves the fact that teenagers waste a lot of time trying to either flirt with each other or to crack each other up? Like, to whom was this a mystery, prior to the launch of Facebook?" He grins in good-natured amazement.

"Look, we got erotic novels, first crack out of the box, once we had printing presses. It took a century and a half for the Royal Society to start publishing the first scientific journal in English. So even with the sacred printing press, the first things you get serve the basest human urges. But the presence of the erotic novels did not prevent us from pressing the printing presses into the service of the scientific revolution. And so I think every bit of time spent fretting about the fact that people have base desires which they will use this medium to satisfy is a waste of time – because that's been true of every medium ever launched."

Shirky concedes that the web's ability to connect people with a common enthusiasm, however obscure or deviant, can create a dangerously distorted impression of what is healthy or normal. "But so the question in all of this stuff, always, always, always, is: is the net trade-off better or worse for society? I've never been a cyber utopian. I've always understood that this is a set of trade-offs. So for all the normalisation of, say, paedophilia, we also get young small-town kids growing up gay who now know they're not abnormal. And it seems to me that the net trade-off of lessening society's ability to project a sense of normal that no one actually lives up to is a good thing.

"I don't mean to say it will therefore be an endless fountain of raindrop-flavoured kittens from now till St Swithin's day. But rather, in the same way that we've generally decided that the printing press was a good thing – and I would contrast that with television, which in my mind is an open question – rather than just saying in the panglossian way that all new technologies are an improvement, it is an on-the-balance calculation."

The neuroscientist Susan Greenfield produced a report last year which suggested that the popularity of online social media was damaging children's brain development, in particular their capacity for empathy. Shirky has two children, aged nine and six, and says they live in "a very restricted media household", with only supervised access to a communal computer. "I would not hesitate to say I was addicted to the internet in the first two years. It can be addictive and things not taken in moderation have negative effects. But the alarmism around 'Facebook is changing our brains' strikes me as a kind of historical trick. Because we now know from brain science that everything changes our brains. Riding a bicycle changes our brains. Watching TV changes our brains. If there's a screen you need to worry about in your household, it's not the one with a mouse attached."

Shirky does not own a television. Americans watch, collectively, two hundred billion hours of television a year, and if online social media diverts even just a fraction of that time, he argues, that has to be a good thing. "As I say in the book, even the stupidest possible creative act is still a creative act. And I'd still take the most inane collaborative website over someone watching yet another half hour of TV."

By now, despite myself, I'm having to reconsider my old snootiness towards social media. There's just one last thing, I say. Had I never been online before, and had just read his book, I'd probably be so inspired by his account of the creative and collaborative instincts of the online community, I'd be rushing to log on. But if I started out on, say, the Guardian's Comment is free site, the sheer nastiness of many of the commenters would floor me like a train. If the web has unlocked all this human potential for generosity and sharing, how come the people using it are so horrible to each other?

Shirky smiles, confident that he has the answer even to this. "So, there's two things to this paradox. One is that those conversations were always happening. People were saying those nasty things to one another in the pub or whatever. You just couldn't hear them before. So it's a change in our awareness of truth, not a change in the truth.

"Then there's this second effect, that anonymity makes people behave more meanly. What I think is going to happen there is we are slowly going to set up islands of civil discourse. There's no way to make the internet not anonymous – and if there was, the most enthusiastic consumers of that technology would be Iranian and Chinese and Burmese governments. But there are ways of saying, while you're here, use your real identity. We need to set up the social norms which say in this space you need to use your real names, or some well-known handle.

"Whenever you say that, people cry censorship, but frankly? Fuck off." He breaks off, laughing. "You know, getting that right is important. The whole, 'Is the internet a good thing or a bad thing'? We're done with that. It's just a thing. How to maximise its civic value, its public good – that's the really big challenge."

• Cognitive Surplus: Creativity and Generosity in a Connected Age by Clay Shirky is published by Allen Lane, price £20

• This article was amended on 5 July 2010. The original referred to Western Union telegrams looking arcane today. This has been corrected.