The large lecture hall of the French Institute in Barcelona is full to overflowing. People line the walls, sit in the aisles and stand three-deep at the back. There are a few middle-aged, smartly dressed people in attendance as well as a handful of old leftists with long hair and caps, but the majority of the audience are young and stylishly dishevelled, the kind of people one would expect to see at a Hot Chip or Vampire Weekend gig.

They have gathered here to listen to a 61-year-old Slovenian philosopher called Slavoj Žižek, whose critique of global capitalism now stretches to more than 50 books translated into more than 20 languages. Žižek describes himself as "a complicated communist" and, as if to complicate things further, he deploys the psychoanalytical theories of the late French thinker Jacques Lacan to illustrate the ways in which capitalist ideology works on the collective imagination. "I don't give clear answers to even the simplest, most direct questions," Žižek says. "I like to complicate issues. I hate simple narratives. I suspect them. This is my automatic reaction."

Žižek's book titles reflect his playful and often self-contradictory theoretical thrust. They include: The Ticklish Subject, which deals with "the spectre of the Cartesian subject in western thought"; The Plague of Fantasies, which analyses the ways in which "audiovisual media clouds the ability to reason and understand the world"; and the wonderfully titled Did Somebody Say Totalitarianism?, a fierce critique of "the liberal-democratic consensus".

He seems drawn to taking unfashionable stances that make him unpopular with traditionalists of whatever political hue. A recent book, In Defence of Lost Causes, argued that, in philosophical-political terms, Heidegger's fascist sympathies and Foucault's support of the Iranian revolution were "right steps in the wrong direction". Rebecca Mead, writing in the New Yorker, dubbed him "the Marx Brother" and described his approach thus: "His favoured form of argument is paradox, and his favoured mode of delivery is a kind of vaudevillian overstatement, buttressed by the appearance of utter conviction." That just about nails it – except that it overlooks the seriousness of Žižek's thinking and the way he has managed to bring dialectics into the mainstream.

"Slavoj is unique in that he operates between two different and, for the most part, exclusive, places," says the film-maker Sophie Fiennes, who directed him in The Pervert's Guide to the Cinema, a documentary that is as provocative as its title suggests, but in a strictly intellectual way. "He has been incredibly successful in taking theory out of the ivory tower of academia and into the world. He challenges the current fear of words like 'ideology' and, correctly in my view, sees this fear as a product of our information culture. It is also, he argues, a fear of what real, deep political thinking might generate in terms of unrest and discontent."

Žižek, though, is also a political provocateur and an absurdist prankster. For one of his books, he wrote a (rejected) fictional autobiographical blurb: "In his free time, Žižek likes to surf the internet for child pornography and teach his small son how to pull the legs off spiders."

As an avowed atheist, he sees no contradiction in arguing, as he did in The Fragile Absolute: Or Why is the Christian Legacy Worth Fighting For?, for a world in which Christians and Marxists unite against "the contemporary onslaught of vapid spirituality". This kind of thing does not sit well with traditional analytical philosophers. Neither does his tendency to roam freely through high and low culture, illuminating the Lacanian undercurrents in Hitchcock as well as Hegel, Leibniz and David Lynch. (In his new book, Living in the End Times, there is a serious, and seriously funny, essay on Kung Fu Panda, the recent DreamWorks animation, which Žižek insists is "a somewhat naive, but nonetheless basically accurate, illustration of an important aspect of Lacanian theory.")

Despite, or perhaps because of, his iconoclasm, his tendency to contradict himself, and his general political incorrectness – which may, one suspects, be more mischievous than heartfelt – Žižek is to today what Jacques Derrida was to the 80s: the thinker of choice for Europe's young intellectual vanguard. This fills him with dismay. Unlike Derrida, though, he is determinedly left wing, if not in the traditional sense.

"I am what you might call abstractly anti-capitalist," he says. "For instance, I am suspicious of the old leftists who focus all their hatred on the United States. What about Chinese neo-colonialism? Why are the left silent about that? When I say this, it annoys them, of course. Good! My instinct as a philosopher is that we are effectively approaching a multicentric world, which means we need to ask new, and for the traditional left, unpleasant questions."

Unlike the dapper Derrida, Žižek is a sight for sore eyes: pale to the point of sallow, bearded, overweight and effortlessly eccentric. In the 2005 documentary, Žižek!, he gives director Astra Taylor a tour of his kitchen, opening drawers and cupboards containing not cutlery and china, but his socks, underpants, trousers and shirts. His day-to-day style – if that is not too extravagant a word – consists of several dull variations on the proletarian outfit of ill-fitting T-shirt, baggy jeans, free airline socks – "Lufthansa are the best" – and lumpen footwear surely sold exclusively by a Slovenian shoeshop that has somehow missed the collapse of the Soviet bloc. (A Slovenian friend claims she recently saw him striding though Ljubljana in a T-shirt bearing the slogan "I Am Beautiful"; it's difficult to imagine any other philosopher doing that.)

When he speaks, or writes, Žižek comes alive and his thoughts flow out in what seem like uncontrollably tangential torrents. His message, at least what one can decipher of it from his scattergun approach, is both politically pessimistic and philosophically elusive.

"If you ask me if I am an optimist, I would have to say no. I am not one of those old-fashioned communists who says, with that old tragi-comic Marxist satisfaction, at least history is on our side. No. If anything, the train of history is hurtling towards a precipice. The task of the leftist thinker today is, to quote Walter Benjamin, not to ride the train of history, but to pull the brake."

In the jam-packed auditorium of the French Institute in Barcelona, Žižek speaks for more than two-and-a-half hours without once pulling the brake. His central thesis, also explored in his new book, Living in the End Times, is that "the global capitalist system is approaching an apocalyptic zero point." Žižek, though, regards the idea of a central thesis in much the same way that the great jazz saxophonist John Coltrane regarded a melody – as something to riff off, extemporise on, and return to only when all associated sub-themes have been exhausted. This approach has its problems, not least the sense that a single Žižek riff could perhaps more profitably be extended into an entire lecture that might be both deeper and more illuminating. Tonight, for instance, he barely addresses the reason why he resolutely believes in communism despite its shredded reputation.

"I don't see any continuity with old-style communism in my approach. So why do I then call it communism?" he says when I ask him about it later. "As to its contents, though, the problem is always the same. It's the enclosure of the commons. Marx was talking about land and property when he wrote about this, but today intellectual property is our commons, information is our commons. Something that Marx could not have predicted is taking place today: we are witnessing a strange regression to the same kind of enclosure of the commons, and people having to pay rent to people like Bill Gates for intellectual property."

He seems a slave to the speed of his thoughts, his motor-mouth delivery barely keeping pace with the frenetic motion of his overcrowded mind. Silence, even a pause for breath, seems to make him intensely uncomfortable. So, too, does the company of strangers. "I avoid other people if I can. The ultimate nightmare for me is a party in my honour in the United States. Having to mix and talk, to strangers, maybe 20 or 30 people who want to have a debate or, even worse, polite conversation. My God, I hate this above all, but it is the nature of my tragic life."

To witness Žižek in full flight is a wonderful and at times alarming experience, part philosophical tightrope-walk, part performance-art marathon, part intellectual roller-coaster ride. Most startling of all are the nervous tics that accompany his every utterance: the constant wiping of his beard and lips, the incessant dabbing of his furrowed brow, the closed eyes, clenched fists and the strange gutteral noises that punctuate his speech. Then, there's his lisp and his odd mispronunciations – in Barcelona, he kept using the term "a dollar cent", which I assumed was an example of fiscal insider jargon until I realised he actually meant "adolescent".

In my notebook, I map out the contours of his lecture in a series of headings. He begins with the fall of the Berlin Wall and the inevitable, in his view, rebirth of some kind of post-digital global communism, before touching on the writings of his beloved Hegel via the thoughts of Pascal. Suddenly though, in the first of many conceptual swerves, he is comparing the fall of communism to the end of the silent movie era which leads him into a riff on ideology as represented by "the disembodied voice" in Chaplin's City Lights and Hitchcock's Psycho. From there, we learn how the scene in Fight Club where Brad Pitt's character punches himself in the face is a metaphor for revolution – "Before you beat the bosses, you must first beat yourself."

By this point, the faithful are enthralled, the curious baffled and the traditionalists utterly bemused. Žižek, though, is just warming up. On and on he roams, through the French and Haitian revolutions, the Iraq war, Rumsfeld's famous speech about "known unknowns". (What about the "unknown knowns?", asks Žižek. "This is exactly how capitalist ideology works; you follow an illusion without even knowing it.") He cites the myth of Santa Claus as a supreme example of ideological indoctrination, dismisses Hollywood's love of the Dalai Lama and "all this vague, insipid Buddhist bullshit". He tells us how cynicism has become western culture's current default mode, what Christianity can teach communism, and why God is essentially a narcissist. He touches on biogenics by way of the inevitable Richard Dawkins – "This kind of extreme atheism misses the point of religion entirely" – and illustrates how science has lost its monopoly on truth. Eventually he realises there is a limit to the collective power of the audience's concentration, and he ends, as he began, with the communist revolution, informing us that the next one will succeed only if it embraces the essentially Christian, conservative social etiquette of politeness and deference. About 155 minutes after he started, he suddenly stops, drenched in sweat and bathed in applause. On cue, an old Trotskyist stands up and takes him to task for betraying the cause….

"I hate these civilised debates followed by the questions from the audience," he tells me the next morning. "So I keep going to subvert this boring ritual, but always there will be one old unreconstructed leftist who will stand up and accuse me of being a Stalinist. This," he says, sighing, "is how it goes."

The son of Slovenian communists, Žižek was born on 21 March 1949 in what was then Yugoslavia. His father was a state economist, his mother an accountant for a state-run business. I ask him if, growing up by the sea in Portorož, he had a happy childhood. "No. You could say, in a vulgar Freudian way, that I am the unhappy child who escapes into books. Even as a child, I was most happy being alone. This has not changed."

As a teenager, living in the capital Ljubljana, he read voraciously and, he says, "did pretty well at high school though I completely ignored the curriculum". At 15, he wanted to be a "movie director" but soon realised that his love of theory surpassed even his passion for film. At university in the 1960s, he was seduced by the new wave of French post-structuralist theorists – Derrida, Michel Foucault, Julia Kristeva and, above all, Jacques Lacan. His postgraduate thesis was initially rejected for being too critical of Marx, and even though he amended it, he was deemed unfit to teach philosophy. "It is very ironic how professors who attacked me for not being a Marxist have now turned nationalist and attack me for being a Marxist. But, really, I don't care."

In the 1970s, Žižek made a living by translating works of philosophy and, at one point, took himself off to France for four years. He also did four years' national service in the Yugoslavian army. He has no bitterness about that. "My formative experience was Yugoslav self-management socialism," he says, "but Slovenia had communist rule without an official philosophy so it was superficially better than anywhere else."

In 1978, he finally landed a job at what he calls" a marginal research institute". It was, he says, "a kind of banishment but also a wonderful post. Just pure research." He made contacts with philosophy institutions in France and the US, which stood him in good stead when he finally published his breakthrough book, 1989's The Sublime Object of Ideology. "Without the communist oppression," he says, quite seriously, "I am absolutely sure I would now be a local stupid professor of philosophy in Ljubljana."

In 1990, he baffled his leftist friends and supporters by standing for election as a Liberal Democratic party candidate. He came fifth. "Politics is my tragedy," he tells me dolefully. "It shadows me."

When not travelling or teaching in America or Europe – he has held posts at Columbia, Princeton and is international director of humanities at Birkbeck College, London – Žižek lives alone in Ljubljana in a small apartment full of books, DVDs, classical music CDs – "I am a committed Wagnerian and, this will shock you, I even like Elgar." Depending on whom you believe, he has been married and divorced two or three times. He is not saying. On April Fool's day, 2005, he famously wed a 27-year-old former lingerie model and Lacanian scholar from Argentina. He has two sons, one in his early 30s, the other nine years old. When I leave him, he heads off to find an iPad as a present for his youngest child. "I am a hypocritical communist, no?"

In the flesh, Žižek is, if anything, more demonic and unhealthy-looking than his photographs, his matted hair and greying beard surrounding a face that looks like it's never seen sunlight. He suffers from diabetes, a condition not helped by his nomadic lifestyle and manic disposition. "I have exploited you," he says by way of greeting, "in order to have a few hours free from the duties these Spanish leftists expect me to perform."

He seems both eager and uncomfortable and ushers me quickly upstairs to the apartment that is his temporary home. As a cleaner flits about, I ask him if he is surprised at his popularity, particularly among the young.

"My God, I am the last person to know the answer to these questions," he says, looking genuinely dismayed. "But, really, I am now thinking there is so much pressure on me to perform. I am getting really bored with it. I am a thinker, but people all the time want this kind of shitty political interventions: the books, the talks, the discussions and so forth." He sighs and closes his eyes and seems to deflate before my eyes. "I will tell you my problem openly and for this my publisher will hate me. All the talk and the writing about politics, this is not where my heart is. No. I have been sidetracked. I really mean this."

He opens a copy of Living in the End Times, and finds the contents page. "I will tell you the truth now," he says, pointing to the first chapter, then the second. "Bullshit. Some more bullshit. Blah, blah, blah." He flicks furiously through the pages. "Chapter 3, where I try to read Marx anew, is maybe OK. I like this part where I analyse Kafka's last story and here where I use the community of outcasts in the TV series Heroes as a model for the communist collective. But, this section, the Architectural Parallax, this is pure bluff. Also the part where I analyse Avatar, the movie, that is also pure bluff. When I wrote it, I had not even seen the film, but I am a good Hegelian. If you have a good theory, forget about the reality."

Why, then, given that he does not like most of his books and does not have any enthusiasm for the lecture circuit, does he not call a stop to the Žižek show? "I am doing that right now!" he shouts. "I am writing a mega-book about Hegel with regard to Plato, Kant and maybe Heidegger. Already, this Hegel book is 700 pages. It is a true work of love. This is my true life's work. Even Lacan is just a tool for me to read Hegel. For me, always it is Hegel, Hegel, Hegel," he says, sighing again. "But people just want the shitty politics."

Reviewing In Defence of Lost Causes, the British Marxist critic Terry Eagleton concluded that it was "a frenetic, eclectic parody of intellectual scholarship, by one so assured in his grasp of the finer points of Kafka or John le Carré that he can afford to ham it up a little." Only time will tell if Žižek is serious about becoming utterly serious, but if he devotes the rest of his brilliant, brainy, slightly bonkers, utterly singular life to Hegel, and Hegel alone, it will be a great gain for pure philosophy and a great loss to radical, risk-taking political theory.

"He is very much a thinker for our turbulent, high speed, information-led lives," says Sophie Fiennes, "precisely because he insists on the freedom to stop and think hard about who you are as an individual in this fragmented society. We need a radical hip priest and Slavoj is that in many ways." The very thought, I suspect, would have him quaking in his proletarian boots – and free airline socks.

Living in the End Times is published on 5 July by Verso, £20. To order a copy for a special price go to observer.co.uk/bookshop or call 0330 333 6847. Žižek appears at the London literature festival, Southbank Centre, London SE1 on 5 July, 7.30pm.