The great mystery of British politics is striding into the room, 15 minutes ahead of schedule. In the flesh, David Cameron looks thinner and younger and smaller than on television. The caricaturists are wrong: his cheeks don't appear full and ruddy at all. He looks sleek, and wired, with an intense gaze. He knows he could be a few months from Downing Street and the history books – so he is here to woo a crucial electoral bloc that is wary of falling into his arms by giving an interview to Attitude, Britain's best-selling gay magazine. He calls for coffee and dispenses with the photographer briskly: he poses for two minutes before saying, "Right, that's enough," and walking out of the shot. He places himself on his sofa, in the shadow of Big Ben, and says: "Right. Let's start."

Until 2005, David Cameron was a conventional anti-gay Tory. He attacked Tony Blair for "moving heaven and earth to allow the promotion of homosexuality in our schools". He mocked Labour for supporting the "fringe agenda" of equality for gay people. He supported the homophobic law Section 28 until its dying breath. But since he became Conservative leader, he has dramatically changed his position. He apologised for Section 28, got a Tory conference to applaud the principle of gay marriage, and has moved a flotilla of gay candidates into winnable seats. It seems at first glance like an amazing starburst of progress – making it possible at last for gay people to pick political parties from anywhere on the spectrum. The party of Norman Tebbit is now led by a man who poses for photographers outside a screening of Brokeback Mountain.

But a fat question mark hangs over Cameron's Yellow Brick Road to Damascus. It is the same question mark that pervades so many of Cameron's policies – and British politics itself. The Conservative leader has had conversion after conversion, on everything from the environment to Sure Start to bank regulation. Is it for real? How can a man's political views really change so far and so fast? Is his party behind him? Of his Shadow Cabinet team, 85 per cent of those eligible voted for Section 28, and 90 per cent voted against equalising the age of consent. By testing how honest he is about gay equality, can we tease out how authentic his claims to a softer, gentler Conservatism are?

I Shedding dead skin

He immediately starts with an apology. "I know there are gay people who have conservative values – like wanting us to be supportive of business and enterprise, wanting to have strong defence, believing in the strong defence of liberty and these kind of things – but in the past have felt held back because the Conservative Party was sending them a signal that we didn't support them or their lifestyle," he says in one long gulp of prose. "That has changed. I think we can look gay people in the eye and say, 'You can now back us... because we now support gay equality'."

Cameron starts to list a range of ways he says the Tories have shed their homophobia like dead skin. "I would particularly point to that speech [at the party conference] where as a Conservative leader I stood up and said I support commitment and marriage – whether it is between a man and a woman, or a man and a man, or a woman and a woman. Find me another Conservative leader, not just in Britain but somewhere in the Western world, who has done that – and been applauded for doing it.

"I didn't have to stand up in front of my own party and say that," he continues. "Politics is about taking some risks. That was a proper good old- fashioned, heart-in-the-throat moment. This is my chance. If you lead the party, it's your chance to put your own stamp on things and do things your own way. And sorting out this issue has been a complete pleasure in terms of that, and badly needed doing. Am I the first person to spot it? No. But I think we've done some big steps on that."

He stresses that any benefits his government gives to marriage will also go to civil-partnered couples, and there are now two people in his front-bench team who have had civil partnerships themselves. He is speaking fast and rhythmically, holding my gaze, like a debater sealing his case.

How did he get from backing homophobic laws to this public homophilia in just four years? "I think, now, looking back, you can see the mistake of Section 28," he says, talking about the Thatcher-era law that made it a crime to "promote homosexuality" to children, which he supported so strongly he put it in his election literature several times. "There's only one thing worse than making a mistake and that's not putting your hands up and admitting it."

But what exactly is he apologising for? He insists he never believed that it was possible to "promote homosexuality" or make children gay. So what did he think the law was about? "You know, we can go over history, but what it came out of was this concern that local authorities were getting too involved in messaging in schools." Yes – about gay sex. "But look, you can have your arguments about what local authorities should and shouldn't be getting involved in," he says, waving his hand. He says his mind was changed by a gay friend who told him: "You can argue forever about this but in the end it's something that a lot of people in this country find very offensive, and on that basis it can't be a sensible thing to do."

The more I ask about Section 28, the more he repeats this point – it was offensive, it was "finger-pointing", so it had to go. Yes, but it wasn't purely a symbol. It was a law that did real harm to gay people. It prevented teachers from stopping homophobic bullying; it prevented proper sex education for gay kids at the height of the Aids crisis. He repeats it again: it was an insult. He isn't going to venture deeper than that.

He says he didn't know any openly gay people as a child, or even at university. The first openly gay people he met were at the Conservative Research Department, after he had graduated. Perhaps this explains how he formed the attitudes that kept him opposed to gay equality for so long. I start to go over his record beyond Section 28 – and slap into a brick wall. In 2002 he voted against allowing gay couples to adopt. Yet when I ask him why, he flatly denies it. He says: "No... we were three-line-whipped on that vote and I abstained on it." I point him to Hansard, which records his vote against gay marriage in cold, black ink. He says "my memory" is that he abstained, and that he now thinks "the ideal adoption is finding a mum and a dad, but there will be occasions when gay couples make very good adoptive parents. So I support gay adoption."

Even since his apparent conversion, he has voted to block a piece of progress. In 2008, he wanted lesbians who receive IVF treatment to be required to name a father figure – a requirement that gay equality groups say would obviously makes it harder for them to receive treatment. "No, I think that's a classic way to try to misinterpret what the vote was about," he says. He insists he only wanted fertility clinics to have to "ask the question" about "the need for a father". But why ask the question, if you don't have an answer in mind? "I think those are important questions to be considered," says, and looks away.

II A whistle-stop tour

On an hour-long tour of the policies he will make as Prime Minister that specifically pertain to gay people, Cameron is by turns impressive, mediocre, and worrying. He is at his best and at his clearest – to my surprise – when it comes to refugees who are fleeing homophobic persecution. He says: "If you are fleeing persecution and that fear is well-founded, then you should be able to stay. As I understand it, the 1951 Convention [on the rights of refugees] doesn't mention sexuality, but because it mentions membership of a social group, that phrase is being used by the courts, rightly, to say that if someone has a realistic fear of persecution they should be allowed to stay."

At the moment, gay refugees are often told – under a Labour government – to go back home and hide their sexuality from police forces who would imprison, torture or kill them for it. I ask him if that is wrong – and he says unequivocally: "I think it is. If you have a legitimate fear of persecution, that it seems to me that is a perfectly legitimate reason to stay."

Similarly, he is admirably disdainful of the ban on gay men giving blood. He says there is an independent investigation into this and he has to wait for its results, but "it sounds perfectly logical and sensible to make the change... Logic would dictate that it's time to change." He even tells the Church of England to follow his lead, saying: "I don't want to get into a huge row with the Archbishop here... but the Church has to do some of the things that the Conservative Party has been through – sorting this issue out and recognising that full equality is a bottom-line, full essential."

Yet on perhaps the two biggest issues affecting gay people in Britain – violence in the playground, and violence on the streets – he doesn't have much to say. Ofsted has found that homophobic bullying is "endemic" in our schools, and a Stonewall study found that 42 per cent of gay kids get beaten up and 17 per cent get told they are going to be killed. Cameron says: "I think there's a broader question of bullying and how we deal with it. A part of it is about trusting teachers and head teachers more to instil a sense of discipline in their schools, which they find very difficult at the moment – partly because of all the bureaucratic rules and regulations about what they are and aren't allowed to do."

But how will he specifically tackle homophobic bullying? "The most important instrument of the state is to allow head teachers to keep order in their schools. To search for things, without having to have evidence that there's weapons involved. To set proper punishments in schools, to exclude pupils who are bullies, or take part in bullying, without being overruled by an appeals panel." He nods, as if agreeing with himself, and continues: "I think you need a framework of what is taught from above, but the discipline and order and actually making sure that bullying is stamped out has got to be done by the head teacher and teachers."

But I point out this is, again, talk about general bullying, rather than the hugely disproportionate amount directed at gay kids. Does he agree with the Liberal Democrat leader, Nick Clegg, that schools should be required to teach that homosexuality is "normal and harmless", just as they respond to racist bullying by saying all ethnicities are equal? He pauses and looks a little sceptical. "I think the point is, there's now proper guidance from the [Department of Education] about this, and I think that's right," he says.

There is, however, some evidence Cameron's policies will unwittingly make homophobic bullying worse. The keystone of his education policy is to allow any group of parents who want to set up a school, and can attract pupils, to receive state funding. But the National Secular Society warns that wherever this has been tried, there is a huge rise in religious fundamentalist schools. We know they are far worse for gay kids: the Stonewall study, for example, found that anti-gay bullying is 10 per cent worse in faith schools.

At first Cameron's response to this is to sound bemused. He says he doesn't understand why homophobia would be worse in faith schools. But I ask: Is it so odd? Some of these religious groups – not all – believe homosexuality is a sin. For the only time in the interview, Cameron looks irritated. "That's so wrong," he snaps, his brow furrowed. "My daughter goes to a church school and it's not like that." He angrily says, "A lot of what you've read in the newspapers is actually a lot of tosh." With a firm glare, he says he will put in place "ground rules" to make sure new religious schools "teach equality", and that's that. He gets up to turn down the radiator next to him.

When it comes to how to tackle the sudden spike in homophobic violence – 40 per cent in a year – Cameron's answer seems strangely scrawny. He says: "Culture is important. Some of the things that rappers and others sing are completely unacceptable. I was sort of laughed at when I first made this point four years ago, but I do believe that it's important." He says he won't ban the songs, but he will argue against them. "Don't underestimate the power of the bully pulpit, it is important. The idea of social and cultural leadership in these things does make a difference."

I assume that's the first step in his answer, and he is going to list many more ways to reduce homophobic violence – but then I realise he is staring at me, expecting the next question. That's it? What else will you do? "Well, I think we can stop some of these people [meaning rappers] coming into the country." When I tell him a Home Office study has found homophobia is "endemic" within the police, he looks surprised. He says the police force "is making some progress", and "what is required now is leadership". In the middle of hugely disproportionate violence against gay people, he's offering a weak cocktail: more prime-ministerial criticism of rappers, more power for headmasters, and a vague call for "leadership" in a police force where homophobia is rife.

III "He is not homophobic"

Yet Cameron has most shocked gay people who want to support him when it comes to Europe, where he has allied with men who accuse gays of paedophilia and destroying Western civilisation. After he became leader, he pulled out of the European People's Party (EPP), an alliance in the European Parliament with the moderate centre-right parties of Angela Merkel and Nicholas Sarkozy, in favour of a new coalition of Euro- sceptics, largely from eastern Europe.

His new grouping is led by Michal Kaminski, a Polish politician who has been filmed calling gay people "faggots". When the interviewer expressed surprise that he had used such an offensive term, Kaminski replied: "What can I say? They are faggots." Tory MEPs now sit under his leadership in Brussels; he was invited to address the Conservative Party conference. Cameron said on Sky News: "He is not homophobic."

When I raise the subject, he nods, sits up, and drinks from his coffee in a big gulp. "I think you should form European alliances on whether you agree with these people's views on the broad direction of the future of Europe, that's what it's about," he says. "Now, does that make it a more difficult message to explain to gay people who want to vote Conservative? Yes it does, I accept that. One of the reasons for doing this interview is hopefully to try and get across a sense that I have not joined with these people because of their views on social issues. I have not."

He stresses that he has joined with these groups because "there should be a centre-right group in Europe that wants [the European Union to be] an open, flexible, trading Europe, rather than the endless progress towards a more federalised Europe".

This is obviously true, and perfectly defensible. But Cameron has gone further than that. He has repeatedly said that Kaminski and his party are "not homophobic", and that he wouldn't ally with them if they were. The evidence shows this is wrong – and shockingly so.

A few days before we met, the MPs of this "not homophobic" Law and Justice Party demanded a crackdown on what they called "positive paedophilia by some homosexual circles." Their senior MP, Stanislaw Pieta, said: "I'm not saying every gay is a paedophile, but in Britain 43 per cent of paedophiles are gay and they only make up 1 per cent of the population." Their leader, Lech Kaczynski, says "the human race would disappear if homosexuality was freely promoted." There are hundreds of such statements from the party, all on video.

"Obviously, I don't agree with that [statement]," Cameron says when I read it to him. So does he now admit they are homophobes? "I'm not allied with parties that have views on homophobia or racism that I think are unacceptable." But these are the leaders of the party. They are not marginal. I read him more and more shocking statements. Poker-faced, Cameron refuses to address the contradiction in his position: he says he wouldn't ally with anti-gay politicians, yet here they are, making blatantly anti-gay statements.

Whenever I raise it, he tries to change the subject. All the parties in Poland are equally bad on gay rights, he says. I tell him that's not what the Polish gay equality groups say. The veteran gay activist Waldemar Zboralski says: "The Law and Justice Party is by far the most homophobic party in Poland, and Mr Kaminski is the leading symbol of homophobia in this country. It's very strange for Mr Cameron to deny this; it is indisputable."

So he throws into the air a confetti of different distractions. These aren't "minor parties", he says, "they were parties of government" recently. The Liberal Democrats have anti-gay allies too: "Where are the questions for Nick Clegg?" Finally, he says: "Funnily enough, who's now in the EPP? Italian fascists. Would you be happier if we went and joined a bunch of Italian fascists? No."

But Mr Cameron, why can't you simply condemn people who call us "faggots" and "paedophiles" as homophobic ? If that isn't homophobia, what is? How can we believe you are not the old Section 28 Tories underneath if you invest so much energy defending these bigots? His brow is furrowed. He says finally, in a quick, snappy tone: "The fact is, in some eastern European countries they need to make progress towards equality and rights... Conservative parties have had to go through a real change over this issue. I think we've done it faster in the UK than some others. Will other European conservative parties be on a similar journey? Yes. Have they finished? No." Finally, after a huge amount of wrangling and jangling, he argues these parties "are changing", and will change more if he engages with them. But change from what? He won't say.

IV The mystery

Is Cameron's reinvention convincing, in the flesh, and in the end? He is a former corporate PR man, so you would expect him to be able to deliver a convincing sales pitch – and he does. He does have some real progress to sell: he talks about getting the Tory conference to applaud gay marriage, and the selection of gay candidates, with passion. His defence of gay refugees and opposition to the blood donation ban went further than he has to politically. Yet there was enough evasion and dissembling in his answers to sow doubts. He didn't tell the truth about his own voting record, and he made ludicrously false statements about his anti-gay European allies. On the biggest obstacles facing gay people – the real, on-going violence – he had little to offer beyond words of condemnation.

David Cameron is a hazy cloud of charm and platitudes: no matter how hard you peer into him, you cannot find anything solid to focus on for long. There are flickers of apparently real pro-gay feeling, but they are soon followed by excuse-making for some of the most anti-gay politicians in Europe. Which is the real Cameron? On this issue, I suspect even he doesn't know. But over the next four years, we are all going to find out: the beaming lights of power will part this mysterious and contradictory fog.