Edinburgh seems the perfect place to be speaking to George RR Martin, who appeared this week at the book festival, because the residents of the city include one of the few other authors able to empathise with his predicament. The closest parallel to the impact made by JK Rowling's seven-book fantasy sequence and related movies about Harry Potter is Martin's projected seven-book series called A Song of Ice and Fire and the globally successful spin-off HBO miniseries called Game of Thrones, named after the first of the five books that have so far appeared.

Rowling once told me in an interview that she still suffered moments of shock at the global industry and cultural phenomenon that had resulted from telling a story. So does Martin feel a similar sensation at what he has started?

"I wouldn't use the word shock. But, yes, it does seem unreal at moments. I'm constantly forgetting that my life has been transformed. I think Rowling was a different case because they were her first books. For me, I'd had 20 years of fantasy and science fiction books that had done well, but not like this. And there's part of me that makes the unconscious assumption that I'm still that person and can live that kind of life. And then I'm reminded that I can't because Game of Thrones has become such a phenomenon and I've become a celebrity. There are nice things about it and not so nice."

The negative aspect, he says, is "loss of privacy and the fact that it's out of your control". He and Parris, his third wife, came to Edinburgh three or four years ago and were "able to listen to the street musicians and go to plays and comedy performances and the military Tattoo. And, in the whole week we were here, maybe three people recognised me, and I was happy to sign an autograph for them. Now, three or four people recognise me every block. I can't go out any more; I can't walk the streets. And it's great to have all these readers and fans who, for the most part, are very nice people, saying they love the books and the TV show. But there are so many of them and it just doesn't end. Oh, and 'selfies'! If I could clap my hands and burn out every camera phone in the world, I swear I'd do it!"

He speaks nostalgically of his years of manageable fame: "I could go to a science fiction convention and put on a name badge. And I'd get this thing that science fiction fans call an 'ego-boo' – ego-boosting – when people see the badge and say: 'Oh, George RR Martin, sign my book.' But then I could take off the name tag and become just some guy and go to a restaurant. Now I can't take off the name badge." Martin's reference to "most" of his readers being nice people sounds calculated to exclude those who speculate about his longevity or aggressively demand the sixth and seventh volumes of A Song of Ice and Fire. Websites devoted to the author increasingly resemble a literary maternity ward with a crowd outside shouting "Push! Push!"

"That's one of the pressures on me," Martin acknowledges. "My publishers want the [next] book, so do HBO, so do my readers. And nobody wants it more than I do. But, when I weigh that pressure, I have to be aware that the books are what I'll finally be judged on. If the novels are still being read in 50 years, no one is ever going to say: 'What's great about that sixth book is that he met his deadline!' It will be about how the whole thing stands up."

In person, the writer is an epic clash of white – his snowy clouds of hair and beard – and black: the colour of the tee-shirt and dungarees. He is a big, amiable, Santa-like man, with an affecting high-pitched giggle. This general demeanour was one reason there was such shock when he recently snarled "fuck you!" at a questioner who queried whether, as a 65-year-old with a high body mass index, he was sure he could complete the last two books.

But the apparent feeling among Martin's fan base that he should lock himself away until the narrative is completed reflects the extraordinary success of the franchise, which began quietly, in 1996, with the publication of Game of Thrones, introducing the fantasy continents of Westeros – where seven kingdoms, uneasily united under a ruling dynasty, are sliding into civil war – and Essos. The northern border of Westeros is protected by an ancient wall of ice, an aspect of his imagined world that has provided Martin with a smart answer to all those who have pressed him, during the Edinburgh visit, to take a side in the Scottish independence debate: "I've suggested building a giant wall of ice between the two nations."

As it happens, fiction has already imitated Scottish history because Westeros's defensive barrier was inspired by Hadrian's Wall. His study of Scottish, Irish and English (especially the wars of the roses) military history has informed the stories, which also draw on literature. The title and general atmospherics of A Song of Ice and Fire were influenced by a poem called 'Fire and Ice' by the appositely-named American writer Robert Frost. Published in 1920 – just after the end of the first world war – the short poem contemplates the nature of a final planetary conflagration: "Some say the world will end in fire / Some say in ice. / From what I've tasted of desire / I hold with those who favor fire. / But if it had to perish twice, / I think I know enough of hate / To say that for destruction ice / Is also great / And would suffice."

Apart from suggesting the extreme and extended seasons of heat and cold that occur in Martin's world, it strikes me as significant that Frost's poem also contains the words "hate" and "desire", which, in intense scenes of sex and violence, are the driving forces of the narrative.

"Yeah. Love and hate, sex, revenge: these are a big part of history and a big part of my books. You read about the wars of the roses and it's an endless series of revenge killings, effectively."

Game of Thrones was named after the first in Martin’s A Song of Ice and Fire series

We were talking on a day this week when the news headlines were again dominated by the conflicts in Syria, Iraq and Gaza, which felt appropriate because one reading of A Song of Ice and Fire is that war and border disputes are almost inevitable to human communities? "Certainly, one of the major themes of the books is war. Almost all fantasy fiction since Tolkien has been concerned with war. In the Tolkien imitators, it's always a fight between good and evil, and the evil ones wear black or are ugly. I wanted to stand some of those things on their heads and so I put my good guys – the Night's Watch men – in black, and there's good and evil on both sides. But it's not an allegory. If I wanted to write a novel about Vietnam or Syria, that's what I'd do."

Despite that disclaimer, it seems to me that one reason for the huge success of both the book and TV versions is that people of different races and places detect their own experience in Martin's alternative world. Depending on your perspective, the great dividing barrier in the books could be Hadrian's Wall, the Great Wall of China or the security fence on the West Bank.

"Well, I can settle that one easily. It was Hadrian's Wall. I've never been to China."

"But the point I'm making is that readers may see their own meanings?"

"Oh, sure. And those meanings may very well be there. An author is not necessarily infallible when discussing his own work because so much takes place in the subconscious."

One example of unintentional influence on fiction is the place in which a writer lives and works, and it strikes me as interesting that Martin has written his epic about clashing civilisations in New Mexico, on the border between an ancient culture and a modern superpower: "Yeah. There are certainly three distinct cultures there – the original Indian culture, the Spanish and then the Anglo civilisation that conquered that – and I'm a kid from New Jersey, so I bring my own perspective to it all."

Living in a US state that has a border with Mexico, Martin is also well placed to observe the fear of invading foreign hordes that seems to have become a feature of most societies.

"Yeah," he sighs. "Lately, it certainly seems to be, which is distressing, especially in America. The US has been the ultimate non-homogeneous culture, created by immigrants. So how hypocritical can we be to get concerned with immigration now?"

Underlying Martin's books are two rejections of institutional dogma: first, lapsing as a young man from his family's Catholic faith – "even as a kid, I asked questions that made the nuns and priests uncomfortable" – and then refusing to fight in Vietnam, succeeding in being classified by his local draft board as a conscientious objector. "Often, to get that status, you had to be a total pacifist on religious grounds. Now, number one: I'm not religious. And number two: I'm not a complete pacifist. They would say, 'Would you fight if Hitler was raping your grandmother?' And I'd say, 'Yeah, probably would.' But they passed me, anyway. I sometimes think the second world war has changed our entire western civilisation's view of war, because, of all the wars in history, the second world war is closest to fantasy war, in which there is a dark lord, whose guys are actually evil and dress in black and wear skulls on their uniforms. The first world war was a much more typical war: what were all those people really fighting for?"

Many of Martin's comments end interrogatively, which, he explains, reflects his temperament: "I'm not the sort of writer who gives answers; I prefer to raise questions. One of the most pleasurable things about the success of the books has been the number of debates they have sparked, with people arguing online about what was the right thing to do."

Are there thousands of thrown-away pages or does he write to a target length? "There are discarded chapters and paragraphs and everything in-between. Because I follow the characters and they sometimes lead me down dead ends. So, at the end of the sequence, I will probably have tens of thousands of words or even hundreds of thousands of words of unused material."

"Would you ever publish them?"

"Mmm, no. Some of them are just earlier versions of scenes that appear."

"But that wouldn't bother your fans."

"I guess not. Well, there's at least one deleted chapter from book five – which left me in the wrong place – which I have been tempted to publish as a sort of short story."

The Frost poem that inspired him is about the end of the world, and this seems to hint that Martin's invented universe must perish, through heat or cold or possibly both, at the end of the seventh book.

The writer cackles: "Well, I'm not going to comment on that. You can worry about that for two books. But it's true that all men must die."

Finally, while it seemed unlikely that he would reveal the state of readiness or probable publication date of the sixth book, The Winds of Winter, I wondered what answer he gives to the fans and publishers who presumably ask him all the time.

"I tell them all the same thing," Martin says, with slightly steely sweetness. "It will be done when it's done."