Lord Tebbit is about to explain why we will have to legalise polygamy if gay people are allowed to marry when a plaintive voice sounds at the door. "Can I come in?" his wife, Margaret, calls softly from her wheelchair. "I'm bored."

The deep tenderness between the octogenarian couple is quite something. They live in a beautiful old townhouse in Bury St Edmunds, and in the course of our afternoon, it's often easy to forget that this frail, genial, doting husband, smartly dressed but shrunken and slightly stooped, was once the Thatcherite bogeyman. He talks haltingly, the reedy voice punctuated by lengthy pauses – but he doesn't flag for three hours, and flashes of the old hardcore rightwinger are never far away. When he says he hopes he would have supported women's right to vote 100 years ago, and I suggest there's little evidence of him ever having been at the progressive vanguard, Margaret muffles a giggle and snorts: "Oh, I'm enjoying this!" Tebbit happily agrees, "Up here, we wear the title turnip Taliban with pride."

When I observe that Tebbit caused quite a splash with his recent Big Issue interview, in which he linked gay marriage to incest and said David Cameron had "fucked things up", he enjoys a silent cackle. "I'm afraid the facetious side of my nature came out." Tebbit thinks his comments made such a splash because the public could see he was right – just as his infamous "cricket test" for immigrants still resonates today "because everyone knew it was true". His even more memorable injunction to the unemployed, to "get on yer bike" remains, he thinks, truer today than ever before. "Some people think 40 miles is a long way to go for a job." He pauses for dramatic effect. "Other people come from Prague."

The former Tory minister will always be associated with the 1980s, but it becomes clear that the defining era for Tebbit was the 40s and 50s. Asked how far would be too far to have to get on one's bike today to find work, he offers: "When I joined BA [his first job was as a pilot], I roomed in digs with a family in Hounslow." But did he have a family of his own with him? "Oh no, I wasn't married, so I was independent. But these days, people don't live in digs in that way. To me that's normal, that's what you had to do. Yet I find that people are resistant to that sort of way of thinking now."

One group that do get on their bikes and rent digs are the people he wants to banish from Britain. Quite right, he agrees; the overwhelming majority of applicants to care for Lady Tebbit (who was injured in the IRA attack on the Grand Hotel in Brighton in 1984) have been immigrants from central and eastern Europe, while the few British applicants never even show up for interview. "In other words, they only apply to tell the guys at the benefit office that they're seeking work." So why isn't he thrilled to see these hardworking immigrants boosting our economy? "Hold on. I'm very fond of a lot of these central Europeans. People of my age don't forget that without the Poles we would probably have lost the Battle of Britain. But, of course, there's a downside to it. They're working here and mostly taking their money home." But paying tax here, too? Tebbit looks thoughtful. "It wouldn't be so bad if there were some way that the British people who could be doing the jobs, but aren't, would disappear," he muses glumly. "But they're still there, as an expenditure to the public purse."

How would sending immigrants home make people want to work? "We might realise that we've got to change our ways." But if laziness is the problem, it's not clear why Britons would suddenly want to care for his wife just because continental Europeans no longer could. "Well, we'd have to go for 'IDS-Plus'. We should require of them perhaps to contribute something [in return for unemployment benefit]. Clearing out canals, repairing footpaths, restoring civic gardens. That sort of thing."

He wishes the Thatcher government had reformed welfare properly – indeed, his only regrets are things they didn't do – but explains that in any cabinet, "You've really only got half a dozen people who make things happen. You put them where the need is greatest. So we didn't get to grips with welfare, or education." Neither Keith Joseph nor Kenneth Baker were education secretaries who knew how to "make things happen".

Lord Baker would take exception to that. "I can understand him feeling that, but it's not so. When I went to get my navigator's licence, part of the theory involved spherical trig. How many 16-year-old school leavers now would be able to cope with spherical trig?" Not many. "But why? I'd left school at 16."

More than most of us, Tebbit seems unable to imagine any life other than his own. He will concede that it might have been easier to be a youngster in his day, but goes on: "When I left school I didn't know anyone who had been mugged. I've long sat and wondered why there are so many more acute problems in society than perhaps there were 50 years ago." And apart from welfare, he can see only one obvious culprit.

"It seems to me that ever since the day that Roy Jenkins said the permissive society is the civilized society, society has got less civilized. Most of what has happened in becoming a more permissive society has featured alongside a society which has become much less agreeable." Probably not if you are black, or in a mixed-race relationship, I suggest – and it certainly wasn't safer back then if you were gay. "Well, particularly if they flaunted themselves as such," Tebbit mutters coldly. But heterosexuals can flaunt their sexuality without being beaten up – and what is marriage, if not a public flaunting of one's sexual identity? And so, inevitably, we come to gay marriage.

He contends that because a same-sex marriage can't biologically produce children, it would be illogical for it to exclude incestuous couples. "I think that you have to say, 'Look, we're going to deal with this on the basis of looking at where the law is discriminatory, and we're going to eliminate discrimination.' Be very careful of that argument, is all I'm saying. If you start basing your argument on non-discrimination, you land in some funny places.

"How about polygamy? Should we legislate to legalise it? What's the argument against it? I'm a polygamist, you see, and I'm discriminated against because I'm not allowed to marry several women. Don't I have a case to say that's discriminatory?" But what is the parallel with homosexuality? "Well, the argument is that we are discriminating against homosexuals by excluding them from marriage. We are discriminating, therefore against a polygamist."

Tebbit can't see why gay people are being discriminated against anyway. "I simply take the view that there is at the moment no difference between my rights and a homosexual man's. It's just that he wants to do something which I don't want to do." But he wants to be married to the person he loves, and cannot, whereas Tebbit can. "OK, he doesn't want to marry a woman, that's fine. I don't want to marry a man, and I can't. It is precisely the same position. I mean, some people want to drive at 120mph on the motorway, but we don't let them." Yes, because it's dangerous. "Well, perhaps deconstructing marriage would be a dangerous and harmful thing to society." Why would allowing more people to marry weaken the institution of marriage? "I just think, why don't they go and do something else?"

He opposed civil partnerships. "I confess I was probably wrong about that." But he can't accept calling a gay relationship a marriage, and quotes from the same-sex marriage bill's guidance notes, which state that a man married to a man could be called a wife.

"I'm a wordsmith by profession. I love words. I pick up words from a big pile and put them into a pattern that is pleasing. But when I pick up the word 'wife' and find it can include a man married to a man, I say, 'Hey! Who's doing this to our language?'" But language is evolving all the time. "Yes, but come, come! This is a bit extreme!"

"But do you remember," his wife intervenes, "the two men who were obviously – I mean they were terribly nice … " "Oh, you mean from whom we bought the house in Islington?"

"Yes. I mean, he was definitely, you just knew, the wife." Well, there we are, I say – they felt that the word accurately described the man's family position. Tebbit looks unimpressed. "Do you think it's time for a cup of tea?" asks Margaret.

I ask if he would consider himself a homophobe. "Well, it's interesting that it's allowed for one group of people to insult and shout names at another group without any restriction, but if I were to shout names at that group they would immediately say I was committing a hate crime. No, I'm not a homophobe."

Most people today would say that if you don't want gay people to "flaunt" their sexuality, or express their love in public, you are probably homophobic. "Well, it doesn't mean you hate them. It doesn't mean you're in fear of them. It means you'd rather they didn't do that." But why? He pauses. "Because that's not the way human beings are constructed."

I ask if he is familiar with psychological studies that have tested heterosexual men's response to gay pornography. As I explain to him that electrodes attached to the penis measure sexual arousal, the temperature in the room plummets. The funny thing is, I go on, heterosexual men with relaxed attitudes to homosexuals are unmoved by gay porn – whereas those hostile to homosexuals exhibit sexual arousal. Why does he think that might be? By now Tebbit is looking ill, and his wife has turned white.

"I think there's something weird about people who want to go and have electrodes attached to their penises and watch pornography," he says brusquely, getting to his feet. "I don't think they're a representative group. Margaret, would you like your tea?"

Tebbit has had only one direct conversation with David Cameron since he became party leader, just before the 2010 election, and it didn't go well. "He got very cross. Thoroughly uncommunicative. And then the meeting was over." But while he thinks Cameron has allowed Ukip to seize traditional Tory territory – "I joined the Conservative Party in 1946, and I'm not going to be pushed out by newcomers" – I suspect another reason is the knowledge that he can cause greater embarrassment to the leadership from his own party benches. And although he shudders when I mention Nadine Dorries, he adds: "It's not so much that Cameron and [George] Osborne didn't know the price of milk, but that they didn't know emotionally that the price of milk was important to people. That's the difference."

Oddly enough, almost the only person to get a good word from Tebbit (apart from Boris Johnson, who he loves, but doesn't see as PM material) is Jimmy Savile, with whom the couple became friends through charitable work at Stoke Mandeville hospital.

"I've got no doubt Jimmy Savile was a very odd fellow, and I'm pretty sure he was in breach of the law on a number of matters. But I do not know that it's possible, 40 years on, to do justice in the sense of knowing just how many of those allegations are complete and true."

His wife is nodding as he goes on: "Jimmy did a great deal of good, as well as wrong. And in anybody's life, you have to look at both sides of the ledger."

I ask if the revelations have changed his feelings towards Savile. "Well, I always had my worries about Jimmy, because he was a very odd fellow." What did he fear he might be up to? "I would not have been surprised to find he was having homosexual relationships with young people." But he wasn't homosexual, was he? "Not in general, no, as I understand it."

"But he had," offers Lady Tebbit, "a homosexual air about him."