The Tories were embroiled in a furious row over lesbian and gay rightson Saturday after the shadow home secretary, Chris Grayling, was secretly taped suggesting that people who ran bed and breakfasts in their homes should "have the right" to turn away homosexual couples.

The comments, made by Grayling last week to a leading centre-right thinktank, drew an angry response from gay groups and other parties, which said they were evidence that senior figures in David Cameron's party still tolerate prejudice.

In a recording of the meeting of the Centre for Policy Studies, obtained by the Observer, Grayling makes clear he has always believed that those who run B&Bs should be free to turn away guests.

"I think we need to allow people to have their own consciences," he said. "I personally always took the view that, if you look at the case of should a Christian hotel owner have the right to exclude a gay couple from a hotel, I took the view that if it's a question of somebody who's doing a B&B in their own home, that individual should have the right to decide who does and who doesn't come into their own home."

He draws a distinction, however, with hotels, which he says should admit gay couples. "If they are running a hotel on the high street, I really don't think that it is right in this day and age that a gay couple should walk into a hotel and be turned away because they are a gay couple, and I think that is where the dividing line comes."

Ben Summerskill, chief executive of the gay rights group Stonewall, said the comments would be "very alarming to a lot of gay people who may have been thinking of voting Conservative".

He added: "The legal position is perfectly clear. If you are going to offer the public a commercial service – and B&Bs are a commercial service – then people cannot be refused that service on the grounds of sexuality. No one is obliged to run a B&B, but people who do so have to obey the law. "I don't think anyone, including the Tories, wants to go back to the days where there is a sign outside saying: 'No gays, no blacks, no Irish.'"

Labour said that Grayling's comments ran contrary to the Equality Act (Sexual Orientation) Regulations 2007, which state that no one should be refused goods or services on the grounds of their sexuality.

Grayling voted in favour of the regulations, which apply to the provision of "accommodation in a hotel, boarding house or similar establishment".

Last month, a Christian B&B owner in Cookham, Berkshire, was reported to the police for refusing to take in a gay couple as guests. Susanne Wilkinson said she had expected a man and woman, but when two men turned up she did not feel she could accommodate them because to do so was "against her convictions". The couple said they were considering suing, not for money, but "for a principle".

Chris Bryant, the Europe minister, who last weekend became the first gay MP to be married in the Commons, said from his honeymoon in Edinburgh: "Anybody who thinks that the Tory party has changed should think what it would be like to have Chris Grayling as home secretary. It is impossible to draw a distinction between bed and breakfasts and hotels. It is very clear that very senior Tories have not realised that the world has moved on."

A Conservative spokesperson said last night that Grayling had been clear about the obligations on hotel owners, but declined to be drawn on his views on B&Bs: "Chris Grayling was absolutely clear that in this day and age a gay couple should not be turned away from a hotel just because they are gay couple."

The row will alarm David Cameron as he prepares for a general election that looks certain to be called on Tuesday. The Tory leader has gone out of his way to win over gay and lesbian voters by stressing his new-look party's liberal credentials. Last year, he apologised for section 28, the law passed by Margaret Thatcher's Tory government in the late 1980s that bans the promotion of homosexuality in schools. Cameron has also voted in favour of civil partnerships.

However, his progress in attracting the gay vote has been halted by a series of disputes involving his own MPs and MEPs. Tory MEPs last year refused to support a motion that condemned a new homophobic law in Lithuania.

Cameron was also left embarrassed during a recent interview with Gay Times, broadcast by Channel 4 News, in which he admitted he did not know his party's position on a series of votes involving gay rights issues in the UK and European parliaments.

Chris Huhne, the Liberal Democrat home affairs spokesman, said: "Chris Grayling's plan would allow discrimination to thrive, as every bigot was given a licence to opt out of equality rules. These views… show how far the Conservative party still has to travel before reaching the modern age."

The culture secretary Ben Bradshaw, who is openly gay, said: "What is critical at this election is whether David Cameron is for real and whether his party has actually changed. Yet again the mask has slipped."