Conservatives rounded on their shadow home secretary today after he attracted fresh concern about intolerance at the top of the party by saying owners of bed and breakfasts should be allowed to turn away gay couples.

Colleagues of Chris Grayling sought to limit the damage done by his comments, made to a thinktank last week, that he believed individuals running B&Bs from their homes should be allowed to turn homosexual guests away.

Increasing speculation that he would not become the home secretary should the Tories win the election, one shadow minister said today: "There's an attempt [by Labour] to suggest Grayling's comments reveal party-wide homophobia, whereas the real reason this is running is that Chris Grayling is just not up to the job."

On Wednesday Grayling was recorded as saying: "I think we need to allow people to have their own consciences. I personally always took the view that, if you look at the case of should a Christian hotel owner have the right to exclude a gay couple from a hotel, I took the view that if it's a question of somebody who's doing a B&B in their own home, that individual should have the right to decide who does and who doesn't come into their own home."

Today the Tories fielded a minimal defence of their shadow home secretary, only pointing to Grayling's own statement and highlighting that these comments were not an expression of his current point of view but a recollection of his previous opinion. He had since changed his mind, reflected in his more recent vote in favour of legislation banning B&B owners from discriminating against gay people, the party said.

The recorded comments emerged on the eve of a general election in which the Tory party is attempting to woo Liberal voters and gain seats in the south currently held by the Liberal Democrats by proving it will be tougher on discrimination and embrace equality.

Gay rights campaigner Peter Tatchell called on Grayling to apologise and said Cameron needed to speak out on the issue.

On Friday Mirek Topolánek resigned as head of the ODS party in the Czech Republic for homophobic remarks. The ODS party has worked with the Conservatives to create the new anti-federal grouping in the European parliament in which both the Tories and ODS now both sit.

In a recent interview with Gay Times, Cameron was unable to recall that his MEPs have voted against European legislation that would protect homosexuals from discrimination.

Labour seized on the comments, saying Grayling was not fit to be home secretary. Ben Bradshaw, the culture secretary, said: "What is critical at this election is whether David Cameron is for real and whether his party has actually changed. Yet again the mask has slipped showing the Conservatives haven't."

Defending his comments, Grayling said: "Any suggestion that I am against gay rights is wholly wrong – it is a matter of record that I voted for civil partnerships. I also voted in favour of the legislation that prohibited bed and breakfast owners from discriminating against gay people. However, this is a difficult area and on Wednesday I made comments which reflected my view that we must be sensitive to the genuinely held principles of faith groups in this country. But the law is now clear on this issue, I am happy with it and would not wish to see it changed."

But Tory blogger Iain Dale criticised the shadow home secretary for presenting the Labour party with an open goal, and also took issue with Grayling's argument about being sensitive to faith groups.

He said in his blog: "I fundamentally disagree with him on the main issue. This is not about property rights. If you open your house to paying guests, it is no longer just your house.

"Do I believe Chris Grayling is homophobic? No, I do not. Do I believe that his views, as expressed, will be damaging. Yes I do.

"I do not think we must be sensitive to faith groups who promote discrimination and bigotry. We wouldn't be sensitive to groups who provoke racial discrimination, so why should we be sensitive to those who appear to condone sexual discrimination ... He is shadow home secretary and will have to defend himself. Because I am afraid I can't."

Grayling had been known as the Cameroons' messenger to middle England. But he has run into trouble, most notably at last year's Tory party conference when he criticised the appointment of Sir Richard Dannatt, the former chief of the army, as an adviser as a stunt. He thought the government was hiring Dannatt but later discovered that it was actually his own party.

Case study

Keith Flood, retired

I will have been with my partner for 40 years, next year. From the start of our relationship, we've been on holiday all over the place: Paris, London, the Lake District.

It was always amusing in the Lakes when you went as part of a group somewhere and you would see people trying to work out who was with who. We have never experienced any huge problems with discrimination, but I do remember a time in a rather nice hotel in Harrogate about 12 or so years ago. My partner was checking in, while I parked the car. When I arrived there was some sort of mild panic as the young man realised that we were two men and he only had a room with a double bed to offer us. We reassured him that was perfectly fine. There was another occasion in Scotland when there was a slight raising of eyebrows as we were checking in. But it's very much in my makeup to say: "So what?" If I feel there is a problem, I just confront it. I think in the past we would always head for places that were gay-friendly, but that was more for fun than to avoid confrontation. Anyway, now you don't even think about it. There used to be that slight nervous apprehension, but nowadays there is no embarrassment at all. For Chris Grayling to say it's OK to discriminate just feels like he is 50 years out of date.

Still, it has become easier in all aspects of life – insurance, flights, next-of-kin. In my lifetime, the major social changes, such as gay rights, have come under a Labour government. Tony Blair changed the Tory party; he made it far more acceptable [to be gay]. I can't imagine the Tories going back on that now, even if David Cameron doesn't have a particularly good record on gay rights.

As far as B&Bs are concerned, I think the idea that anyone running a business should even think of asking about your sexuality is absurd. What on earth has it got to do with them? What do they ask? "Are you planning to touch each other?" Our money is just as good as anyone else's. If you are offering a service, you have no right to impose your so-called moral values on others.

Interview by Alexandra Topping