The health secretary, Matt Hancock, has ruled out scrapping GP home visits after doctors backed the idea in a vote, arguing that they were too over-stretched to deliver the service.

Delegates representing GPs across England at a British Medical Association conference voted to try to remove the duty from their standard contract, after complaints that they were wasting time driving around the country.

The move clears the way for BMA representatives to lobby NHS England over the measure, but Hancock said the idea was a “complete non-starter”, and he was firmly opposed to the plan.

“The GPs had a vote on what their opening negotiating position should be for the next GP contract. The idea that people shouldn’t be able, when they need it, to have a home visit from a GP is a complete non-starter and it won’t succeed in their negotiations,” he told BBC Radio 4’s Today programme.

“Of course, people need to be able to see their GP and we’re putting huge amounts of extra resources into training, funding and hiring more GPs, and the other practice staff, which are also so important. But you do need, sometimes, for the GP to be able to go and see somebody.”

Challenged on whether it specifically needs to be an overstretched GP visiting, rather than another medical professional, Hancock replied: “Sometimes it should be somebody else. Indeed, most home visits are done by nurses and that’s right but sometimes you need the GP.”

At the Local Medical Committee England Conference on Friday, a majority of delegates supported a proposal to remove home visits from the doctors’ key contract.

Kent local medical committee, an independent body working with the BMA to help to shape policy, called for the change. It argued that “GPs no longer have the capacity to offer home visits”, urging the BMA’s general practitioners committee to renegotiate with the NHS to “remove the anachronism of home visits from core contract work, negotiate a separate acute service for urgent visits and demand any change in service is widely advertised to patients”.

After passing the motion, Kent LMC clarified that it was not trying to stop home visits entirely but change policy to ensure suitable provision. A separate motion for the general practitioners committee to negotiate an acute service for urgent home visits was also passed. A Kent LMC statement said: “This motion is not intended to remove the ability of GPs to perform home visits. More complex, vulnerable and palliative patients are best served by their GP visiting them when needed.

“Currently there is no universal consistency for patients. Increasing demand and falling GP numbers are further compounding pressures in general practice.”

But some critics have rounded on the measure. Joyce Robins, of the campaign group Patient Concern, is among those condemning the vote, arguing that seriously ill people rely on the visits.

Nikita Kanani, a London GP who is also the NHS’s national medical director for primary care, said patients who clinically required a GP home visit would still receive one.

The shadow health secretary, Jonathan Ashworth, said his party’s plan for the NHS would mean recruiting and training more GPs “so home visits can continue”.

He said: “The crisis in the family doctor service is just one more area where the Conservative government has let down patients and the NHS.

“In 2015 they promised more GPs, which like their pledges on everything else, never happened. And now there is a serious threat that elderly and vulnerable patients who rely on home visits will have that vital support removed.”