The UK has defeated European Union attempts to set legally binding environmental regulations for the continent's fledgling shale gas industry, the Guardian has learned.

David Cameron has led intense lobbying against the proposals, arguing that existing rules are strict enough to keep fracking safe and that new rules would delay investment and increase costs.

On Monday, the prime minister said the UK was "going all out for shale" and announced millions of pounds of incentives for local authorities to accept fracking. But opponents have accused him of "hypocrisy" and being "cavalier", while the EU's environment commissioner says there are "clear gaps" in current safety rules.

Leaked documents from the European commission, obtained by the Euractiv news service and seen by the Guardian, show that attempts to safeguard the environment with a new legally binding directive have been defeated by the UK and its allies, which include Poland, Hungary and the Czech Republic. Instead, a set of non-binding "recommendations" covering protection against water contamination and potential earthquakes will be published on 22 January.

Every EU nation will be asked to produce a public "scorecard" within six months stating which recommendations have been implemented: if the commission is unhappy with the results it will push again for new legal rules within 18 months.

The UK and Poland have led the opposition to binding European regulations on fracking, in contrast to France, Germany and Spain. In December, Cameron wrote to the president of the EC, José Manuel Barroso, stating: "It is essential the EU minimise the regulatory burdens and costs on industry and domestic bill payers by not creating uncertainty or introducing new legislation." He added: "The [shale gas] industry in the UK has told us that new EU legislation would delay imminent investment."

The letter was part of a major lobbying effort revealed in another letter, seen by the Guardian, from the UK's top civil servant in Brussels. Ivan Rogers, a former banker at Barclays Capital and Citigroup, wrote in November that "seeing off" the proposals for new laws would require "continued lobbying at official and ministerial level using the recently agreed core script".

Rogers noted that, even if the current proposals were defeated, "we will need a longer term strategy to manage the risks including ... an influencing strategy for the new European parliament and commission".

The European environment commissioner, Janez Potočnik, remained adamant on Tuesday that action was needed to ensure shale gas exploration developed safely. "The studies done by the EC show there are gaps in existing legislation which have to be filled one way or another," Potočnik's official spokesman told the Guardian.

Keith Taylor, the Green party MEP for the south-east – including Balcombe in Sussex where major anti-fracking protests took place in 2013 – said: "It's deeply disappointing that the EC is set to publish proposals that will do nothing to protect EU citizens from the dangers of fracking. The UK government may be pleased with this result but those living near shale gas reserves will be very worried."

Tom Greatrex, shadow energy minister, said: "These documents demonstrate that Cameron sees regulation as an inhibitor of activity rather than a safeguard. By taking such a cavalier stance, he is likely to entrench opposition to fracking rather than taking the responsible approach of ensuring robust regulation and comprehensive monitoring to address the legitimate environmental concerns many people have."

Friends of the Earth's energy campaigner, Tony Bosworth, said: "The [Rogers] letter exposes the UK government's hypocrisy on fracking. In public Cameron and others trumpet the benefits of regulation while behind the scenes the government uses Machiavellian manoeuvres to scupper the regulations and silence the concerns of other member states."

However, a spokeswoman for the Department of Energy and Climate Change said: "The UK has more than 50 years' experience regulating onshore oil and gas and we believe that decades of ambitious EU environmental directives has left the EU uniquely suited to handling the challenges of shale gas development without further legislation. Guidance will be needed to clarify how existing EU legislation applies to shale gas."

Ken Cronin, chief executive of UKOOG, the UK's onshore oil and gas trade body, cautiously welcomed the UK government's defeat of the EC proposal: "There are already nine different European directives covering the onshore oil and gas industry, so it appears to be good news that we don't have to comply with a tenth. However, we would need to see the detail before we can wholeheartedly welcome them."

The UK has already rejected a call from its own Royal Society and the Royal Academy of Engineering to develop national "guidelines specific to shale gas extraction".

Concern about the environmental risk posed by fracking has fuelled widespread public opposition. Following Monday's announcement that local authorities would be able to keep 100% of business rates from fracking projects rather than the usual 50%, the Conservative leader of East Cheshire council ruled out shale gas exploration, an embarrassment for shale gas supporter George Osborne, whose constituency is in the area.