David Cameron's use of exclusive dining clubs, where major donors can meet top government ministers, has been shown to be a key part of Conservative party funding, as concern grows among voters about Tory links with the wealthy.

As Ed Miliband seeks to reduce Labour's dependency on the unions, who have given his party some £125m since 2001, today's Opinium/Observer poll shows more people (56%) are concerned about Tory links to donors than they are with Labour's to the unions (48%).

Analysis of the dinners – attended by hedge fund managers, the business elite and some of the City's most powerful financiers – confirms that they have become a major income generator for the party. In the first quarter of this year, 31 people who figure on the party's list of major donors attended an elite Leader's Group event, where they dined with a Tory minister. Of these, 12 stumped up almost £1m in donations between them, in their own names or through their companies.

While Labour is struggling to refute claims that it is in thrall to the unions, the prime minister is finding it hard to reject the charge that he is too close to some major donors. Bloggers have been quick to point out that in 2006 Cameron opened a factory in India for JCB whose chairman, Sir Anthony Bamford, is one of the Tories' biggest donors.

Bamford, and several Lib Dem and Labour donors, also joined Cameron on a subsequent high-profile trade mission to India, as did representatives of textiles firm J&H Sales, which has given the Tories almost £200,000 since 2010.

Others who have enjoyed the Cameron personal touch include the staff of a Bestways Cash and Carry, who were treated to a Q&A with the Tory leader in 2010. Their employer has given more than £200,000 to the Tories in the past three years.

Midland Chilled Foods gave the Tories £10,000 in February this year, and in 2006, the year Cameron opened its new depot in Basingstoke, gave £3,000. The company is owned by Peter Shirley, a member of the Midlands Industrial Council, an obscure organisation that has funnelled hundreds of thousands of pounds to the Tories and whose backers include Chris Kelly, whose company, Keltruck, is a global distributor for Scania lorries. Cameron and George Osborne visited Scania's production plant in Sweden when in opposition and were photographed driving one of its trucks.

Another PR opportunity was provided when Cameron met several key donors on his recent visit to Kazakhstan, a trip criticised by human rights groups because of the country's autocratic political regime. Ayman Asfari, chief executive of UK oil and gas firm Petrofac, who has given the Tories nearly £300,000 in the past three years, signed a deal to expand his company's operations in Kazakhstan during the trip.

Gary Lydiate, a businessman who has given more than £200,000 to the Tories and was given five minutes with Cameron during the trip, told his local paper: "When you're travelling with a high-level delegation led by the prime minister, you get to see the right decision-makers."

During last week's prime minister's question time, Cameron was asked about the chief executive of Tullow Oil, Aidan Heavey, who has given the Tories more than £60,000. Foreign secretary William Hague branded Labour's shadow Treasury minister Cathy Jamieson a "stupid woman" for querying whether the payments were linked to his decision to take up the oil company's case in a tax dispute. But a press release from the Ghanaian government confirms that Cameron, too, discussed the company's interests in Ghana when its president, John Mahama, visited No 10 recently. Cameron is also expected to tour the company's oilfields when he visits Ghana later this year.

While there is no suggestion of impropriety, some donations have not been without controversy. There were calls recently for Cameron to return more than £500,000 that had been donated by Ian Taylor, the chief executive of oil trader Vitol, which paid Serbian war criminal Arkan $1m in 1996 for his help in recovering a debt. The size of Taylor's donations make him one of the elite invited to intimate Leader's Group dinners with the prime minister and other senior Tories.

Others who qualify for a personal dining invitation with the prime minister include May Makhzoumi, the wife of Fouad Makhzoumi, a former Lebanese arms dealer who was involved in the scandal that brought down Jonathan Aitken; former non-dom Lord Ashcroft, who was appointed by the prime minister to conduct a review of UK military bases abroad; and Michael Farmer, whose metal-broking business operates several funds in the tax haven of Bermuda, who has given the party around £1.5m in the past two years.

Critics said that until the prime minister reforms the laws on party donations, public concern about the links between big business and Westminster will endure, to the detriment of trust in political life. "It's about time David Cameron got out of hock to the National Union of Diners and Donors," said Lib Dem peer Lord Oakeshott. "He pretends to be in favour of a donation cap but he won't do anything about it."

A spokesman for the Conservative party said: "All donations are fully permissible and declared with the Electoral Commission."