Naturally, it kicks off with a message from the PM. "Our conference is the perfect opportunity … for all delegates and participants to discuss the key issues of the day with a wide variety of politicians, charities, business leaders, international guests, and interest groups." This is the image the Tories want to push in their sales brochure for sponsors: that this week's get-together in Manchester is less about the clash of ideas, and more an exclusive networking event.

On it goes, touting "exciting opportunities" to highlight your firm's message. For a minimum of £20,000 plus VAT, you get to sponsor the VIP lounge (the Blue Room, natch) where "high-profile guests can relax, hold briefings". Illustrating the point is a photo of Iain Duncan Smith and Tory pollster Lord Ashcroft having a private moment by a tray of sundaes. Too expensive? Then £5,000-plus will get you access to an invite-only party, with the implicit promise of facetime with a minister. Another picture of Ashcroft, this time at dinner with George Osborne and Ken Clarke.

But the most telling page has neither ad copy nor photos. Under the headline "Who comes to our conference?" is a pie chart: 4% event support; 2% international; 20% media. Then the two biggest slices: 38% party members and 36% commercial/charity/exhibitors. In other words, barely a third of those swarming Manchester this week are paid-up Conservatives – and they are almost equalled by corporate lobbyists.

Of all the mainstream parties, it is the Tories that have retreated furthest and fastest from being a mass party. In 1953, the party had 2.8 million members. By 2001, that was 311,000. Under Cameron, the figure is 134,000. And that's if you believe the numbers from Tory HQ; some observers, not all hostile, think the total is on the verge of dropping below six figures. As well as withering, the grassroots are ageing: the average member is now 68 years old. The party "for hardworking people" is in reality the party of OAPs.

This is not a column bemoaning the decline of mass parties: enough of those are written every conference season. No, the question I'd like to ask is what happens when a party of government loses its membership and has to go outside for funding. In the case of the Tory party, the answer appears to be: the bankers move in.

At the last general election, more than half of all Conservative funding came from the City, according to extensive research by the Bureau of Investigative Journalism. That's double the amount when Cameron won the leadership in 2005, and financiers made 25% of Tory donations. By financiers, I'm not referring to the high-street giants that you bank with. I mean Adrian Beecroft, Michael Farmer and Lord Stanley Fink: hedge fund speculators, private-equity barons and other members of what is sometimes called the shadow banking industry.

I've used these figures before on this page, but they deserve to be quoted again. Because what they suggest is that the shadow bankers used the boom of the last decade to buy themselves unprecedented political influence. And the primary way they did that was by buying a substantial stake in the Tory party.

As investments go, it's paid a handsome dividend, because the Conservatives have fought fiercely for the interests of the City. Yesterday, George Osborne announced a plan to make life much tougher for those out of work for a long time, including a proposal to make them sign on at a jobcentre every day. Last week, the chancellor announced he would fight the EU in court for the right of bankers to be paid big bonuses. Life for the poorest in austerity Britain to get harder; life for the richest to be made easier: surely even the most tin-eared of politicians wouldn't get caught out like that, unless their war chest depended on it.

Even while negotiating over last autumn's mini-budget, the Conservatives sent out begging letters to donors to stymie Vince Cable's plans for a tax on £2m mansions. The pleas read: "We promise that no homes tax will be introduced during the course of this parliament, but the only way of taking it off the table in the future is the election of a majority Conservative government in 2015, and we can only win with your generous support."

And then there are the other ways in which the shadow bankers have prospered thanks to the Conservatives. The scrapping of the 50p tax rate for the super-rich. The indulgence shown to private-equiteer Adrian Beecroft as he drafted his report on how to take rights off workers. And the way that contracts to run academy schools and parts of the NHS keep going to Tory financiers. In 2011, the contract to run NHS Hinchingbrooke hospital in Cambridgeshire was awarded to Circle, a company part-owned by hedge funds run by three of Cameron's biggest donors: Paul Ruddock, Crispin Odey and Michael Platt.

By becoming a party for the shadow bankers, the Tories are going against what their own members want. This week at a Tory fringe event, political scientist Tim Bale presented results of a survey of 852 party members. The rank and file, he showed, want more local and working-class MPs. And between 20% and 25% of them believe that big business benefits owners at the expense of workers; and also agree with the proposition that there is one law for the rich and one for the poor. These are the classic little guys of British Conservatism; and in Cameron's party they no longer count.

The standard rap on the party is that it is shrinking into the south-east, with not a single councillor in Manchester, Newcastle, Sheffield or Liverpool. But its ability to reach into the north will always be limited as long as it follows an agenda partly set by the City.