A key member of David Cameron's inner circle was at the centre of controversy tonight after he was filmed stating that the prime minister and his deputy, Nick Clegg, want their "people power" revolution to unleash "chaotic" effects across local communities. The comments, by Nicholas Boles, Tory MP for Grantham and Stamford, were made 10 days ago during a debate in Westminster hosted by the polling organisation Ipsos MORI. During a question-and-answer session on the "big society", Boles – viewed by Cameron's circle as an "outrider" for imaginative thinking on policy – was asked why he seemed to prefer "chaos" to central planning of services.

Boles replied that he, Cameron and Clegg did not believe in central planning and that it would be a good thing to have different communities offering differing types of services, even if the appearance was chaotic.

"I mean, bluntly, there comes a question in life," he told the audience. "Do you believe planning works? That clever people sitting in a room can plan how people's communities should develop, or do you believe it can't work? I believe it can't work, David Cameron believes it can't, Nick Clegg believes it can't. Chaotic therefore in our vocabulary is a good thing.

"Chaotic is what our cities are when we see how people live, where restaurants spring up, where they close, where people move to. Would you like to live in a world where you could predict any of that? I certainly wouldn't. So I want there to be chaotic in the sense I want lots of organisations doing different things, in different areas."

Contacted by the Observer, Boles did not try to withdraw the remarks but said he had merely been trying to explain that in his view central, top-down planning did not work and local variations in services could be a good thing. But his remarks will be a serious embarrassment to the coalition after local government secretary Eric Pickles announced the most severe cuts in local government funding for a generation, with some of the poorest areas receiving the biggest reductions.

On the back cover of his recently published book Which Way's Up?, Boles is described as having been "one of David Cameron's most influential advisers before the election".

Tonight Labour accused the coalition of destroying local services for ideological reasons. Shadow local government secretary Caroline Flint said: "Nick Boles' alarming comments reveal how out of touch David Cameron and Nick Clegg's government is with ordinary people. They want to bring chaos to towns and cities simply to satisfy their own ideological curiosity."

Jon Cruddas, Labour MP for Dagenham, who wants Labour to develop its own vision of a "big society" based on strong local institutions, said: "This reveals that the Tory approach to the big society is literally a recipe for chaos, bordering on anarchy."

At the same Ipsos MORI event Lord Adonis, the former Labour transport secretary, said the coalition should be careful not to suggest "chaos" as a desirable outcome of policy, warning that if they did there would be "shades of the poll tax". Boles' statements have echoes of comments made in 2005 by another key Cameron ally and one-time adviser Danny Kruger. He was forced to resign as a Tory candidate for Sedgfield after he was quoted as saying that the Conservatives "plan to introduce a period of creative destruction in the public services".