David Cameron's new minister for civil society has been branded patronising and dismissive after he told charities to "stick to their knitting" and keep out of politics.

Brooks Newmark, who was appointed in the summer reshuffle, made the comments amid worries among charities that the new Lobbying Act that will limit their ability to campaign on issues of the day.

In his first major speech since he took on the role, Newmark used the opportunity to criticise charities who "stray" out of their remit of helping people.

Asked about the ability of charities to campaign, he said: "We really want to try and keep charities and voluntary groups out of the realms of politics. Some 99.9% do exactly that. When they stray into the realm of politics that is not what they are about and that is not why people give them money."

In comments first reported on Civil Society, he added: "The important thing charities should be doing is sticking to their knitting and doing the best they can to promote their agenda, which should be about helping others."

Newmark's comments were condemned as "patronising rubbish" by Lisa Nandy, the shadow minister for civil society. "It's his first speech as charities minister, and I think it's not just patronising but actually deeply offensive at a time when charities are picking up the pieces from this government's awful, unfair policies, that their ministers would talk about them in such a dismissive way," she said.

"This comes from a government that hasn't just introduced the Lobbying Act but has also restricted charities using judicial review, cut legal aid, ramped up employment tribunal fees and clamped down on immigration appeals. What you're looking at is a government that doesn't like challenge. That is quite a frightening place for a government to have got itself into."

Alexandra Runswick, director of Unlock Democracy, a campaign group, said charities "should be able to advocate, not just knit" and suggested that Newmark's tone was condescending. Frances Crook, chief executive of the Howard League for Penal Reform, also tweeted that it was "incredibly insulting for charities minister to tell civil society to 'stick to its knitting' and I think sexist too".

Several backbench Tory MPs earlier this year attacked Oxfam for opposing benefit cuts and zero-hour contracts, saying this amounted to being too political.

However, charities have strongly defended their right to comment on and campaign on issues relevant to their work. At the time, Oxfam policy director Ben Phillips said the charity is a resolutely non-party political organisation that has a "duty to draw attention to the hardship suffered by poor people we work with in the UK".

Newmark later claimed he had only been referring to keeping charities out of "party politics", not stopping them from campaigning in general. "Charities, with all their expertise, have an important role to play in helping to shape government policy. While they have the right to campaign they should stay out of the realm of party politics," he said.