John Whittingdale has said the BBC’s approach to impartiality drives him “insane” and joked about abolishing the corporation in a talk at Cambridge University’s Conservative Association.

In a marked departure from careful public statements made ahead of the publication of a white paper on the next BBC charter, the culture secretary also indicated his commitment to hand part of the licence fee to outside organisations to make programming such as children’s TV, arts coverage and local news. “There is a case for having some plurality, so that the decision as to what programmes are commissioned isn’t exclusively taken by the small group of commissioning editors at the BBC,” he said.

Director general Tony Hall last week sought reassurances from the chancellor, George Osborne, that so-called “top-slicing” would not take place, having considered the prospect ruled out under the funding agreement struck with the Treasury last year.

The comments, which were first reported by Cambridge University’s student newspaper Varsity and verified by Broadcast, were made at an event last Friday, in which he described the BBC as “a market intervention of around £4bn by government”.

They reveal Whittingdale’s dissatisfaction with the way the BBC handles complaints over impartiality. He told the association that “sometimes they will drive me insane” and confirmed he would strip responsibility for handling complaints from the BBC Trust.

The pro-Brexit minister added that the corporation “finds it difficult to take seriously people who have a different view of the world” and “has always regarded people who want to leave Europe as faintly mad [and] it has generally been in favour of spending public money rather than those who want to see lower taxes”.

He also jokingly said of the BBC’s charter, which runs out at the end of 2017, “if we don’t renew it, it may be that the BBC will cease to exist, which is occasionally a tempting prospect”.