When the war reporter Sean Langan was leaving for Afghanistan in 2008, he left his bosses with a strange request: “If I get kidnapped, don’t let my friend Toby Young try to help me. If he can alienate liberal people in London, God forbid if he starts talking about towelheads and Islamic nutcases. I’ll get beheaded.”

It was prescient. Langan was kidnapped by the Taliban and when Channel 4 convened to discuss a rescue plan, Young showed up proposing he fly immediately to Kabul and spearhead negotiations.

Impulsive, ambitious, vain and generous, the gesture, which was firmly rebuffed, was Young to a T. Langan was eventually freed after three months with the help of professional negotiators.

Young, 54, was unveiled on Monday as a board member of the government’s new higher education regulator to a chorus of boos. It is his first public office appointment after a prolific writing career in which he has trawled his life for material, from taking so much cocaine in the Groucho Club that half his face froze, to taking his four young children grouse shooting.

The outrage this week was sparked by a series of comments, mostly on Twitter around seven years ago, about women’s breasts, a subject which appears to obsess him. His observations that his wife has “Baywatch tits”, Padma Lakshmi had “massive boobs” when pregnant and “Danny Boyle’s wife’s got huge knockers”, had Labour and teachers’ unions calling for his appointment, to the board of the new Office for Students, to be rescinded.

Juvenile and politically incorrect as he admits his comments were, they were part of a career-long tendency to seek attention and to bait liberals. In a 2012 attack on the “ghastly” term inclusive, he questioned why schools needed wheelchair ramps. He has written about the difficulty of finding reliable domestic servants and why he thinks embryos could be screened to weed out those with lower intelligence potential.

Then, as if that wasn’t enough to irritate the left, in 2011 he launched the West London Free School (WLFS), a “comprehensive grammar” teaching classics and insisting on tough competition between pupils. Its logo is a bust of Cicero and it has a Latin motto, sapere aude, “dare to know”. It was the first free school under Michael Gove’s reforms, which have been attacked for destabilising the education system and diverting resources to pushy middle-class parents who want to set up independent-style schools in the state sector. Young became the face of the policy.

WLFS has already gone through four headteachers and is only ranked as “good” by Ofsted. It has however achieved better than average results, albeit with an intake with higher than average attainment.

“One of Toby’s successes was to represent it as something idealistic,” said Melissa Benn, the educationalist and daughter of Tony Benn who has repeatedly clashed with Young over free schools. “I think it has turned out to be completely ordinary, but part of a bigger picture that has led us to a semi-crisis.”

Even for a professional troublemaker (in the 1980s he was sacked from a plum job at the Times for hacking the editor’s computer and circulating the senior executives’ salaries), the response to his appointment has shocked him. Young cares intensely about his own profile, to the extent he is said to have edited his own Wikipedia profile 282 times in the last decade, but this week he switched off the Google alert which lets him know when his name pops up online. He has told friends the trawling of his Twitter accounts for remarks he now admits are puerile and stupid feels unfair, because he is determined to develop a more serious side to his life.

That wish is rooted in his relationship with his father, Michael Young, a Labour peer who wrote the 1945 Labour manifesto, coined the term meritocracy, established the Consumers’ Association and the forerunner of the Open University. When he died in 2002, he was described by the Guardian as a “towering figure in postwar social policymaking”.

At Christmas, there were cards from people like Nelson Mandela. The founder of the SDP, Shirley Williams, and James Callaghan’s foreign secretary, Tony Crosland, passed through.

Young was in his father’s shadow and was drawn towards Thatcherism, celebrity culture and hedonism to escape, said his friends.

Having failed all but one of his O-levels at King Edward VI comprehensive school in Devon, he went to sixth form at William Ellis grammar school in Kentish Town, north London.

“In came this posh bloke,” recalled Langan. “He had negative charisma. Everyone immediately hated him. Toby could alienate a room before he got to the end of it. But I thought he was really interesting and fun.”

Young sneaked into Brasenose College at Oxford University, which was seeking more state school pupils. He just missed the grades he needed (he got B,B,C) but after a call from his father he was allowed in and flourished, securing a first in Politics, Philosophy and Economics, and winning a Fulbright scholarship to Harvard. He started but did not finish a PhD at Cambridge on the theory of democracy.

Back in London in 1990, he started the Modern Review with Cosmo Landesman and Julie Burchill. It was a magazine that aimed to take popular culture seriously.

Asked what drove Young, Burchill said it was “living up to his father”.

“Toby, quite understandably, compared this with his own life of coke-snorting, status-chasing and model-making and found himself wanting,” she said. “The Modern Review was a fun way to do something moral. The whole point was to go after hypocrisy and snobbishness.”

After the magazine’s collapse, he became a contributing editor at Vanity Fair in New York. So little of what he wrote was published in his two and a half years that he estimated he was effectively paid $100 per word.

Back in London a decade later, now a parent to four young children with his wife Caroline, he became interested in how Gove, the shadow education secretary, was talking about replicating Swedish-style free schools. Young wrote a column in the Observer setting out his plan to open a “comprehensive grammar ... traditional curriculum, competitive atmosphere, zero tolerance of disruptive behaviour, but with a non-selective intake”.

“He spent 20 years rebelling against his father’s core principles, but I believe he actually shares the values of his father,” said Langan. “The school was a way of him moving towards what his father was doing.”

Stefan Bojanowski, an economist living in Acton, was among about 40 local people, many parents, who squeezed into Young’s living room soon afterwards to plan the West London Free School.

“My impression of him was of a deep and considered thinker and someone who believed in the goal,” said Bojanowski. “Quite a few people needed convincing. Without question, without him the school would never have gone anywhere.”

This week’s row has been a collision of two competing sides to Young as his deletion of thousands of tweets showed. If his appointment to the Office for Students survives the public anger at his previous comments, his desire to be finally seen as a serious man may move a step closer to being achieved.