William Hague is to stand down as foreign secretary with immediate effect and will take on the lesser role of leader of the House of Commons as David Cameron embarked on the most far reaching reshuffle of the parliament that was dubbed a new "night of the long knives".

A far wider than expected cull of male ministers saw the former chancellor Kenneth Clarke leave the government alongside Damian Green, the policing minister, Dominic Grieve, the attorney general, and David Willetts, the universities and science minister.

Philip Hammond, the highly eurosceptic defence secretary, emerged as the leading candidate to replace Hague at the foreign office. Hammond, who said last year that he would vote to leave the EU if it remained in its present form, would assume the mantle of the cabinet's most senior eurosceptic as foreign secretary.

As shell-shocked former ministers walked the corridors of Westminster on Monday night – ahead of the promotion today of a series of women at all levels of the government – there were the first signs of a backlash as the scale of the cull of middle aged men became clear. "It's the night of the long knives and that went really well last time," one Tory said sarcastically, referring to Harold Macmillan's desperate attempt to shore up his government in 1962 when he sacked a third of his cabinet.

Tories on the right of the party were also alarmed by the sacking of the arch eurosceptic Owen Paterson as environment secretary. They are hoping that Liam Fox, the former defence secretary, will return. There were suggestions that he would be offered a middle ranking post.

In the prime minister's most far reaching reshuffle, Downing Street announced that Hague will take on the role of leader of the House of Commons before he stands down as MP for Richmond at the election to allow him to focus on his writing career.

A major reshaping of Whitehall was also signalled as it became clear that Sir Bob Kerslake, the head of the home civil service who had clashed with the cabinet office minister Francis Maude, will retire next spring. Kerslake turns 60 in February, the usual retirement date for senior civil servants.

The ministerial reshuffle was signalled when the former chancellor Kenneth Clarke, who has championed the European cause as a member of every Tory government since 1972, resigned from the cabinet with a warning to Cameron that he will remain in parliament to fight in favour of Britain's membership of the EU.

The prime minister praised the outgoing foreign secretary as "one of the leading lights of the Conservative party for a generation" as Hague announced his intention to stand down.

Hague's move to become leader of the Commons has echoes of the path taken by Robin Cook after the 2001 election. But that was a clear demotion, unlike Hague whose decision to stand down at the election paved the way for a less onerous cabinet post.

The surprise announcement about Hague, who will remain as a member of the national security council and will play a leading role in reaching out to voters in the north of England in the run up to the election, followed a wider-than-expected cull of middle aged male ministers. As expected Sir George Young, the chief whip, and David Jones, the Wales secretary, were sacked from the cabinet.

There was surprise in Tory ranks as Stephen Hammond, the transport minister, Hugh Robertson, the foreign office minister, and Nick Hurd – son of the former foreign secretary Lord Hurd of Westwell – who was minister for civil society, were all sacked. Alan Duncan, the international development minister, and Greg Barker, the climate change minister who was an early supporter of Cameron during the 2005 Tory leadership contest, left the government at their own request.

As expected, the universities and science minister David "two brains" Willetts lost his job as did the Northern Ireland Office Andrew Robathan, another early Cameron supporter. Willetts, who is to stand down as an MP at the election, was praised by the prime minister who told him: "I have been proud to have 'two brains' at the heart of my team, both in opposition and in government and you, in turn, can take enormous pride in the contribution you have made."

Tory sources have made clear that Cameron wants the "old lags" to move on to make way for women and younger men who will be promoted on the second day of the reshuffle on Tuesday. Esther McVey, the employment minister and former breakfast television presenter, Liz Truss, the childcare minister, Nicky Morgan, the women's minister, Amber Rudd, the whip, Anna Soubry, the defence minister, Priti Patel and Margot James, members of the No 10 policy board, are all expected to be promoted. This should take the prime minister close to his target of ensuring that a third of his ministers are women.

One senior Tory warned of tokenism as the prime minister prepares to promote large numbers of women. "This really is the worst form of tokenistic gesture politics," one senior figure said. "Merit is out of the window."

The departure of Hague overshadowed all the other departures at Westminster where it had been assumed that he would remain as foreign secretary until the election, although it had been widely thought he might stand down next year to return to writing history books. Hague led the Conservatives in opposition from 1997 until 2001, when he resigned after being heavily defeated by Tony Blair at the general election.

Paying a warm tribute to Hague, Cameron said: "William Hague has been one of the leading lights of the Conservative Party for a generation, leading the party and serving in two cabinets. Not only has he been a first class foreign secretary – he has also been a close confidant, a wise counsellor and a great friend. He will remain as first secretary of state and my de facto political deputy in the run up to the election – and it is great to know that he will be a core part of the team working to ensure an outright Conservative victory at the next election."

Hague, who will remain as a member of the national security council, said: "By the time of the general election next year, I will have served 26 years in the House of Commons and it will be 20 years since I first joined the cabinet. In government there is a balance to strike between experience on the one hand and the need for renewal on the other, and I informed the prime minister last summer that I would not be a candidate at the next general election.

"Accordingly I am stepping aside as foreign secretary, in order to focus all my efforts on supporting the government in parliament and gaining a Conservative victory in the general election – after four years in which we have transformed Britain's links with emerging economies, significantly expanded our diplomatic network and the promotion of British exports, restored the Foreign Office as a strong institution, and set a course to a reformed European Union and a referendum on our membership of it."

Clarke, the former chancellor, made clear that is prepared for a final great battle of his political career in the runup to the prime minister's planned EU referendum in 2017. In his resignation letter to the prime minister, typed on plain paper, Clarke said that the case for Britain's membership of the EU is stronger than ever.

Setting himself at odds with the prime minister, who says that Britain's current EU membership terms are unacceptable, Clarke wrote: "I intend to remain as an active backbencher in the House of Commons. My belief in Britain's membership of the European Union remains as firm as ever and I think the political and economic case is made even stronger in today's globalised economy and dangerously disturbed world. We must not diminish Britain's ability to influence events in the next few decades."

Clarke, who secured his first job as a whip in 1972 in Edward Heath's government, made a point of making a high-profile Downing Street visit to the prime minister, who is 26 years his junior, rather than seeing Cameron in the more discreet setting of his House of Commons office. Friends of Clarke, who recently celebrated his 74th birthday, say he was in a jovial mood on Monday as he joked with colleagues that he was looking forward to enjoying more time watching his beloved cricket. In his resignation letter to the prime minister, he wrote: "I have been doing red boxes at night for a high proportion of my adult life. There are plenty of other able people who could take on the work that I was doing in government and I think the time has come to return to being a veteran backbencher."

Some of Clarke's friends have said that the prime minister is risking Britain's membership with what they regard as pointless rows such as the recent one over the appointment of Jean-Claude Juncker as president of the European Commission. Clarke recently undermined the prime minister by questioning why everyone was making such a fuss over Juncker.