A single politician has seldom made as much individual impact on the leadership of a political party as did Michael Spicer, Lord Spicer, former chairman of the Conservative 1922 Committee, who has died aged 76. During four turbulent years that followed the 2001 general election, in his role as the party’s backbench “shop steward”, he personally supervised the election of three Conservative leaders, Iain Duncan Smith, Michael Howard and David Cameron, an unrivalled record.

In an extraordinary corollary, as a founder of the European Research Group, established in 1993 in the fevered early days of the party’s problems over Europe, he helped create a powerful Eurosceptic pressure group that arguably brought about the downfall of the Conservatives’ latest two prime ministers, Cameron and Theresa May.

This record is all the more remarkable for the fact that Spicer, during his 36 years in the House of Commons, only rose to be a junior minister in Margaret Thatcher’s government – apart from a few months as a minister of state just before her loss of office – and never became a well-known public figure. It was a source of, perhaps justified, resentment to him that this was so and that he never secured the political promotion he thought he deserved, but he more than compensated for this with the unseen influence he was subsequently able to exert throughout the party. That continued until his last years when, as chairman of the political and parliamentary honours committee from 2012, he was able to help reward those whose services might not otherwise have been recognised.

His puzzling failure to achieve high office was held by some friends to be an oversight by Thatcher, who had failed to recognise how much he shared her political ideals, but he was also said to lack the killer instinct required to reach political eminence.

He was a charming, amiable and clever man with an entirely human fallibility of liking to be liked. When his weighty book of memoirs, The Spicer Diaries, was published in 2012, some of the juicier anecdotes from his original drafts were either bowdlerised or missing from the final version. “I don’t want a friendless old age,” he told his publisher, Iain Dale.

Some of his colleagues regarded him as scheming, but in 2001 he nevertheless comfortably defeated the former cabinet minister Gillian Shephard to win the chair of the 1922 Committee, in what is effectively a parliamentary beauty contest.

Spicer was born in Bath into a military family, disparagingly referring to himself as “not a real toff at all”. His father, Leslie Spicer, became a brigadier, and his mother, Winifred (nee Carter), was a concert pianist. He was educated at Wellington college, of which he would later become a governor, and studied economics at Emmanuel College, Cambridge. He made his first political mark while still an undergraduate, by founding a pressure group with another Conservative student activist who would also later become a Tory MP, Keith Hampson, from Bristol University. They first chose the acronym Pest, as a statement of their declared intention, and subsequently devised its title as Pressure for Economic and Social Toryism.

After a short period as a financial journalist in Fleet Street, Spicer joined the Conservative research department in 1966, a route then acknowledged as a fast track for aspirant politicians. He fought his first general election that year, at 23 the youngest candidate in the country, standing against the veteran 82-year-old Emmanuel Shinwell, the oldest candidate, in the rock-solid Labour seat of Easington, Co Durham.

Two years later he became director of the Conservative Systems Research Centre, helping to introduce IT into central office. In 1970, having unsuccessfully contested Easington once again, he started one of the first economic consultancies in the UK, Economic Models Ltd; he was managing director until 1980. During this period he was credited with persuading the Conservative party of the importance of economic trends and statistics, and was regarded as a galvanising force on modernising strategic political thinking; he also successfully developed a personally lucrative international business career.

Following the death of Sir Gerald Nabarro, MP for South Worcestershire, Spicer was selected from 125 other candidates to contest a projected byelection in the constituency. Before it was scheduled, however, the unexpected February 1974 general election was announced by Edward Heath and swept Spicer into the Commons. He was picked for promotion when the Conservatives returned to office under Thatcher in 1979, being appointed parliamentary private secretary to Cecil Parkinson at the Department of Trade.

Thereafter he became a vice-chairman of the party at central office and was optimistically expecting appointment as party chairman in succession to Parkinson after the 1983 election. He was profoundly disappointed instead to be made deputy chairman to John Gummer, and given all the hard grind of running the party, with a noticeable lack of glamour. That was matched by his subsequent posts, as the junior minister at Transport (1984-87) and at the Department of Energy for the next three years. When he was appointed minister of state at Environment in 1990, he found himself at odds with Thatcher over housing policy in the short months before she was ousted. When John Major became prime minister he was immediately dropped.

By this time Spicer, who had once been regarded as on the party’s left wing and was described as a “neo-socialist” by a critic when first selected for parliament, had refined his views, particularly with reference to Britain’s membership of Europe. His opposition to the exchange rate mechanism precluded his membership of Major’s government and he became a serial rebel over the Maastricht treaty, voting against the government 37 times and abstaining 27 times as the controversial legislation was enacted.

It was, however, with the professed original intention of healing the party’s divisions that the European Research Group, which would later provide the focus of hardline Euroscepticism, was established. In his last major speech in the Lords, in June 2018, Spicer insisted it was his belief in the nation state as the best unit for democratic accountability that inspired his support for Brexit.

Spicer was knighted in 1996 and made a member of the privy council in 2013. He left the Commons in 2010, having moved constituency from South to West Worcestershire in 1997, and was immediately appointed to the House of Lords. His last Commons term brought him considerable personal embarrassment over the MPs’ expenses scandal, which in his case involved a £620 chandelier and £5,650 for gardening costs including hedge-clipping for his helipad.

He was an accomplished tennis player, a talented artist and the author of eight political thrillers.

He is survived by his wife, Patricia (nee Hunter), whom he married in 1967, and their three children, Edward, Antonia and Annabel.

• William Michael Hardy Spicer, Lord Spicer, politician, born 22 January 1943; died 29 May 2019