From 1885, when the Labouchere amendment created the new offence of gross indecency, until 1967, when the recommendations of the Wolfenden committee were enacted, all homosexual behaviour was illegal in Britain. It continued to attract intense social disapproval for long after that: as late as 1993, according to the annual British Social Attitudes survey, most people still considered it to be “always wrong”. In this cruel and illiberal atmosphere, politicians with gay inclinations had to keep them secret from the public at large, and ran enormous risks. They were obliged either to repress their sexual feelings and lead celibate lives, or else to lead double lives, indulging their tastes clandestinely while outwardly conducting conventional, heterosexual existences, often married with children.

Yet paradoxically, “closet queens” (to use an expression that came into vogue in the 1960s) often made effective politicians. They were past masters when it came to keeping secrets, and taking calculated risks; they were also actors on life’s stage, with a strong sense of showmanship, and a flair for intrigue and subterfuge. And there was probably a far higher proportion of homosexuals in politics than in most other walks of life, partly because, as actors and risk-takers, they were drawn to the profession, and also because 20th-century British politicians were often educated at all-male boarding schools, which fostered intense (often sexual) friendships among their pupils, and provided unintentional training in the art of breaking the rules and getting away with it.

I became aware of this aspect of British political life when I wrote the biography (published after his death last year) of Jeremy Thorpe, the charismatic Old Etonian who led the Liberal party from 1967 to 1976. Always gallant with “the ladies”, Thorpe married twice, and fathered a son; but he led a hair-raising promiscuous gay life behind the scenes. When it came to both sex and politics he was a master of intrigue, exhibited outstanding seductive qualities and got a kick out of his risk-taking. He was eventually obliged to resign as Liberal leader when a former male model revealed that they had conducted an affair many years earlier, when it was still illegal; and he was subsequently tried at the Old Bailey for having allegedly conspired to have the man in question murdered – though he was acquitted, the revelations of his sex life and his efforts down the years to cover it up left him a discredited figure.

Jeremy Thorpe in 1977. Photograph: PA/PA wire

***

Many years earlier, two other Old Etonian Liberal politicians had found their careers ruined when they were hounded by men who claimed to have proof of their homosexual activities – the 5th Earl of Rosebery, prime minister in the 1890s, and the 7th Earl Beauchamp, a government minister from 1905 to 1915, subsequently Liberal leader in the House of Lords. They were cousins, who partly owed their early success to handsome looks; both were offered high positions in their 20s and joined the cabinet in their 30s. Both married fabulously rich heiresses – Rosebery, Hannah de Rothschild; Beauchamp, the sister of the 2nd Duke of Westminster. Rosebery’s marriage produced four children, Beauchamp’s no fewer than seven – but both were drawn emotionally to their own sex. (Both seem to have been infatuated by their handsome younger sons, by whose early deaths they were devastated.)

Rosebery had a liking for personable private secretaries. One of these was Francis, Viscount Drumlanrig, heir to the 9th Marquess of Queensberry, by whose qualities Rosebery was so taken that he arranged for him to be made a junior member of the government at the age of 26. This incensed “the mad marquess”, who was convinced that Rosebery was having an affair with his eldest son, just as he believed that Oscar Wilde was conducting one with his youngest son, Lord Alfred “Bosie” Douglas. Queensberry’s accusations were at first regarded as the ravings of a madman, but received some credence when in the autumn of 1894 – Rosebery having meanwhile become prime minister – Drumlanrig shot himself. Soon afterwards Rosebery suffered a breakdown and went into a state of seclusion: he was said to be terrified that his name would be raised in the libel case that Wilde brought against Queensberry. In 1895, less than a month after Wilde had been sentenced to two years’ hard labour for gross indecency, Rosebery tendered the resignation of his government (though it still commanded a parliamentary majority and no election was due).

Lord Rosebery speaking at the Foreign Office, c1900. Photograph: Hulton Archive/Getty Images

Whereas Rosebery had a (possibly platonic) taste for secretaries, Beauchamp’s (by no means platonic) penchant was for footmen and grooms. He continued these escapades for decades without being exposed, and might have done so indefinitely but for a vendetta by the Duke of Westminster, who was resentful that Beauchamp had, through his marriage to Westminster’s sister, produced three sons, while Westminster’s sole male heir had died aged four. In 1930, Westminster’s agents obtained statements from former servants of what he termed his “bugger-in-law”, testifying to his peccadilloes; confronted with this evidence, Beauchamp resigned and fled into exile. His story provided the background for Evelyn Waugh’s novel Brideshead Revisited: Lord Marchmain, the distinguished peer obliged to live abroad, is based on Beauchamp, while his son, the beautiful but wayward Sebastian, is based on Beauchamp’s adored younger son Hughie, with whom Waugh had been intimate at Oxford.

***

As a young cavalry officer, the Old Harrovian Winston Churchill was accused of having “participated in acts of gross immorality of the Oscar Wilde type” with fellow cadets at Sandhurst; but he successfully sued his accuser for libel, and there is no evidence that, as an adult, he engaged in physical homosexual relationships. Yet he was far from being straightforwardly heterosexual. Although he worshipped his beautiful American mother, he showed a lifelong aversion to women. (In Churchill’s only novel Savrola, the obviously autobiographical hero has a purely chaste relationship with the heroine, obviously based on Churchill’s mother.) He seems to have had a low sex drive, and married rather cold-bloodedly, aged 33, for social and dynastic reasons, just after being appointed to Asquith’s cabinet. Though he came to depend on his “Clemmie” in many ways, she was often exasperated by his emotional unresponsiveness and treatment of her as a child-bearer and housekeeper, and more than once considered leaving him.

Winston Churchill c1900. Photograph: Everett/Rex Shutterstock

While he never showed much interest in women other than his wife, Churchill’s life was marked by a series of close platonic relationships with attractive young men. Prominent among these were Eddie Marsh, a civil servant who served devotedly for 25 years as his private secretary, whom Churchill described as “a friend I shall cherish and hold on to all my life”; Archie Sinclair, a cavalry officer, whom he chose as his second in command on the western front in 1916, as his personal assistant as war secretary and colonial secretary in 1919-22, and finally (after Sinclair had become leader of the Liberal party) as air minister in 1940; Bob Boothby, the youngest and handsomest MP, whom Churchill, as chancellor of the exchequer in the 1920s, appointed his PPS despite the bumptious Boothby having criticised his policies; and Brendan Bracken, a young man on the make whom Churchill (to the horror of his family) effectively adopted in the 1920s and who served as his right-hand man during the 1930s, finally becoming information minister during the second world war. (Of these four, Marsh, who developed crushes on young writers and actors, and Boothby, who had a taste for “rough trade” from the criminal underworld, were certainly predominantly homosexual, and Bracken, though he cloaked his private life in impenetrable secrecy, probably, too.) Like Rosebery and Beauchamp, Churchill seems to have had feelings towards his son, Randolph, that verged on the amorous (another factor that estranged him from his wife, who disliked the boy from birth) – though these feelings cooled as Randolph lost his looks owing to excessive drinking, and became a coarse womaniser. Arguably, the intensely narcissistic and exhibitionistic Churchill was romantically drawn to men rather than women, even if his relations with them stopped short of the physical.

***

The long parliament of 1935-45 contained a clandestine fraternity of homosexual and bisexual politicians who knew each other well, were aware of each other’s tastes and often met socially. Boothby featured prominently in this circle, as did his friend the leftwing MP Tom Driberg, who was addicted to having sex with strangers in lavatories (including those of the House of Commons). Another friend of Boothby was the National Labour MP and wartime minister Harold Nicolson; Nicolson was an active homosexual married to a lesbian, as was the Conservative MP Viscount “Hinch” Hinchingbrooke. Nicolson had two sons by his wife, the writer Vita Sackville-West, while Hinch’s artist wife Rosemary bore him seven children before leaving him to live with a female companion. Also in the circle were two Conservative MPs and close friends, the American-born Henry “Chips” Channon and the Churchill protege Alan Lennox-Boyd, who married sisters, Guinness heiresses, but enjoyed riotous homosexual adventures behind the scenes. Lennox-Boyd had a distinguished ministerial career in several governments, while Channon achieved posthumous fame with his racy diaries (from which any hint of his gay sex life was expunged when they were published in the 1960s). Other MPs in the coterie included the outrageously camp millionaire aesthete Sir Philip Sassoon, who as undersecretary for air befriended young airmen and seems to have had a close relationship with at least one of them; the rich and handsome Sir Paul Latham, who in 1941 became the only serving MP of modern times to go to prison for homosexuality, after attempting to seduce his batman; and Malcolm Bullock, a balletomane and former guards officer on close terms with the royal family. All these men undoubtedly enjoyed the sensation of belonging to a secret society, as well as the thrill of danger that their illegal sex-lives involved (though this should not blind us to the fears, disappointments and lack of self-fulfilment suffered by most gay men, including politicians, until recent times).

Bob Boothby, photographed for the Picture Post in 1949. Photograph: Haywood Magee/Getty Images

Boothby is an interesting case, for while continuing relationships with young men he also pursued a long affair, which was common knowledge at Westminster, with Lady Dorothy Macmillan, wife of a fellow Conservative MP, the future prime minister Harold Macmillan; it was she who seduced Boothby and made the running throughout the affair. (When asked what he saw in her, Boothby replied that she reminded him of a caddie he had once met on the links at St Andrews.) The affair, and the impression it gave that Boothby was “a ladies’ man”, provided a smokescreen for his homosexual activities, and also proved helpful when, in 1953, he pressed the home secretary, Sir David Maxwell Fyfe, to set up a body to reconsider the law relating to homosexuality. Maxwell Fyfe was a vicious homophobe, then organising the unprecedented national persecution of homosexuals; and it is curious that he went out of his way to get the cabinet (which was far from keen on the idea) to appoint a committee to examine the matter, chaired by the former public school headmaster John Wolfenden.

When, in 1957, Wolfenden reported in favour of decriminalising homosexual acts performed between adult males in private, Maxwell Fyfe, now Viscount Kilmuir and lord chancellor, declared that he was “not going down in history as the man who made sodomy legal”. The Conservative government, now led by Macmillan, shared his view, refusing to lay the Wolfenden proposals before parliament; and a long campaign began to get them enacted – which they finally were in 1967, thanks to several courageous parliamentarians who sponsored the necessary legislation, notably the charismatic bisexual Welsh Labour MP Leo Abse. The Labour government that then held office was no keener to be associated with the reform than its Conservative predecessor – though time was allowed for a private member’s bill thanks to the efforts of two ministers who, though they had become thoroughly heterosexual, had enjoyed homosexual adventures as Oxford undergraduates: the home secretary, Roy Jenkins, who had been in love with his future cabinet colleague Tony Crosland; and the leader of the House of Commons, Richard Crossman, who had had an affair with the poet WH Auden.

***

It is ironic that the Macmillan cabinet that resisted implementing Wolfenden seems to have contained more “closet queens” than any other of the century. During the years 1959-60, for example, it contained the following homosexual or bisexual members: the chancellor of the exchequer, Derick Heathcoat-Amory, who narrowly avoided scandal after engaging in frolics with “teddy boys” at Margate; the foreign secretary, Selwyn Lloyd, who was in love with his handsome godson and personal assistant Jonathan Aitken; and the colonial secretary, Alan Lennox-Boyd, who though married with three sons conducted a longstanding relationship with a major in the Coldstream Guards.

The cabinet also included the health secretary, Enoch Powell, who as a classics professor in Australia before the war had written to his parents about his infatuation with his male students; and the minister of labour, the unmarried and misogynistic Edward Heath.

Three junior members of the government were obliged to resign owing to homosexual scandal: the Foreign Office minister Ian Harvey, who in 1958 was caught with a guardsman in St James’s Park; the Home Office minister Charles Fletcher-Cooke, who in 1961 was found to be living with a former borstal boy; and the science minister Denzil Freeth, whom Lord Denning, during his inquiry into the Profumo affair in 1963, discovered to have “attended a party of a homosexual character and there engaged in homosexual conduct”. It was even rumoured that Macmillan himself had been expelled for homosexuality from Eton (where his elder brother Daniel certainly had a “reputation”; his lovers including the future economist John Maynard Keynes) – though his latest biographer believes it more likely that he was “removed” from the school by his parents when they discovered that he was being “used” by older boys.

Edward Heath after his election victory in 1970. Photograph: Peter Kemp/AP

Heath, who became Conservative leader and prime minister, seems to have repressed his sexual nature totally in order to realise his political ambitions. He came from modest social origins, which he never tried to conceal: he was not close to his father, a former carpenter who ran a small building firm, but he worshipped his mother, who had a certain refinement, having worked as a lady’s maid in a country house. She adored her “Teddie”, and pressed him to achieve great things. Inspired by her, Heath was the star of his grammar school and won a scholarship to Balliol College, Oxford, where he became president of both the Union and the Conservative Association. Unlike his Oxford contemporary and future rival, Harold Wilson, Heath was not brilliant, but an intelligent and dedicated plodder. Following distinguished war service, in which he rose to be a colonel in the artillery, he was elected to parliament in 1950 and spent the ensuing decade as a whip, a role to which his personality – upright, thorough, affable but aloof – was ideally suited. That as government chief whip he managed to keep the party together after the Suez debacle of 1956 was a considerable achievement.

The Conservatives elected Heath as their leader in 1965 believing he would help them shed their elitist image, but he was an odd choice, awkward in his behaviour and lacking “the common touch”. It caused surprise when he defeated the charismatic Wilson to become prime minister in 1970. In 1975, after his administration been brought down by union militancy, he was replaced as leader by Margaret Thatcher, whom he regarded with an intense loathing that he never tried to conceal. (“When I look at him and he looks at me,” she once remarked, “it doesn’t feel like a man looking at a woman, more like a woman looking at another woman.”) Heath (like his hero, Churchill) was notoriously averse to women, and barely managed to be polite even to the wives of colleagues, but he does not seem to have been close to any men either, and was a self-absorbed, friendless figure.

Apart from Arthur Balfour, Conservative prime minister from 1902 to 1905 (who, though a narcissistic dandy, was in fact heterosexual and enjoyed sado-masochistic relationships with sophisticated women), Heath is modern Britain’s only bachelor premier, and his unmarried state gave rise to widespread rumour to the effect that he was actively gay – though no reliable evidence has emerged to suggest that he ever had a sexual or even a platonic relationship with anyone. Only in old age did he unbend to the extent of developing flirtatious friendships with some younger gay Tory MPs such as Alan Duncan and Matthew Parris, as well as with the gay couple who decorated his house in Salisbury. During his later years he also supported the lowering of the age of consent for homosexuality from 21 to 18 and then 16, whereas during his active political career he had studiously avoided showing interest in homosexual law reform.

***

Norman St John Stevas in 1950. Photograph: Personalities/AP / TopFoto

Clandestine homosexuality also flourished in the Labour party; here two contradictory influences were at work. A strain in the party’s development (epitomised by the Fabian Society, and Edward Carpenter’s philosophy of “the dear love of comrades”) preached sexual toleration and the rejection of bourgeois morals; but the party was also deeply rooted in the fierce evangelical Christianity that spearheaded the crackdown on homosexuality during the Victorian era. The Old Etonian Hugh Dalton, a leading figure in the Labour movement and a senior cabinet minister throughout the 1940s, was in love with Rupert Brooke as a Cambridge undergraduate, and later became intimate with several younger Labour politicians whose careers he helped, notably the bisexual Hugh Gaitskell and Tony Crosland. There were also some Labour-supporting peers who, while continuing to “live like lords”, shocked their families by flaunting both their socialism and their homosexuality, such as Oliver Baldwin, son of the Conservative statesman Stanley Baldwin and successor to his earldom, who, appointed a colonial governor by Attlee, barely concealed his “married” relationship with another man; and Gavin Henderson, 2nd Baron Faringdon, who had a penchant for firemen, and turned Buscot, his country house in Oxfordshire, into a venue for party meetings. These men were relaxed about their sexuality; but it was quite otherwise with a man of working-class origins such as the Welsh Labour politician George Thomas, speaker of the House of Commons from 1976 to 1983 (in which role he succeeded another “closet queen”, Selwyn Lloyd), who lived in terror of the exposure of his guilty secret. His homosexuality was revealed after his death by his friend and fellow MP Leo Abse, who had on several occasions helped extricate him from blackmail situations involving young men.

***

At the end of the 20th century, two of the most colourful figures in British public life, whose ambiguous sexual natures had recently been made public, were the Conservative Michael Portillo and Labour’s Peter Mandelson (“Polly” and “Mandy”). Despite their political differences, there were some remarkable similarities in their backgrounds. Both were born in 1953, the youngest in families of boys; both had strong-willed mothers, who had married husbands with foreign ancestries of whom their families disapproved. (Portillo’s mother, daughter of a Scottish linen tycoon, married an eminent Spanish refugee professor; Mandelson’s mother, daughter of the Labour statesman Herbert Morrison, married the descendant of Russian revolutionaries who had sought sanctuary in Australia.) Both had suburban London upbringings, became prefects at their excellent local grammar schools, and took part in amateur theatricals. Portillo went to Peterhouse, Cambridge, where he fell under the influence of the rightwing historian Maurice Cowling, while Mandelson went to St Catherine’s College, Oxford, becoming a protege of the leftwing historian Alan Bullock. Portillo, recommended by Cowling, began his career in the Conservative Research Department, while Mandelson, recommended by Bullock, started as a researcher at the TUC. Both subsequently had a spell working in television – in the early 1980s, Portillo was a researcher on Channel 4’s The Week in Politics, Mandelson a producer on LWT’s Weekend World.

Peter Mandelson announcing one of his resignations, in 2001. Photograph: David Sillitoe

Their careers then diverged in two respects. Portillo, whose party was in power throughout the years 1979-97, became a cabinet minister in his late 30s, just after Mandelson had entered parliament as a backbench MP. And whereas Portillo was famously the most swaggery politician of his generation (especially after John Major had appointed him defence secretary in 1995), Mandelson was the ultimate scheming “backroom boy”, whose achievement was to engineer Tony Blair’s succession as Labour leader and steer the “New Labour” brand to its great electoral triumph in 1997. (Mandelson worshipped Blair much as Portillo had worshipped Thatcher.) Both performed great services for their parties while becoming widely unpopular: Portillo’s loss of his supposedly safe London seat was the great shock of the 1997 election, while Mandelson, regarded by many as “a prince of darkness”, was twice made to resign from the cabinet during Blair’s first term of office owing to minor transgressions. Both made comebacks: Portillo (having experienced a Damascene conversion and become a “compassionate Conservative”) returned to parliament in 1999 and was appointed shadow chancellor by William Hague, whom he was widely expected to succeed as Conservative leader; while Mandelson, having become an EU commissioner in 2004, was recalled to the cabinet with a peerage by Gordon Brown in 2008, showing brilliance as a strategist during Labour’s last two years of office. (Though effectively retired from frontline politics, both made their voices heard during and after the recent election campaign.)

Both Portillo and Mandelson enjoyed the company of women. In his late 20s, Portillo married the girlfriend of his teenage years, Carolyn Eadie; Mandelson shared a flat with Neil Kinnock’s press secretary, Julie Hall. However, it was widely known that Portillo, though enjoying relationships with women, had homosexual interests, while Mandelson was frankly gay. Portillo was attached during his 20s to two institutions in which homosexuality flourished, Peterhouse and the Conservative Research Department, and he did not hold himself entirely aloof from the opportunities offered. Mandelson had two longstanding close relationships, first with a fellow researcher at the TUC, later with the Brazilian Reinaldo da Silva, who was some years younger than himself. Within the space of a year, both their private lives were made public in a spectacular fashion. In 1998, Mandelson, not yet in the cabinet but wielding great (and, in the view of some, sinister) influence behind the scenes as the eminence grise of the Blair government, was inadvertently “outed” on the BBC’s Newsnight by the journalist and former MP Matthew Parris. And in 1999, Portillo, about to stand for re-election to Parliament, admitted in an interview with the Times that he had had “some homosexual experiences as a young person” – whereupon a businessman named Nigel Hart declared in the Mail on Sunday that they had had an affair lasting eight years. Although these revelations do not seem to have harmed Portillo’s standing with the public, concern among fellow Conservative MPs about possible further embarrassing disclosures undoubtedly contributed to his narrow failure to win the party leadership when Hague stood down after the Tory defeat in the 2001 general election.

Michael Portillo listens to Stephen Twigg’s victory speech in Enfield Southgate in the 1997 general election. Photograph: Kevin Lamarque/Reuters

***

The climate for gay politicians showed some dramatic swings during the last quarter of the 20th century. Despite her Methodist upbringing and strait-laced image, Margaret Thatcher, who became prime minister in 1979, was quite tolerant of homosexuality: she was one of the few Tories consistently to support Wolfenden, chose many gay men to serve her, and appointed the flamboyant Norman St John Stevas as one of her leading ministers (though she became irritatated by his love of gossip and sacked him after less than two years). However, the espousal of gay rights by some elements of the left, and the Aids epidemic, resulted in much public disquiet about homosexuality during the 1980s. In the Bermondsey byelection of 1983, the openly gay Peter Tatchell managed to lose the supposedly safe Labour seat to the Liberal Simon Hughes (who later admitted to having gay relationships). In this atmosphere, no MP felt able to “come out” – with the brave exception of Chris Smith, Labour MP for Islington South and Finsbury. In 1988, Section 28 of the Local Government Act provided that a local authority “shall not intentionally promote homosexuality or publish material with the intention of promoting homosexuality”. Though heterosexual, John Major, who succeeded Thatcher in 1990, was favourable to the cause of gay equality. He lifted the ban on homosexuals in “high security positions”, and in 1994 the age of consent was reduced from 21 to 18. However, the great terror for gay MPs in the 1990s was of being “outed” by the press, something which would scarcely have been imaginable in previous decades. The decision of the Murdoch press to launch a “morality crusade” against the Major administration led to the resignation of several gay Tory MPs from minor government positions, and reinforced the impression that the regime was mired in “sleaze”. Meanwhile the militant gay organisation Outrage! targeted MPs it believed to be gay who failed to support gay equality, one of whom, the Ulster Unionist Sir James Kilfedder, died of a heart attack after learning of his “outing”.

New Labour, elected in 1997, promised a new era of “diversity” and “inclusiveness”. Labour candidates who triumphed in former Tory strongholds included two openly gay men, Stephen Twigg in Enfield Southgate (where he unseated Michael Portillo) and Ben Bradshaw in Exeter (where he won a large majority in the face of a homophobic Conservative campaign). Chris Smith became culture secretary and the first openly gay cabinet minister, his male partner being included in such official occasions as Buckingham Palace garden parties. Great strides were made in reforming the law. The ban on homosexuals in the armed forces was lifted; the age of consent was reduced to 16; Section 28 was repealed; civil partnerships were introduced, giving gay couples the same legal privileges as married couples; and a series of measures made it illegal to discriminate against people, or “incite hatred” against them, on the grounds of their sexual orientation.

Chris Smith at the Tate Modern in 2000. Photograph: Martin Argles for the Guardian

Yet in 1998 the Blair government witnessed one of the most bizarre gay scandals of the century, when Ron Davies, secretary of state for Wales, resigned after what he termed “a moment of madness” on Clapham Common, which had led to his being robbed at knifepoint and relieved of his wallet, phone, car and Commons pass. And the Conservative-Liberal Democrat coalition that took office in 2010, which was equally committed to the cause of gay equality and legislated for “gay marriage”, similarly suffered the loss of a cabinet minister when the Lib Dem David Laws resigned as chief secretary to the Treasury after the Daily Telegraph revealed that he had claimed parliamentary expenses for London accommodation belonging to his “long-term partner and secret lover”, a situation disallowed by the expenses rules; he explained that he had wished to keep his sexuality secret out of respect for the feelings of his mother. It seems the age of the political closet queen may not be quite dead yet.

***

Is a study of this subject historically or politically relevant? I believe it is, because until recently many politicians (possibly more than one might imagine, for the men in question usually did their utmost to ensure that no details of this aspect of their lives emerged) were obliged to keep secret something important about themselves. And this inevitably affected the way they operated as politicians: apart from inculcating quick wits and sharp antennae, it tended to foster skills in several areas – tact and discretion; acting and showmanship; the “dark arts” of presentation and manipulation; awareness and management of risk. Following the election, Westminster now boasts 32 openly gay MPs and while one must be thankful that, in Britain, official homophobia is a thing of the past, and that gay men and women, in politics as elsewhere, can now be honest about their sexuality, the necessity for homosexuals in public life to hide their nature, though in itself deplorable, did not, perhaps, exercise an entirely negative influence on the 20th-century political scene.

• Michael Bloch’s Closet Queens is published by Little, Brown this month.