J Edgar Hoover was a phenomenon. The first Director of the FBI, he remained in office for 48 years, from his appointment after the First World War to his death in 1972, achieving fame and extraordinary power. For public consumption when he died, President Richard Nixon eulogised him as: "One of the giants… a national symbol of courage, patriotism and granite-like honesty and integrity." He ordered flags to fly at half-mast and that Hoover's body lie in state in the Capitol.

In private, on hearing that he had died, Nixon had responded merely: "Jesus Christ! That old cocksucker!" Months earlier, closeted with key advisers, he had held forth on the need to persuade the elderly Hoover to resign. "We have on our hands here a man who will pull down the temple with him, including me."

Nixon, soon to be disgraced and forced to resign, was of course himself no paragon. Most presidents before him, though, had had cause to fear Hoover or been troubled by what his FBI had become. Harry S Truman wrote during his presidency: "We want no Gestapo or secret police. FBI is tending in that direction. They are dabbling in sex-life scandals and plain blackmail… Edgar Hoover would give his right eye to take over, and all congressmen and senators are afraid of him."

Hoover himself, meanwhile, had a personal secret that – in his era – could have destroyed him if revealed. Clint Eastwood referred to it this year before the launch of his movie, when he assured the J Edgar Hoover Foundation that J Edgar would not "portray an open homosexual relationship" between Hoover and his long-time male companion, Clyde Tolson.

Eastwood stretched the truth. Though there is just one passionate kiss between Leonardo DiCaprio and Armie Hammer, the two actors portraying them in the movie, the relationship with Tolson is a central theme. In real life, all Washington knew was that the pair dined daily together, vacationed together, did everything but move in together, and the whispers flew. When a magazine article in the 1930s referred to Hoover's "mincing" gait, and a diplomat commented on his "conspicuous perfume", Hoover struck back. He gathered derogatory information on the offending journalist, and asserted – falsely – that he did not use perfume. Real information on the Hoover-Tolson relationship surfaced only long after both men were dead, during research for my book.

A surprising find was the account by Luisa Stuart, once a celebrated model, tracked down because she featured in a droll photograph taken with Hoover and Tolson one New Year's Eve in the late 1930s at the Stork Club – the place to be seen in New York at the time. In the photo, Hoover is shown holding his hands up as Stuart, armed with a toy shotgun, "threatens" him. Later that night, in the dark of a limousine when they left the club, she remembered: "I noticed they were holding hands all the way, just sitting there talking and holding hands with each other… I was so young and those were different times. But I'd never seen two men holding hands."

Joseph Shimon, a former Washington police inspector, recalled a taxi driver reporting the pair had been "kissing and ass-grabbing" during a cab journey. Harry Hay, founder of America's first gay rights group, remembered that on vacation in California, in "a circle in which they didn't have people who weren't gay… They were nodded together as lovers."

The Eastwood movie includes a bizarre scene that depicts Hoover, after his mother's death, donning one of her dresses. It is a nod towards allegations I first reported, that he on occasion cross-dressed. I had information from three sources, two men who said an "easily recognisable" photograph of Hoover in an evening gown circulated in the gay community in 1948, and an account by a millionaire's former wife of secret sex parties that she claimed to have witnessed in the late 50s. Hoover, the woman said, had been "dressed like an old flapper, like you see on old tintypes".

Bill Clinton, who as president in 1993 was mulling over who to appoint as FBI Director, thought the cross-dressing reports were hilarious. "It's going to be hard," he grinned during a speech at a press function, "to fill J Edgar Hoover's… pumps." That I published such allegations at all, however, to this day draws roars of fury from old Hoover loyalists.

Other accounts of the Director's alleged sexual activity, if true, would certainly have destroyed him had they become public. A former Bureau inspector and trusted associate named Jimmy Corcoran said years later that Hoover, youthful at the time, had once asked him to deal with a serious "problem". He had been arrested on sex charges involving a young man during a trip to New Orleans. Corcoran, who had powerful contacts in the state, said he intervened to hush the matter up.

There is, too, a claim that as late as 1969, when Hoover was in his early 70s, he dallied with teenage boys during his habitual summer break in California. An element of corroboration came from Don Smith, an officer on the Los Angeles police vice squad, who told me of interviews he conducted with youngsters during a paedophile investigation. "The kids," Smith said, "brought up several famous names, including those of Hoover and his sidekick".

For me, the most significant, credible information on Hoover's sexuality came with the discovery that Hoover for a while consulted Marshall de G Ruffin, a Washington psychiatrist who became president of the Washington Psychiatric Society. De Ruffin's widow Monteen recalled learning from her husband that his distinguished patient was "definitely troubled by homosexuality". After several sessions, however, "Hoover got very paranoid about anyone finding out he was a homosexual, and got scared." As if to compensate, Hoover lashed out at and sought to expose other homosexuals. For years he had his agents infiltrate and monitor homosexual-rights groups, while he sounded off publicly about "sex deviates in government service".

My conclusion after five years' research was that while Hoover may have spent much of his life repressing his private urges while building an image of himself as the acme of sexual purity, he did sometimes lapse – risking catastrophe every time. Having studied the information I assembled, two noted specialists in psychiatry and psychology said they believed Hoover's sexual torment was very pertinent to his use and abuse of power as America's top law-enforcement officer.

Dr John Money, professor of medical psychology at Johns Hopkins University, thought Hoover "needed constantly to destroy other people in order to maintain himself. He managed to live with his conflict by making others pay the price." Dr Harold Lief, professor emeritus of psychiatry at the University of Pennsylvania, concluded that Hoover suffered from "a personality disorder, a narcissistic disorder with mixed obsessive features… paranoid elements, undue suspiciousness and some sadism. A combination of narcissism and paranoia produces what is known as an authoritarian personality. Hoover would have made a perfect high-level Nazi."

The eight decades of Hoover's life tell their own story. As early as his teen years, his mind was closing on issues that were to dominate his era. In the school debating society, he argued against women getting the vote and against abolition of the death penalty. He could never bear to come second in anything. When his father began to suffer from mental illness, a niece told me, Hoover "couldn't tolerate the fact. He never could tolerate anything that was imperfect." Another relative said: "I sometimes have thought that he really had a fear of becoming too personally involved with people." William Sullivan, a close FBI associate, thought his boss "didn't have affection for one single solitary human being".

Hoover joined the Bureau – at that time just the Bureau of Investigation (the word "Federal" was only added in the 1930s) – as America's first great Communist scare was getting under way, and handpicked as his assistant a man named George Ruch. One of two key associates to name their own sons J Edgar, Ruch expressed astonishment that left-wingers should even "be allowed to speak and write as they like". Hoover and Ruch favoured deporting people merely for being members of radical organisations, and used the Bureau to spy on lawyers representing those arrested in the infamous Red Raids of 1920. One of them, on whom he was to keep tabs for half a century and deem "the most dangerous man in the United States", was future Supreme Court justice Felix Frankfurter.

Hoover never joined a political party and claimed he was "not political". In fact, he admitted privately, he was a staunch, lifelong supporter of the Republican party. He secretly aspired to be president and considered running against Franklin D Roosevelt, whom he thought suspiciously left-wing. Hoover publicly expressed support for Senator Joe McCarthy shortly before McCarthy claimed Truman's State Department was harbouring 200 members of the Communist party. His agents slipped file material to the senator for use in his infamous inquisition, while publicly denying doing so.

The favourable publicity Hoover enjoyed was partially deserved. He cleaned up a Bureau that had been notorious for corruption and inefficiency, replacing it with an agent corps that became a byword for integrity. One veteran defined the ideal new recruit as a man who had to represent "the great middle class", who "will always eat well and dress well, but will never get that sleek Packard or sumptuous house. He belongs to the Bureau body and soul".

Hoover brought modernity and co-ordination at a time of disorganisation. He built the first federal fingerprint bank, and his Identification Division would eventually offer instant access to the prints of 159 million people. His Crime Laboratory became the most advanced in the world. He created the FBI National Academy, a sort of West Point for the future elite of law enforcement.

While all this was positive, Hoover's Division 8, euphemistically entitled Crime Records and Communications, had a priority mission. Crime Records pumped out propaganda that fostered not only the image of the FBI as an organisation that spoke for what was right and just, but of the Director himself as a champion of justice fighting "moral deterioration" and "anarchist elements". Hoover used the department to preach the notion that the political left was responsible for all manner of perceived evils, from changing sexual standards to delinquency.

Crime Records portrayed Hoover as the dauntless scourge of serious crime. In the movie J Edgar, long sequences are devoted to his supposed role in tracking down the murderer of the aviator Charles Lindbergh's baby son. In real life, while Hoover postured as the Sherlock who led the probe, the case was in fact broken thanks to work done by another federal agency. Similar phoney self-promotion featured in the fight against the bandits of the 30s, Bonnie and Clyde, Machine Gun Kelly, John Dillinger and Alvin Karpis. Hoover hogged the limelight when the thugs were killed or captured and was jealous and vindictive when it fell instead on one of his proteges.

Late in the Eastwood movie, his companion, Clyde Tolson, peruses a memoir Hoover has just completed about his life and career. Then, reproachfully, he remarks that the account is a pack of lies. There was no real-life memoir, but the line is perceptive. Issues of fact versus fabrication and distortion, truth versus outright lie or self-delusion, dominate Hoover's story.

Hoover's public position on race, Southerner that he was, was that of the paternalistic white nativist. Less openly, he was racially prejudiced. He shrugged off the miseries of black Americans, preferring to claim they were outside his jurisdiction. "I'm not going to send the FBI in," a Justice Department official recalled him saying testily, "every time some nigger woman says she's been raped." FBI agents paid more attention to investigating black militants than pursuing the Ku Klux Klan.

In the 60s, Hoover went to extreme lengths to establish that Martin Luther King and his movement were under Communist control. When surveillance established only that King was having sex with women other than his wife, FBI aides worked to "neutralise" him by slipping prurient information to the press. When the civil rights leader was awarded the Nobel Peace Prize, Hoover was enraged. When thousands mourned King's assassination, Hoover went to the races. He later tried to prevent King's birthday being declared a national holiday.

All this took place against a personal background of which few are today aware – a rumour that Hoover himself had black ancestry. Early photographs do show him looking somewhat negroid, with noticeably wiry hair. Gossip along those lines was rife in Washington and – true or not – Hoover must have been aware of it. Did anxiety on that front shape the way he behaved towards blacks – just as he lashed out at homosexuals while struggling with his own homosexuality?

Research into the sex angle, meanwhile, may explain why – at the very time in US history that organised crime was on the rise and could have been effectively countered – Hoover failed to act. The man who had found fame for hunting down the bank robbers and bandits of the 30s let the Mafia flourish.

It seemed at first, before the Second World War, that Hoover would clamp down on the mob. Then, abruptly, he turned off the pressure. In the 50s, he actively obstructed the Kefauver Committee, which concluded there was indeed "a nationwide crime syndicate known as the Mafia". Not so, said Hoover. When a 1958 report by his own agents also said the Mafia was real, he dismissed it as "baloney". The FBI would take vigorous action only very belatedly, in the 1960s, under pressure from Attorney General Robert Kennedy.

Former officials I interviewed, including three former attorney generals and several former assistant directors of the FBI, were at a loss to explain why Hoover refused to tackle the threat of organised crime. "Hoover's attitude," said Neil Welch, a senior former agent who eventually distinguished himself fighting the Mafia, "was so contrary to reality as to be a reason for great speculation."

Hoover himself, it is now clear, had contacts with organised criminals or their associates in circumstances that made it possible – likely even – that they learned of his sexual proclivities. More than one top mobster claimed the outfit had a hold on Hoover. Meyer Lansky, the syndicate's co-founder, was said to have "pictures of Hoover in some kind of gay situation" and an associate quoted Lansky as claiming, "I fixed that sonofabitch." Carmine Lombardozzi, who was known as "the Italian Meyer Lansky", said: "J Edgar Hoover was in our pocket."

Blackmail was the tactic that worked for Hoover, too, in his dealings with politicians. The title of my biography of him, Official and Confidential, derives from the name of a file group that was held in locked cabinets in Hoover's office. By an official count after his death, the Director held 883 files on senators and 722 on congressmen. Many documents were shredded after Hoover's death, but those that survive speak for themselves. An example is this 1959 report:

Dear Mr Hoover,

You may be interested in the following information… (NAME WITHHELD) [said] she had spent the afternoon of 3 June 1959, with Senator (NAME WITHHELD) in his private office. She also said she had sexual intercourse with the senator during the afternoon "on the couch in the senator's office…"

Sincerely yours,

James H Gale, Special Agent in Charge

Such reports, I learned, were used to bend politicians to Hoover's will. He might need their co-operation to procure funds, to gain political muscle, or to avert investigation of operations he preferred kept hidden. An aide to Senator Edward Long, the Democrat from Missouri, was to swear an affidavit describing what occurred when Long was planning hearings on the FBI – with a special focus on electronic eavesdropping. A senior Hoover aide came to call, and the conversation went as follows: "Senator, I think you ought to read this file that we have on you. You know we would never use it, because you're a friend of ours… We just thought you ought to know the type of stuff that might get around and might be harmful to you… They handed him the folder… Long read it for a few minutes. [Then] they went on their way. The next thing I knew we had orders to skip over the FBI inquiries."

Hoover snooped not just on politicians but on officials high and low, on Supreme Court justices – at least 12 of them – even on presidents. He built files on writers, actors, on citizens across the spectrum who caught his malignant eye. Many feared what the Director might have found – whether he had compromising information on them or not.

In life, Hoover denied time and again that there were such "secret dossiers". Acting Attorney General Laurence Silberman, the first person to peruse the secret files after Hoover's death in 1972, learned otherwise. "J Edgar Hoover," he told me, "was like a sewer that collected dirt. I now believe he was the worst public servant in our history."

The Director more than got away with his excesses. He was showered with honours. Even today, in spite of the ugly truths that have surfaced since his death – an official probe found that on top of everything he had also been personally corrupt – the sign on the façade of FBI headquarters in Washington proclaims, in gold lettering, that it is the "J EDGAR HOOVER BUILDING".

"American society", mused Dr Lief, the psychiatrist who thinks the facts indicate Hoover would have made a perfect high-level Nazi, "has a strangely polarised attitude towards its heroes. On the one hand people love to find the idol has clay feet, to find the flaw in the famous man. On the other, they are reluctant to take the hero off his pedestal. This is a curious contradiction in our society, and sometimes a dangerous one."

Anthony Summers is the author of eight non-fiction books; the most recent is The Eleventh Day, on 9/11. A new edition of Official and Confidential: The Secret Life of J Edgar Hoover, is published this month (Ebury, £8.99). The movie J Edgar is out on 20 January