(CNN) -- One American president's legitimacy was questioned because he was accused of wearing women's underwear.

Another's qualifications were questioned because he got drunk at an inaugural ball.

A third president didn't belong in office because critics said his rich daddy stole the election.

A recent CNN poll revealed that one out of four Americans doubt that President Obama is a citizen. Many are "birthers" who believe he is an illegitimate president because he wasn't born in this country.

But historians say Americans have long accused their presidents of being illegitimate officeholders for all sorts of dark, and bizarre, reasons.

"Heavens, where do I start?" says David Crockett, an associate professor of political science at Trinity University in Texas.

"No one wants to admit that they got spanked in an election contest. It's always nicer to think the opponent somehow cheated the system. So yes -- Obama wasn't born here, Bush stole the election, Clinton won only because Ross Perot screwed up Bush's [the elder's] chances, Bush the elder won only because he demagogued Willie Horton. ... I could go back further."

You can go back to the 19th century, where calling a president illegitimate was a common -- and often nasty -- practice, says Thomas Alan Schwartz, a presidential historian at Vanderbilt University in Tennessee.

President John Tyler was known as "His Accidency" because in 1841 he became the first president to assume office due to the death of his predecessor.

Rutherford B. Hayes was dubbed "His Fraudulency" because he assumed office after losing the popular vote in 1876 in a disputed election that he won by one electoral vote.

Though questioning a president's legitimacy is common, it can turn dangerous, Schwartz says.

"On the extreme side, it encourages nut cases to take the law in their own hand."

A presidential cross-dresser?

For the most part, though, calling a sitting American president illegitimate seems part of our political tradition.

Crockett, the Trinity University history professor, says Obama wasn't the first president whose legitimacy was questioned because of his cosmopolitan background.

Foes accused Thomas Jefferson, our third president, of not being a legitimate American because of his foreign ties as well. They said he was too French -- he spoke French and had a French cook.

They also said he had a slave mistress (a charge that a DNA analysis later suggested might be true), Crockett says.

Some other presidential "imposters" included Martin Van Buren, who was accused of not being fit for office because he allegedly wore a woman's corset, Crockett says.

Some claims of presidential illegitimacy rested on nothing but open dislike of the man in the Oval Office.

Theodore Roosevelt deemed his successor, William Howard Taft, illegitimate because he didn't like the man or his policies.

"Teddy Roosevelt called Taft a fathead," Crockett says.

Sometimes the erratic behavior of a president caused people to deem him illegitimate.

Andrew Johnson, Abraham Lincoln's successor, was deemed unqualified because of his personality and his policies, says Randall Miller, a history professor at St. Joseph's University in Pennsylvania.

Johnson enraged Lincoln's supporters when he pursued policies that undercut Reconstruction, the series of policies Lincoln helped create to integrate newly freed slaves into American society, Miller says.

Johnson, who was later impeached, also got drunk at Lincoln's second inaugural ball, Miller says.

"Johnson took everything personal," Miller says. "He would respond to hecklers. He became engaged in these back-and-forth shouting matches, and that contributed to the idea that he was illegitimate."

Questions of presidential legitimacy even once played a role in leading the country to war. When Lincoln was elected in 1860, Southerners did not consider him legitimate.

Why illegitimate president claims are so popular

Claims of presidential illegitimacy faded away in the mid-20th century because of several landslide presidential elections.

But they revived again with the election of another Oval Office pioneer, John F. Kennedy, in 1960. Kennedy, like Obama, was an Oval Office first -- he was the first Irish-Catholic president.

But critics said Kennedy's election was illegitimate because of his razor-thin victory over Richard Nixon. Some accused Kennedy's millionaire father of buying the election; others said Democratic operatives had stolen votes for Kennedy in Chicago, Illinois, a Democratic stronghold.

"Nixon didn't challenge the election of 1960, though he had a very good case," says Schwartz, the Vanderbilt University historian. Schwartz says Nixon didn't protest because he didn't want to spark a political crisis during the Cold War.

Kennedy's assassination fed into even more claims of presidential illegitimacy, another historian says.

There were some who thought Lyndon B. Johnson's presidency was illegitimate because they believed he was complicit in the death of Kennedy, who was killed in Johnson's home state.

"It goes back to the assassination of John Kennedy and people's feelings that they weren't getting the straight story about really happened," says Kevin Gutzman, a history professor at Western Connecticut State University.

"People were predisposed to think of conspiracy at the highest levels."

Subsequent political events didn't lessen America's Oval Office paranoia. Watergate. Iran-Contra. "Slick Willie." Bush v. Gore.

Americans began to lose trust in their presidents, and with that, their reverence for the office, says Mary Ellen Balchunis, a political science professor at La Salle University in Pennsylvania.

She says claims of presidential illegitimacy have risen as respect for the office has fallen.

Her proof: A prominent politician can talk about the president's testicles in public without much backlash.

"What is different today is the lack of respect or deference to the office of the president," Balchunis says. "How many times in the last few days have we heard on the news about Sarah Palin who made comments about President Obama's testicles, or the lack of them, in Spanish of course?"

How 'imposter' claims soothe the soul

The Internet and the 24-hours news cycle feed into claims of presidential illegitimacy, says Randall Miller, a history professor at St. Joseph's University in Philadelphia.

"The charges against a president are so constant and pervasive," he says. "You just pound away, and the capacity for people to step back and ask if this makes sense is almost gone."

Presenting people with facts won't quell the cries of presidential illegitimacy, research suggests.

A series of studies conducted at the University of Michigan in 2006 showed that political partisans, when exposed to facts that corrected their assumptions, rarely changed their minds. They actually held even tighter to their assumptions.

Proof of Obama's citizenship has been presented: Two newspapers in Honolulu published announcements of his birth in Hawaii in August 1961. The Republican governor of Hawaii recently certified Obama's birth certificate as legitimate.

But claims that Obama isn't a legitimate president still resonate with some Americans because of Obama's blackness and his "otherness," says Adam Hanft, a cultural critic who writes and blogs on social trends.

Obama's exotic background -- his migratory childhood, having Muslim relatives, coming seemingly out of nowhere to win the presidency -- is difficult to accept for some people who don't "have a high tolerance for diversity," Hanft says.

"People are hanging on to this notion that he's not one of us," Hanft says. "That adds another level of mystery. He is this guy from nowhere."

And who knows what this guy from nowhere is up to? Americans have a long tradition of believing that some conspiracy is afoot to rob them of their freedoms, says Miller, the historian at St. Joseph's University.

The enemies of freedom change -- the British, the bankers, the communists, and now, an Oval Office interloper -- but they keep coming, Miller says.

"It's in our political DNA to want to believe in conspiracy theories," he says. "That contributes to the mind-set that people in power are illegitimate."

Our love of Oval Office conspiracies and illegitimate presidents covers up a deeper fear -- of change, he says.

"As life becomes more complicated and you feel alienated, you look for things to make sense out of a world that's a swirl," Miller says. "You can comprehend these simpler explanations."