A man working on his 1940 tax returns while his son sits near him playing with toys. (Credit: William C. Shrout/The LIFE Picture Collection/Getty Images)

American colonists paid very little in taxes.

Great Britain found itself deep in debt following its victory over France in the French and Indian War. As a result, it enacted the Sugar Act of 1764, the Stamp Act of 1765, the Townshend Acts of 1767 and other measures aimed at raising funds from its 13 American colonies. Taxes remained orders of magnitude higher in Great Britain than across the pond. But the colonists fiercely resisted them anyway, protesting against “taxation without representation” and eventually taking up arms. Resistance to taxes continued after the Revolutionary War, most notably when a group of Pennsylvania farmers initiated the so-called Whiskey Rebellion in 1794.

The first federal income tax was imposed during the Civil War.

The tariff, a tax on imported goods, once served as the main source of revenue for the U.S. government. But when the Civil War broke out in 1861, President Abraham Lincoln began looking for ways to raise additional funds. Later that year, he signed a bill that imposed the first tax on income, which, without an enforcement mechanism, never raised much money. A second bill then passed in 1862 that taxed those making between $600 and $10,000 a year at a rate of 3 percent and those earning over $10,000 a year at a rate of 5 percent. Because the average annual income at the time was only around $300, most people did not have to pay anything.

Congress allowed the income tax to expire a decade later. It approved a new one in 1894, but the U.S. Supreme Court invalidated it the following year as unconstitutional. Finally, in 1909, Congress passed the 16th Amendment, which was designed to overturn the court’s 5-4 decision. After being ratified by more than three-fourths of the states, the amendment took effect in February 1913. President Woodrow Wilson signed another income tax bill a few months later, and the United States has had an income tax ever since.

Tax rates have varied wildly, particularly for the highest earners.

America’s richest citizens handed only 7 percent of their income to the federal government from 1913 to 1915. By 1918, however, that rate had skyrocketed to 77 percent due to U.S. involvement in World War I. It then steadily decreased to 25 percent during the Roaring Twenties before picking back up to 79 percent during the Great Depression.

Tax rates for the wealthy topped out at more than 90 percent from the mid 1940s to the early 1960s. They then dropped to 70 percent during Lyndon B. Johnson’s presidency before nose-diving to 28 percent at the end of Ronald Reagan’s presidency. Under George H.W. Bush they increased to 31 percent, under Bill Clinton they increased to 39.6 percent and under George W. Bush they dropped to 35 percent. Yet another change took place in 2013, mandating that the richest Americans—categorized as individuals who make over $400,000 and married couples who make over $450,000—once again pay 39.6 percent.

A number of celebrities have been jailed for tax fraud.

Transcendentalist author Henry David Thoreau spent a night in jail in 1846 for refusing to pay a poll tax. Although this was purportedly a protest against the Mexican-American War and slavery, other celebrities have been jailed for less principled tax evasion. The first winner of the reality TV show “Survivor,” for example, was convicted in 2006 and sentenced to more than four years in prison for failing to pay taxes on his $1 million grand prize. Actors Richard Pryor, Wesley Snipes and Sophia Loren also spent time in jail on tax charges, as did Chicago mob boss Al Capone. Meanwhile, in 1990 country singer Willie Nelson received a bill for $16.7 million in back taxes, which was followed by the auctioning of most of his assets.

Even the Internal Revenue Service’s own supervisors occasionally land in hot water. Former commissioner Joseph Nunan was busted in 1952 for not reporting $1,800 in gambling winnings. And Timothy Geithner’s 2009 nomination to head the Treasury Department, which oversees the IRS, was nearly derailed over disclosures that he had failed to pay about $35,000 in self-employment taxes.

Richard Nixon also struggled with tax irregularities.

In 1973 an IRS employee leaked President Richard Nixon’s tax information, which showed that he only paid $793 in federal income tax in 1970 and $878 in 1971 on annual income of more than $200,000. Nixon faced criticism for claiming a number of controversial deductions, including one of $576,000 for donating vice presidential papers to the National Archives. Even worse, it soon came out that a White House aide had illegally backdated the National Archives donation so as to meet the cutoff for writing it off. In response to a reporter’s questions about these irregularities, Nixon famously declared in November 1973 that he was “not a crook.” Though he was never charged with fraud, he was forced to pay $465,000 in back taxes and interest following an IRS re-audit. By August 1974 the unrelated Watergate scandal had forced Nixon from office. Since then, every president has voluntarily disclosed his tax data.

The U.S. tax system is extraordinarily complex.

When the income tax came into being a century ago, the U.S. tax code was only 11,400 words. It has since ballooned to roughly 3.8 million words, with complementary IRS regulations that are even longer. By contrast, Leo Tolstoy’s epic novel “War and Peace” is fewer than 600,000 words, and all of William Shakespeare’s works combined don’t reach 900,000 words. Even the IRS’s own internal watchdog concedes that the federal tax system has become unwieldy. With complications galore, the average American now spends about 27 hours a year preparing their returns.