The devastation wrought by Hurricanes Irma and Maria has reawakened many Americans to the existence of Puerto Rico as well as the archaic laws and domineering bureaucracies that continue to burden the island. News outlets have scrambled to explain the Jones Act, the 1920 law that restricts shipping between U.S. ports, the PROMESA board that Congress set up last year to oversee Puerto Rico’s finances, and the fact that territory residents can’t vote for president.

But beneath these data points lurks something deeper and more problematic, yet rarely discussed: the Insular Cases, a series of Supreme Court decisions from the early 20th century. When we talk about the differences between states and territories—and when we ask why the United States even has territories in 2017—we’re really talking about the legacy of the Insular Cases. The recent controversies are, in fact, the latest iteration of a conversation about American empire that goes back more than a century.

The earliest Insular Cases were decided by the same Supreme Court that allowed “separate but equal” segregation in Plessy v. Ferguson in 1896. That case was overturned, but the Insular Cases, which are built on the same racist worldview, still stand today.

Legal scholars disagree about how many Insular Cases there are—some say six; others include more than two dozen—but the general view is that they begin with Downes v. Bidwell in 1901. Up to then, the U.S. government considered territories to have the complete protection of the Constitution and a clear, straightforward path to statehood. This was true for all of the territories that existed on the North American continent at the time—Oklahoma, New Mexico, Arizona, and Alaska. But as the new century dawned, the federal government’s definition of what it meant to be a territory was shifting rapidly, as manifest destiny morphed into empire-building. President William McKinley and other leaders of the day aimed to bolster U.S. global stature by following the template of European powers: controlling the oceans by controlling islands, holding them not as equals but as colonies, as possessions.

Hawaii, which the USA annexed in 1898, largely fit this this new plan. In legal terms, though, it followed the existing territory model, as Congress followed the precedent of quickly granting it full Constitutional rights. But no such courtesies were offered when the U.S. acquired Puerto Rico, Guam, and the Philippines from Spain following the Spanish-American War, in 1898, or when the U.S. took over the islands now known as American Samoa in 1900.

That year, McKinley won re-election on a platform that celebrated overseas expansion and referred to residents of the newly-acquired territories as “rescued peoples.” His successor, Theodore Roosevelt, was even more explicit in praising “the expansion of the peoples of white, or European, blood” into the lands of “mere savages.” It’s worth noting that President Trump has pointed to this period as the era “when we were great, when we were really starting to go robust.”

Downes v. Bidwell was the codification of McKinley-Roosevelt mindset. The case, which centered on a question of whether shipments from Puerto Rico to New York were interstate or international, established a new hierarchy of territories: They were now either “incorporated” with the United States or “unincorporated,” with only the former having the full protections of the Constitution. The court reasoned that Puerto Rico and the other new territories were “inhabited by alien races,” so governing them “according to Anglo-Saxon principles may for a time be impossible.” These islands, then, were “foreign in a domestic sense.”

Three years later the court considered the Americanness of individual territory residents in Gonzales v. Williams, which concerned a woman who moved from Puerto Rico to New York and was detained as an “alien immigrant.” The court ruled that she was American, sort of: She was a “noncitizen national,” a distinction new to the United States and inspired by the designations European empires used for their colonial subjects. Puerto Ricans finally gained citizenship in 1917 through legislation passed by Congress. Residents of most of the other current territories later gained it the same way, although Guamanians had to wait until 1950. The “noncitizen national” designation still holds in American Samoa, where residents are, by birth, not citizens of any country. (Some of them prefer it this way; it’s complicated.)

Balzac v. Porto Rico in 1922 added yet more complexity, finding that the Constitution didn’t apply in full to the territories—only “fundamental” rights were automatically in effect. What “fundamental” means, though, is unclear: Legal matters resolved long ago in the states are still in question in the territories. For example, according to Christina Duffy Ponsa, a professor of legal history at Columbia Law School, the Sixth Amendment right to a criminal trial by jury has not been held applicable in local courts in Puerto Rico. Balzac v. Porto Rico also conclusively established that only incorporated territories were headed for statehood and that incorporation could occur solely through “an express declaration from Congress,” something that body has never granted.

The Insular Cases are the main reason no territory has become a state since Alaska and Hawaii 1959, the longest stretch the United States has ever gone without adding stars to the flag. The cases are why Congress effectively has full say over every law in the territories, from the PROMESA board to small things like a quickie divorce law in the U.S. Virgin Islands that came before the Supreme Court in 1955. And the Insular Cases’ legal offspring have created further disparities, in particular the 1980 case Harris v. Rosario, in which the court ruled it was acceptable for the federal government to fund programs like Medicare, Medicaid, and Aid to Families with Dependent Children Program at lower rates in the territories than in the states as long as there was a “rational basis.”

I spent more than two years researching the U.S. territories for my book The Not-Quite States of America and saw the harmful long-tail effects of the Insular Cases at every turn, from lack of funding and investment to gaps in oversight. In American Samoa, I met a woman from the U.S. Department of Labor who had recently flown in from the states to investigate labor complaints. It had been years since her colleagues had visited the territory, she told me, and the backlog of cases was such that she had to lengthen her stay from a few weeks to four months.

When I spoke with residents in each territory, they often expressed frustration over their lack of political voice—they can’t vote for president; their members of Congress can’t vote on the floor of the house—as well as the fact that Americans in the states are so clueless about the territories, largely unaware of their very existence. (“When I say I’m from Guam, they go, ‘You mean Guatemala?’ ” one woman told me.) A poll taken last month found that only 54 percent of Americans know that people born in Puerto Rico are U.S. citizens, and it’s unlikely that even half of Americans can name all five inhabited territories—or even know that there are five. And while there are plenty of reasons for this lack of awareness, the Insular Cases undoubtedly played a critical role by forcefully legally distinguishing the territories as “foreign” and impeding their paths to statehood, giving cover for federal neglect and societal indifference.

For a moment, a few years ago, the Insular Cases finally seemed to be cracking in Boumediene v. Bush (2008), which concerned a detainee held at Guantanamo Bay. The U.S. naval base in Cuba is also a quasi-territory. Writing for the majority, Justice Anthony Kennedy said, “It may well be that over time the ties between the United States and any of its territories strengthen in ways that are of constitutional significance” and said that the federal government did not “have the power to switch the Constitution on or off at will.”

Last year, two new cases offered a new round of tests. One sought birthright citizenship for American Samoans; the other concerned whether or not Puerto Rico was, like the states, a sovereign separate from the federal government. The latter, Puerto Rico v. Sanchez Valle, was especially contentious given that Puerto Rico is officially a commonwealth, a designation that—in theory—offers more local autonomy than being a territory and keeps the island off the United Nations’ official list of colonies. Considering it a separate sovereign would make it more like a state, while ruling it to be part of the federal government would effectively put it back into territory—and colony—status.

In both cases, the Obama administration, like its predecessors, fought to uphold the Insular Cases. I was finishing my book at the time and, with uncharacteristic optimism, wrote in the epilogue that change was coming for the territories. But the court declined to take the citizenship case and ruled that Puerto Rico was not, in fact, a separate sovereign.

I made my last-minute revisions and hoped that, well, maybe the next presidential administration, and the next Supreme Court, would work to undo the Insular Cases at long last.

Once again, I was naively hopeful: It’s abundantly clear nothing is going to change anytime soon. President Trump and his administration have repeatedly demonstrated indifference to Puerto Rico and the other territories, not just in the last few weeks but even during the presidential campaign, when Trump’s primary comment on the territories was an op-ed plagiarized from Ben Carson.

“No more will those who reside in the territories or commonwealths be ignored,” Trump wrote, but there’s no sign he intends to follow through in any meaningful way. Puerto Rico and the other U.S. territories remain foreign in a domestic sense, hard to categorize, easy to disregard.