Picture a map of the United States and what do you see? From the two coasts that frame the mental snapshot up to the friendly mitten of Michigan and down to the staggered edge of Texas, you might be thinking of the contiguous states.

That’s the “logo map” of the country, writes Northwestern historian Daniel Immerwahr, and it’s not quite right. In fact, he pointed out in a recent interview, it’s “only been a correct map of the country for three years of its history.”

It’s not just because the map is missing Alaska and the peppering of the Hawaiian islands. It also excludes the places that are still territories of the United States—Puerto Rico, Guam, the Commonwealth of the Northern Marianas, American Samoa and the U.S. Virgin Islands. It was also common for earlier iterations of U.S. maps to ignore former territories like the Panama Canal Zone, which the U.S. held from 1904 to 1976, and the Philippines, which the U.S. controlled from 1898 to 1946, minus when it was occupied by Japanese forces during World War II.

In his upcoming book How to Hide an Empire, Immerwahr sets out to tell the history of the Greater United States, what lies beyond the mainland. He traces the legacy of empire to the U.S.’s founding, explores why the nation avoids this particular part of its past and fills the book with fascinating stories from past and present territories. Immerwahr spoke with Smithsonian about these missing chapters of American history and what the U.S.’s empire looks like today.

How to Hide an Empire: A History of the Greater United States In How to Hide an Empire, Daniel Immerwahr tells the fascinating story of the United States outside the United States. In crackling, fast-paced prose, he reveals forgotten episodes that cast American history in a new light. Buy

What would you tell people who are surprised–or upset, even–to hear the United States described as an empire?

From day one the United States has included both states and territories, with the territories treated differently. In 1940, when the Philippines was a U.S. possession, 19 million U.S. nationals lived outside of the 48 states. That’s more than the immigrant or African-American populations at the time.

“Empire” can be used as a pejorative, but I’m not talking about the country’s character. I’m talking about its shape—its territories and outposts. Today, around four million people live in the U.S. overseas territories. That’s about the populations of Maine, New Hampshire and Rhode Island combined.

In your book, you lay out three acts of American history. Can you briefly summarize them?

The first act is a familiar one: it’s Western expansion. But if you pay attention to territory, it starts to look a little different. As soon as the U.S. becomes independent from Britain and becomes sovereign, it becomes a union of states and territories.

The second, which overlaps a little bit chronologically with the first, is overseas colonization, which is really easy to miss if you grew up in the mainland. Very quickly after the United States does all of its contiguous territorial expansion within North America, it starts claiming overseas territories. The difference is that it’s not even clear that these places are going to become states. In fact, some of them never do.

The third act is where we are today. The United States still has remnants of its colonial empire, for example, Puerto Rico, Guam, the Commonwealth of the Northern Marianas, American Samoa and the U.S. Virgin Islands. But it also has another form of territorial extent, which is 800 military bases overseas, outside of its borders and strewn across the planet.

You argue that the U.S. is one of the only countries that suffers this chronic confusion about its history as an empire. Why is that?

One of the fantasies of the United States since its founding is that it’s a republic and it differentiated itself from empire. But it’s true that from the moment of its independence, it was an empire of sorts—it had states and territories and had a differentiated political space in the way that empires do. It’s been a persistent myth that the U.S. is uniquely virtuous on the political stage among other great powers because it refuses to engage in empire.

How did the founders view Western expansion and early frontiers prior to Manifest Destiny?

If you were just reading a textbook, you might imagine that the United States was an expansive republic from day one and that George Washington and Thomas Jefferson just couldn’t wait to expand the borders of their small country within North America. The original founders were circumspect about Western expansion—they were deeply worried about what would happen if white people went too far away from eastern centers of power. There, they were harder to govern and also a little more likely to get into wars with Native Americans over lands, wars that might draw the government in. So at least for the first few decades, what you can see is leaders of the United States government trying to tightly control the process of white settlement.

You write that 1898 was a momentous break from the past in our history. Why?

In going to war with Spain and claiming some of Spain’s territories after its defeat, the United States starts incorporating places with very large non-white populations and starts to wonder whether territorial status will lead to statehood. In fact, it turns out that it doesn’t. The imperialists who are most excited about claiming, for example the Philippines, are also steadfast that the Philippines shouldn’t become a state in the same way that Kansas had passed from territory to state.

American territories have served as labs for various mainlanders’ experiments. What’s an example of such a test?

One alarming example is that of Dr. Cornelius Rhoads who went to Puerto Rico and was able to act with a kind of license that is unthinkable on the mainland. He tried to induce diseases in some of his patients just to see how they would fare. He intentionally didn’t treat the patients.

He actually wrote a letter in which he claimed to have murdered multiple of his patients and to have tried to murder more of them. It’s unclear if he actually did this, but this became a major scandal in Puerto Rico. Rhoads is still remembered there to this day as a villain. Yet, he was able to retreat to New York, not face any trial, not face any real consequences, not even get fired from his job.

Indeed, he instead got promoted within the medical establishment and became one of the founders of chemotherapy. Yet, even after all of this, after he died and there was a sort of major awards given in his honor, his mainland colleagues never even heard of what he’d done in Puerto Rico. So he’s known in Puerto Rico as a villain, but there’s so much informational segregation, so much blindness from the mainland that this career, this intensely ugly career that he had in Puerto Rico, he could just walk away from it. What happens in San Juan stays in San Juan, or at least it did for Cornelius Rhoads.

How does American history change when we look beyond the mainland?

Think about Pearl Harbor. It wasn't just an attack on Hawai'i. Within a period of 24 hours, the Japanese attacked Hawai'i, the Philippines, Guam, Midway Island and Wake Island. They also attacked the British colonies of Malaya, Singapore and Hong Kong. The Philippines was the largest U.S. colony when Japan invaded in 1941, and the war there was the bloodiest event to occur on U.S. soil, though you’ll find little mention of it in most textbooks.

What role did technology play in the United States giving up territories?

Traditionally, countries claimed colonies to secure resources or military outposts. By the 1940s, the United States had satisfied some of those needs with technology. Developing synthetic rubber meant it didn’t need tropical colonies to grow rubber. With aircraft and wireless communications, it didn’t require continuous swaths of land for railroad tracks and telegraph cables to maintain its military presence.

How did the U.S. decide which territories to make into states, which ones to let go, and which ones to kind of hold in limbo?

I don’t think it was totally obvious which of the territories would be states and which would be independent, but as far as I can tell, the one determinant that is most important is the presence of a large white settler population. Hawai'i never had the kind of white settlement that you see in Western territories like Minnesota, but it nevertheless did host a significant white population that made it more palatable for statehood from the perspective of the mainland. The same is true of Alaska.

How have U.S. courts been used in tandem with America’s empire-building?

When the U.S. starts claiming large populated overseas territories, it starts defining itself as a legal entity and its body of law differently through a series of court cases known as the Insular Cases. The Supreme Court rules that the Constitution, which one might previously have assumed applied to the entire country, actually was restricted in this application. The United States went out to the Philippines and up to Alaska, but the Constitution didn’t follow it to all of those places. That was accommodating empire by dealing with this potential paradox between being, at the one end, a republic, and the other, an empire. The way to handle this was through a legal split whereby there’s one part of the country that’s governed by the Constitution, and there’s an extra-constitutional zone that’s governed by a different set of laws.

How do we think about our territories today?

A lot of people don’t. Woodrow Wilson spoke of them as lying “outside the charmed circle of our own national life.” That attitude is ingrained. The territories almost never appear on maps of the country, and census statistics usually exclude them. (If it had been included, Manila would have been one of the top ten largest cities in the country in the 1940s.) You can see that neglect today in how little aid Puerto Rico and the U.S. Virgin Islands got [last year] after hurricanes Maria and Irma. Or by the lack of national attention to Typhoon Yutu, which laid waste to the Northern Marianas [this fall].

What steps could we take to make the territories a more recognized part of the country?

Right now, in mainland schools the territories usually come up only in a single history lesson, around 1898 and the United States’ war with Spain. It’s as if the territories exist only in the moment when they are acquired and then promptly vanish. It’s entirely possible to read, say, about the Second World War without learning about the million U.S. nationals that were killed in the Pacific colonies or about the internment of Alaska Natives. Rewriting textbooks to include the territories wouldn’t just help mainlanders think about the present plight of the territories. It would also yield a richer, more interesting, and more honest version of U.S. history, one that shows the country not as it wants to be, but as it is.