There aren’t many historical episodes more firmly lodged in the United States’s national memory than the attack on Pearl Harbor. It is one of only a few events that many people in the country can put a date to: 7 December 1941, the “date which will live in infamy,” as Franklin D Roosevelt put it. Hundreds of books have been written about it – the Library of Congress holds more than 350. And Hollywood has made movies, from the critically acclaimed From Here to Eternity, starring Burt Lancaster, to the critically derided Pearl Harbor, starring Ben Affleck.

But what those films don’t show is what happened next. Nine hours after Japan attacked the territory of Hawaii, another set of Japanese planes came into view over another US territory, the Philippines. As at Pearl Harbor, they dropped their bombs, hitting several air bases, to devastating effect.

The attack on Pearl Harbor was just that – an attack. Japan’s bombers struck, retreated and never returned. Not so in the Philippines. There, the initial air raids were followed by more raids, then by invasion and conquest. Sixteen million Filipinos – US nationals who saluted the stars and stripes and looked to FDR as their commander in chief – fell under a foreign power.

Contrary to popular memory, the event familiarly known as “Pearl Harbor” was in fact an all-out lightning strike on US and British holdings throughout the Pacific. On a single day, the Japanese attacked the US territories of Hawaii, the Philippines, Guam, Midway Island and Wake Island. They also attacked the British colonies of Malaya, Singapore and Hong Kong, and they invaded Thailand.

At first, “Pearl Harbor” was not the way most people referred to the bombings. “Japs bomb Manila, Hawaii” was the headline in one New Mexico paper; “Japanese Planes Bomb Honolulu, Island of Guam” in another in South Carolina. Sumner Welles, FDR’s undersecretary of state, described the event as “an attack upon Hawaii and upon the Philippines”. Eleanor Roosevelt used a similar formulation in her radio address on the night of 7 December, when she spoke of Japan “bombing our citizens in Hawaii and the Philippines”.

That was how the first draft of FDR’s speech went, too: it presented the event as a “bombing in Hawaii and the Philippines”. Yet Roosevelt toyed with that draft all day, adding things in pencil, crossing other bits out. At some point he deleted the prominent references to the Philippines.

Why did Roosevelt demote the Philippines? We don’t know, but it’s not hard to guess. Roosevelt was trying to tell a clear story: Japan had attacked the US. But he faced a problem. Were Japan’s targets considered “the United States”? Legally, they were indisputably US territory. But would the public see them that way? What if Roosevelt’s audience didn’t care that Japan had attacked the Philippines or Guam? Polls taken slightly before the attack show that few in the continental US supported a military defense of those remote territories.

Roosevelt no doubt noted that the Philippines and Guam, although technically part of the US, seemed foreign to many. Hawaii, by contrast, was more plausibly “American”. Although it was a territory rather than a state, it was closer to North America and significantly whiter than the others.

Yet even when it came to Hawaii, Roosevelt felt a need to massage the point. So, on the morning of his speech, he made another edit. He changed it so that the Japanese squadrons had bombed not the “island of Oahu”, but the “American island of Oahu”. Damage there, Roosevelt continued, had been done to “American naval and military forces”, and “very many American lives” had been lost.

An American island, where American lives were lost – that was the point he was trying to make. If the Philippines was being rounded down to foreign, Hawaii was being rounded up to “American”.

One reporter in the Philippines described the scene in Manila as the crowds listened to Roosevelt’s speech on the radio. The president spoke of Hawaii and the many lives lost there. Yet he only mentioned the Philippines, the reporter noted, “very much in passing”. Roosevelt made the war “seem to be something close to Washington and far from Manila”.

This was not how it looked from the Philippines, where air-raid sirens continued to wail. “To Manilans the war was here, now, happening to us,” the reporter wrote. “And we have no air-raid shelters.”

Hawaii, the Philippines, Guam – it wasn’t easy to know how to think about such places, or even what to call them. At the turn of the 20th century, when many were acquired (Puerto Rico, the Philippines, Guam, American Samoa, Hawaii, Wake), their status was clear. They were, as Theodore Roosevelt and Woodrow Wilson unabashedly called them, colonies.

That spirit of forthright imperialism didn’t last. Within a decade or two, after passions had cooled, the c-word became taboo. “The word colony must not be used to express the relationship which exists between our government and its dependent peoples,” an official admonished in 1914. Better to stick with a gentler term, used for them all: territories.

Yet a striking feature of the overseas territories was how rarely they were even discussed. The maps of the country that most people had in their heads didn’t include places such as the Philippines. Those mental maps imagined the US to be contiguous: a union of states bounded by the Atlantic, the Pacific, Mexico and Canada.

That is how most people envision the US today, possibly with the addition of Alaska and Hawaii. The political scientist Benedict Anderson called it the “logo map”, meaning that if the country had a logo, this shape would be it:

The US ‘logo map’

The problem with the logo map, however, is that it isn’t right. Its shape does not match the country’s legal borders. Most obviously, the logo map excludes Hawaii and Alaska, which became states in 1959 and now appear on virtually all published maps of the country. But it is also missing Puerto Rico, which, although not a state, has been part of the country since 1899. When have you ever seen a map of the US that had Puerto Rico on it? Or American Samoa, Guam, the US Virgin Islands, the Northern Marianas or any of the other smaller islands that the US has annexed over the years?

In 1941, the year Japan attacked, a more accurate picture would have been this:

A map of the ‘Greater United States’ as it was in 1941

What this map shows is the country’s full territorial extent: the “Greater United States”, as some at the turn of the 20th century called it. In this view, the place normally referred to as the US – the logo map – forms only a part of the country. A large and privileged part, to be sure, yet still only a part. Residents of the territories often call it the “mainland”.

On this to-scale map, Alaska isn’t shrunken down to fit into a small inset, as it is on most maps. It is the right size – ie, huge. The Philippines, too, looms large, and the Hawaiian island chain – the whole chain, not just the eight main islands shown on most maps – if superimposed on the mainland would stretch almost from Florida to California.

This map also shows territory at the other end of the size scale. In the century before 1940, the US claimed nearly 100 uninhabited islands in the Caribbean and the Pacific. Some claims were forgotten in time – Washington could be surprisingly lax about keeping tabs. The 22 islands included here are the ones that appeared in official tallies (the census or other governmental reports) in the 1940s. I have represented them as clusters of dots in the bottom left and right corners, although they are so small that they would be invisible if they were drawn to scale.

The logo map is not only misleading because it excludes large colonies and pinprick islands alike. It also suggests that the US is a politically uniform space: a union, voluntarily entered into, of states standing on equal footing with one another. But that is not true, and it has never been true. From its founding until the present day, the US has contained a union of American states, as its name suggests. But it has also contained another part: not a union, not states and (for most of its history) not wholly in the Americas – its territories.

What is more, a lot of people have lived in that other part. According to the census count for the inhabited territories in 1940, the year before Pearl Harbor, nearly 19 million people lived in the colonies, the great bulk of them in the Philippines. That meant slightly more than one in eight of the people in the US lived outside of the states. For perspective, consider that only about one in 12 was African American. If you lived in the US on the eve of the second world war, in other words, you were more likely to be colonised than black.

My point here is not to weigh forms of oppression against one another. In fact, the histories of African Americans and colonised peoples are tightly connected (and sometimes overlapping, as for the African-Caribbeans in Puerto Rico and the US Virgin Islands). The racism that had pervaded the country since slavery also engulfed the territories. Like African Americans, colonial subjects were denied the vote, deprived of the rights of full citizens, called racial epithets, subjected to dangerous medical experiments and used as sacrificial pawns in war. They, too, had to make their way in a country where some lives mattered and others did not.

What getting the Greater United States in view reveals is that race has been even more central to US history than is usually supposed. It hasn’t just been about black and white, but about Filipino, Hawaiian, Samoan and Chamoru (from Guam), too, among other identities. Race has not only shaped lives, but also the country itself – where the borders went, who has counted as “American”. Once you look beyond the logo map, you see a whole new set of struggles over what it means to inhabit the US.

Looking beyond the logo map, however, could be hard for mainlanders. The national maps they used rarely showed the territories. Even the world atlases were confusing. During the second world war, Rand McNally’s Ready Reference Atlas of the World – like many other atlases at the time – listed Hawaii, Alaska, Puerto Rico and the Philippines as “foreign”.

A class of seventh-grade girls at the Western Michigan College Training School in Kalamazoo scratched their heads over this. They had been trying to follow the war on their maps. How, they wondered, could the attack on Pearl Harbor have been an attack on the US if Hawaii was foreign? They wrote to Rand McNally to inquire.

“Although Hawaii belongs to the United States, it is not an integral part of this country,” the publisher replied. “It is foreign to our continental shores, and therefore cannot logically be shown in the United States proper.”

The girls were not satisfied. Hawaii is not an integral part of this country? “We believe this statement is not true,” they wrote. It is “an alibi instead of an explanation”. Further, they continued, “we feel that the Rand McNally atlas is misleading and a good cause for the people of outlying possessions to be embarrassed and disturbed”. The girls forwarded the correspondence to the Department of the Interior and asked for adjudication. Of course, the seventh-graders were right. As an official clarified, Hawaii was, indeed, part of the US.

Yet the government could be just as misleading as Rand McNally on this score. Consider the census: according to the constitution, census takers were required to count only the states, but they had always counted the territories, too. Or, at least, they had counted the continental territories. The overseas territories were handled differently. Their populations were noted, but they were otherwise excluded from demographic calculations. Basic facts about how long people lived, how many children they had, what races they were – these were given for the mainland alone.

The maps and census reports that mainlanders saw presented them with a selectively cropped portrait of their country. The result was profound confusion. “Most people in this country, including educated people, know little or nothing about our overseas possessions,” concluded a governmental report written during the second world war. “As a matter of fact, a lot of people do not know that we have overseas possessions. They are convinced that only ‘foreigners’, such as the British, have an ‘empire’. Americans are sometimes amazed to hear that we, too, have an ‘empire’.”

The proposition that the US is an empire is less controversial today. The case can be made in a number of ways. The dispossession of Native Americans and relegation of many to reservations was pretty transparently imperialist. Then, in the 1840s, the US fought a war with Mexico and seized a third of it. Fifty years later, it fought a war with Spain and claimed the bulk of Spain’s overseas territories.

Empire isn’t just landgrabs, though. What do you call the subordination of African Americans? Starting in the interwar period, the celebrated US intellectual WEB Du Bois argued that black people in the US looked more like colonised subjects than like citizens. Many other black thinkers, including Malcolm X and the leaders of the Black Panthers, have agreed.

Or what about the spread of US economic power abroad? The US might not have physically conquered western Europe after the second world war, but that didn’t stop the French from complaining of “coca-colonisation”. Critics there felt swamped by US commerce. Today, with the world’s business denominated in dollars, and McDonald’s in more than 100 countries, you can see they might have had a point.

Flags on top of the fortress in Old San Juan in Puerto Rico. Photograph: Anton Gorbov/Alamy

Then there are the military interventions. The years since the second world war have brought the US military to country after country. The big wars are well-known: Korea, Vietnam, Iraq, Afghanistan. But there has also been a constant stream of smaller engagements. Since 1945, US armed forces have been deployed abroad for conflicts or potential conflicts 211 times in 67 countries. Call it peacekeeping if you want, or call it imperialism. But clearly this is not a country that has kept its hands to itself.

Yet among all the talk of empire, one thing that often slips from view is actual territory. Yes, many would agree that the US is or has been an empire, for all the reasons above. But how much can most people say about the colonies themselves? Not, I would wager, very much.

It is not as if the information isn’t out there. Scholars, many working from the sites of empire themselves, have assiduously researched this topic for decades. The problem is that their works have been sidelined – filed, so to speak, on the wrong shelves. They are there, but as long as we have the logo map in our heads, they will seem irrelevant. They will seem like books about foreign countries. The confusion and shoulder-shrugging indifference that mainlanders displayed at the time of Pearl Harbor hasn’t changed much at all.

I will confess to having made this conceptual filing error myself. Although I studied US foreign relations as a doctoral student and read countless books about “American empire” – the wars, the coups, the meddling in foreign affairs – nobody ever expected me to know even the most elementary facts about the territories. They just didn’t feel important.

It wasn’t until I travelled to Manila, researching something else entirely, that it clicked. To get to the archives, I would travel by “jeepney”, a transit system originally based on repurposed US army jeeps. I boarded in a section of Metro Manila where the streets are named after US colleges (Yale, Columbia, Stanford, Notre Dame), states and cities (Chicago, Detroit, New York, Brooklyn, Denver), and presidents (Jefferson, Van Buren, Roosevelt, Eisenhower). When I would arrive at my destination, the Ateneo de Manila University, one of the country’s most prestigious schools, I would hear students speaking what sounded to my Pennsylvanian ears to be virtually unaccented English. Empire might be hard to make out from the mainland, but from the sites of colonial rule themselves, it is impossible to miss.

The Philippines is not a US territory any more; it got its independence after the second world war. Other territories, although they were not granted independence, received new statuses. Puerto Rico became a “commonwealth”, which ostensibly replaced a coercive relationship with a consenting one. Hawaii and Alaska, after some delay, became states, overcoming decades of racist determination to keep them out of the union.

Yet today, the US continues to hold overseas territory. Besides Guam, American Samoa, the Northern Mariana Islands, Puerto Rico, the US Virgin Islands and a handful of minor outlying islands, the US maintains roughly 800 overseas military bases around the world.

None of this, however – not the large colonies, small islands, or military bases – has made much of a dent on the mainland mind. One of the truly distinctive features of the US’s empire is how persistently ignored it has been. This is, it is worth emphasising, unique. The British weren’t confused as to whether there was a British empire. They had a holiday, Empire Day, to celebrate it. France didn’t forget that Algeria was French. It is only the US that has suffered from chronic confusion about its own borders.

The reason is not hard to guess. The country perceives itself to be a republic, not an empire. It was born in an anti-imperialist revolt and has fought empires ever since, from Hitler’s Thousand-Year Reich and the Japanese empire to the “evil empire” of the Soviet Union. It even fights empires in its dreams. Star Wars, a saga that started with a rebellion against the Galactic Empire, is one of the highest-grossing film franchises of all time.

This self-image of the US as a republic is consoling, but it is also costly. Most of the cost has been paid by those living in the colonies and around the military bases. The logo map has relegated them to the shadows, which are a dangerous place to live. At various times, the inhabitants of the US empire have been shot, shelled, starved, interned, dispossessed, tortured and experimented on. What they haven’t been, by and large, is seen.

The logo map carries a cost for mainlanders, too. It gives them a truncated view of their own history, one that excludes part of their country. It is an important part. The overseas parts of the US have triggered wars, brought forth inventions, raised up presidents and helped define what it means to be “American”. Only by including them in the picture do we see a full portrait of the country – not as it appears in its fantasies, but as it actually is.

How to Hide an Empire: A Short History of the Greater United States by Daniel Immerwahr will be published by Bodley Head on 28 February. Buy it at guardianbookshop.com

• Follow the Long Read on Twitter at @gdnlongread, or sign up to the long read weekly email here.