I guarantee you’ll hear the phrase “My ancestors came here legally” in the aftermath of President Obama’s immigration address. It’s almost impossible to find any conversation about immigration—between elected officials, pundits, online commenters—in which at least one participant doesn’t use the phrase. It’s an understandable position, through which the speaker can both defend his or her family history and critique current illegal immigrants who choose to do things differently. It helps deflect charges of hypocrisy (since most Americans are descended from immigrants). It’s hard to argue with. And it’s also, in nearly every case, entirely inaccurate.

Prior to 1875’s Page Act and 1882’s Chinese Exclusion Act, there were no national immigration laws. None. There were laws related to naturalization and citizenship, to how vessels reported their passengers, to banning the slave trade. Once New York’s Castle Garden Immigration Station opened in 1855, arrivals there reported names and origins before entering the U.S. But for all pre-1875 immigrants, no laws applied to their arrival. They weren’t legal or illegal; they were just immigrants.

Moreover, those two laws and their extensions affected only very specific immigrant communities: suspected prostitutes and criminals (the Page Act); Chinese arrivals (the Exclusion Act); immigrants from a few other Asian nations (the extensions). So if your ancestors came before the 1920s and weren’t prostitutes, criminals, or from one of those Asian nations, they remained unaffected by any laws, and so were still neither legal nor illegal. This might seem like a semantic distinction, but it’s much more; the phrase “My ancestors came here legally” implies that they “chose to follow the law,” yet none of these unaffected immigrants had to make any such choice, nor had any laws to follow.

The 1892 opening of Ellis Island didn’t change these fundamental realities. Ellis arrivals had to wait in line and answer a list of questions, and could be quarantined if they had a communicable disease or were visibly insane. But if they weren’t in those aforementioned few illegal categories, they still weren’t affected by any law, made no choice of how to immigrate. Moreover, many arrivals during this period came not through Ellis but across the borders, which were unpatrolled and open.

Only with the 1920s Quota Acts did Congress establish national immigration laws encompassing most arrivals. But those acts were overtly discriminatory, extending the Exclusion Act’s principles by categorizing arrivals by nationality and drastically limiting certain groups; South Carolina Senator Ellison Smith put it bluntly: “It seems to me the point as to this measure is that the time has arrived when we should shut the door.” Since immigrants had no control over their nationality, it’s difficult to argue that post-1920s arrivals “chose” to immigrate legally or illegally. And since the borders remained largely open and there were multiple entry points, it’s hard to say that any individual arrival was under the quota and thus legal or illegal.

The 1965 Immigration Act ended national quotas, instituting preferences based on less overtly discriminatory categories such as family connections and educational/professional training. Subsequent laws (such as the 1986 IRCA) further adjusted national policy. But as the ubiquitous “my ancestors” phrase proves, current immigration debates aren’t just about present policies—they’re always informed by ideas about history, and specifically about legal and illegal immigration in our past. So it’s vitally important that we begin to use those terms accurately—to recognize that for so many of us, our ancestors were neither legal nor illegal immigrants. That they came in the same way contemporary undocumented immigrants do: by crossing a border.

Ben Railton is an Associate Professor of English at Fitchburg State University and a member of the Scholars Strategy Network.