Fueled by explosive reporting, foreign totalitarian regimes, and a lifetime of dystopian sci-fi, the movement to ban the government’s use of face recognition technology is growing. As of now, seven cities across the country—San Francisco, Oakland, and Berkeley in California, and Brookline, Somerville, Northampton, and Cambridge in Massachusetts—have banned the government use of face recognition. The entire state of California also passed a three-year moratorium on the use of face recognition on police body cameras. Part of the opposition stems from the fact that many don’t want police to have the ability to track movement around a city or learn the identities of protesters. But it’s also related to people’s unwillingness to be regarded with perpetual suspicion. Inclusion in a face recognition database means your face is always part of a lineup, and every time grainy surveillance footage surfaces of a robbery or assault, the suspect’s face is being compared with your own. And as a result of many states sharing driver’s license photos with the FBI, or private companies like Clearview A.I. harvesting billions of pictures off social media, this searching could happen from anywhere in the country. This doesn’t just mean there’s a small possibility that you could be mistaken for the suspect—it also means that your government doesn’t trust you.

The government has always sought a way to file away and compare the faces of the guilty, but until very recently the technology only allowed for it to occur in a much more rudimentary way. Before there was the fingerprint, or even the police file, there was the rogues’ gallery, which you could find in most U.S. police departments. The gallery was a large wall or cabinet filled with photographs of alleged criminals that could be used as a way of identifying repeat offenders and coordinating surveillance, and as an example for witnesses. Police often scrawled on the back of the photographs a basic biographical sketch of the suspect, including known aliases and previous arrests. In many larger cities during the 1890s and early 1900s, the booking and photographing of an arrested person was also accompanied by the taking of Bertillon measurements. Developed in France, the system involved taking at least five specific bodily measurements including head length, foot length, and length of the middle finger. This data could be easily filed away and cross-referenced in case a suspect changed his appearance drastically. As cumbersome as this technology was, its use in the early 20th century posed the same ethical questions about guilt, innocence, and the nature of governance that we continue to grapple with on an exponentially larger scale.

Suspicion is a circular process. To police departments of the early 20th century, a person was suspicious because his photograph hung in the rogues’ gallery, and people whom officers deemed suspicious got their picture taken and hung in the rogues’ gallery. This meant that a person, arrested on suspicion once, had the possibility of being arrested again and again by virtue of his initial arrest. Similarly, today a single encounter with the government that involves a photograph—whether for a driver’s license or a mug shot—has the chance to beget more.

“Nobody is going to employ a young man or a young woman whose picture adorns the rogues’ gallery.”

And also like today, there were those who opposed the vicious cycle of suspicion and arrest that biometric surveillance brought about. In 1899, the arrest of a known cheater and chronic gambler, Jacob “Doc” Owens, set in motion a debate over one of the 21st century’s most pressing issues regarding technology and privacy. On the night of May 10, 1899, Owens was arrested on suspicion after he was accused of fleecing passengers of a trans-Atlantic ocean liner at cards. At the station, he was photographed, and his measurements were taken and retained by the New York City Police Department. Owens was let go because there was no warrant out for his arrest and no reason to keep him—but his photograph remained.

Owens took objection to the fact that his face and measurements would continually be used by the NYPD—compared again and again against the descriptions and faces of murderers and arsonists and thieves. The continual retention of his biometric information was, according to his lawyers, a “great injury to his honor and reputation, causing him severe mental pain and anguish. … [H]is constitutional rights are being absolutely disregarded.” His outrage reflected that of many people at the time who felt that the growing technological advancements within the police department threatened to create a society without an assumption of innocence or the ability to be reformed after an indiscretion. Owens’ was one of many similar cases that peppered newspapers across the country. As new ways of organizing police departments, and new technologies, spread and proliferated across the United States (and the world) many cities grappled with the possibilities of police surveillance simultaneously.

As one columnist in Chattanooga, Tennessee, who covered the story wrote, “Once a photograph is hung in the gallery there is no removing it. A policy of that kind is utterly indefensible. It is a cruel injustice and a constant bar to reform. Youthful indiscretion is put on the same plane as murder. It provides no loophole through which the victim may escape the dire results of criminal publicity. Nobody is going to employ a young man or a young woman whose picture adorns the rogues’ gallery.” Simply put, once people were found guilty, or even deemed suspicious, they would remain that way forever.

After three months, the chief justice of the New York Supreme Court refused to grant “Doc” Owens his injunction to take the photograph down. Future challenges to this policy turned out to be more successful. But whether a person had success in getting his mug shot taken down—a stand-in for whether the government found you capable of redemption—had more to do with race and status than with the nature of the infraction. Some cases, however, demonstrated the type of person able to liberate themselves from constant gaze of police.

In 1909, another case brought the question again to the fore. George B. Duffy was from a higher socioeconomic status than Owens. When he was 19 years old, he was arrested in New York on suspicion of being connected with a robbery—with the official charge later listed as “obstructing the street.” Duffy was acquitted, and almost immediately his father began a campaign to get his son’s photograph taken down off the rogues’ gallery. His case went better than Owens’. “The object of putting someone in the rogues’ gallery,” a reporter at the Brooklyn Eagle wrote during the case, “is to have the police force familiarized with his personal appearance, name, and residence, in order to watch and hunt him as one who has settled on a life of crime.” To many, Duffy was not that kind of person.

Duffy’s parents, employers, neighbors, and family pastor all rallied to his cause. Newspapers as far away as San Francisco covered the story and reported on the injustice of a respectable white young adult being subjected to unfair police surveillance. When the police commissioner fought public opinion by refusing to take down the photograph and destroy Duffy’s measurements, the mayor intervened. Over the fate of a 19-year-old’s reputation, a veteran police commissioner was eventually asked to resign.

Police found the ability to erase a criminal record or hide a potential criminal from the view of police troubling. To them, and to citizens concerned with crime, everyone who had gotten far enough to be photographed by police deserved a life under the scrutiny of the public and surveillance of the state. Their fear of criminals run amok was compounded that same year by the introduction of the bill into the New York State Assembly that would outlaw the photographing, measuring, or fingerprinting of an arrestee until after he had been convicted of a crime.

“There were continual outbreaks of protest against this photographing of criminals, caused by arrested persons who did not feel that their portraits should be made,” wrote NYPD Commissioner Theodore Bingham, who had been ousted as a result of the Duffy controversy.

After one of these, a splendidly worded bill introduced in the State Senate, which forbade the taking of any portraits by the police until after conviction. … The only trouble with it was that, if it had passed, the Police Department would have had no pictures of criminals at all, for the simple reason that all criminals pass out of the custody of the police, not merely upon conviction, but from the very first moment they are arraigned.

A number of police detectives testified that the bill would make cities unsafe and that its support within the Senate was propped up by “rich rogues” who want police to forget their faces so that they could continue their wrongdoing.

Ultimately, the bill was defeated. One of the more compelling arguments against it came from the ability of the department to share its information. Testifying before the New York Senate, Deputy NYPD Commissioner Arthur Woods talked about the value of hoarding biometric information, including photographs, by relaying the story of four English pickpockets caught in Grand Central Terminal. Their information was sent to London, where the London police confirmed their identities and sent a copy of their existing criminal records back to New York.

Technology may have changed, but the moral dilemmas over the guiding principles behind state surveillance have not. In 1909, the NYPD’s files were not accessible instantaneously, and the limited archives of a police department didn’t have the capacity to accumulate faces of millions of people. These technological advances have meant that the moral quandary of the rogues’ gallery is a supersize version of what it once was. While people no longer laugh and sneer at people whose faces hang in the police station, they can be denied jobs or passports because of misreadings or misunderstandings buried in hoarded data. It is the continuation of this mentality that led police to subject a large percentage of black men in Los Angeles to constant harassment when, for instance, wearing a sports jersey got them entered permanently into the LAPD’s gang database. To be constantly read as suspicious, one need not have been arrested and acquitted, but simply to have sat for a photograph in the DMV or wear a jersey in public.

As one 1909 letter to the editor of the Atlanta Constitution read, “The penology of the future will make just as adequate and intelligent provisions,” when navigating the “perplexing juncture” of suspicion, identification, and surveillance. But we have not. In fact, the criminal justice system of the 21st century has only succeeded in making suspicion more ubiquitous. Independent of the usual arguments activists make when advocating for governmental bans on the use of face recognition—that it’s unreliable or that it misidentifies people—is the fundamental and historical truth that face recognition means we are all constantly under suspicion. We’re all in the rogues’ gallery now.

Future Tense is a partnership of Slate, New America, and Arizona State University that examines emerging technologies, public policy, and society.