Cameron Britton (left) as serial killer Ed Kemper, Jonathan Groff (right) as FBI Agent Holden Ford in Mindhunter. Photo: Merrick Morton/Netflix/Merrick Morton/Netflix

In a scene from the new Netflix series Mindhunter, an FBI agent asks, “How do we get ahead of crazy if we don’t know how crazy thinks?” It’s an apt summary of the project of criminal profiling: If cops could truly understand the twisted psychology of serial killers and psychopaths, they could catch them and stop their horrible crimes. If only it were so simple.

Mindhunter, based on the real-life experiences of FBI agents who interviewed serial killers in order to learn how they think, is the latest offering in a modern media landscape littered with depictions of the criminal profiler as a kind of all-seeing Sherlock Holmes. In movies like Silence of the Lambs and TV shows like Criminal Minds, FBI profilers piece together telltale clues to form an unerring picture of a warped and dangerous mind. But despite its firm grasp on the popular imagination, the real-life validity of criminal profiling remains a matter of some dispute. Profilers point to anecdotal successes. Critics point to bungled investigations. Systematic studies show that the practice is dubious at best, and nothing like it’s portrayed on TV.

Modern criminal profiling began with the hunt for the Mad Bomber, who planted dozens of bombs in a variety of locations in New York City and evaded capture for 16 years in the ’40s and ’50s. At their wits’ end, the police consulted psychiatrist James Brussel, who provided them with an incredibly detailed picture of the man behind the bombs, including the claim that he was an unmarried man and wore a double-breasted suit, buttoned.

When police eventually arrested the bomber, George Metesky, he was in fact single and wearing a buttoned double-breasted suit. And even though that could have simply been a lucky guess (accurate thanks to men’s fashion at the time), and despite the fact that many of Brussel’s other predictions were either wildly off or were of little value in actually solving the crime, criminal profiling took root as a legitimate investigative tool.

In the 1970s, the FBI began developing a system of criminal profiling based on interviews with 36 of the most notorious violent, imprisoned criminals, including Ted Bundy and John Wayne Gacy. (Mindhunter is based on a book by one of the FBI agents who conducted these interviews.) They came up with a six-stage process for developing a profile by putting together telltale signs of a killer’s signature — the particular crime-scene behaviors they believed betray the offender’s unique personality.

Mary Ellen O’Toole, an ex-FBI profiler, said experts can look at a crime scene and draw conclusions about how much planning was involved, whether it was a crime of opportunity or a targeted attack, how sophisticated the offender was about cleaning up forensic evidence, and if injury patterns show evidence of sexual sadism — conclusions that help form a picture of the offender. But according to criminologist Dan Kennedy, this kind of profiling rests on a fundamental fallacy, what he calls the homology problem — “The idea that there’s going to be some correlation between your day-to-day self and what you do at a crime scene.” It may seem like common sense that consistent criminal behavior reflects a consistent personality or character, but if that correlation does exist, Kennedy said, it’s either too weak or too irregular to be of any use.

A study by researchers at the University of Liverpool provides some support for this claim. A review of 100 cases involving stranger rapists showed that similarities between crime scenes had no correlation with similarities between criminals. “These findings indicate no evidence for the assumption of a homology between crime scene actions and background characteristics for the rapists in the sample,” the authors wrote.

Even at the time that FBI profilers were developing their techniques, Kennedy writes, psychologists believed that human behavior was largely responsive to situational conditions, and even performing a task the same way repeatedly is not an indication of predictable behavior in other areas of life.

Kennedy also points out that the FBI’s original profiling principles were based largely on information collected from known psychopaths, liars, and manipulators. The FBI interviews with serial killers represent not only a tiny sample — it’s also a self-selecting group of men who wanted to talk about themselves, and who had been caught.

“They question 36 sociopaths and take everything they say as gospel,” said Mike McGrath, a forensic psychiatrist and professor at the University of Rochester. As a result, FBI profilers often end up with suspiciously similar-sounding profiles. “Everybody is a white male 25 to 35 years old with a problem with their mother,” McGrath said. “And the profiles are useless from an investigative point of view.”

This failing became apparent when an unknown serial killer was terrorizing the Washington, D.C., area, picking off innocent victims with a sniper rifle. Criminal profilers were full of ideas about what kind of person police should look for. They announced that the murderer was probably an employed but disgruntled white man in his 20s or 30s, likely in a white van.

But when police finally arrested the two so-called Beltway Snipers in October 2002, they found that the murderers were unemployed black men, ages 17 and 40, in a blue sedan. Police had previously noticed the car near several shooting locations, but hadn’t closed the loop. They had even pulled the car over two hours before one of the deadly attacks, but never questioned the two men — because they didn’t fit the profile.

It’s not easy to scientifically test the usefulness of criminal profiling. One method researchers have tried, however, is to show case files from solved crimes to profilers and others and compare the accuracy of the profiles they generate. The results have been mixed; profilers were more accurate on a rape case file, but did no better than average on a murder file. Another study of this type showed that profilers were more accurate than laypeople, but it also showed that they weren’t uniformly skilled. The profile group had the greatest variability of any of the other groups, which included psychics and psychologists. A 2007 meta-analysis of several of these studies found that “profilers do not decisively outperform other groups when predicting the characteristics of an unknown criminal,” and referred to criminal profiling as a “pseudoscientific technique.”

Mark Safarik, a former FBI profiler, said these studies often use a multiple-choice format, which limits their value as a validity test, since it’s not a true representation of the task of constructing a profile.

Profiling problems are often due to overly specific claims and overconfidence in those claims, which can derail an investigation by giving detective’s tunnel vision, said Kennedy. He easily rattled off a list of cases where FBI profilers came up short. Their description of the Unabomber turned out to be a mixture of accurate (he was a loner living in a rural area) and inaccurate (he was older and more educated than predicted). Profilers investigating a 1989 blast on the U.S.S. Iowa said it was a murder-suicide due to unrequited homosexual love, but investigators later concluded that it was caused by either human error or equipment failure. Profilers in the investigation of the Green River Killer dismissed a letter from the murderer as fake (it turned out to be real) and helped target the wrong man, an innocent taxi driver. But Kennedy hastened to add that profiling isn’t entirely useless; it just hasn’t “lived up to the hype.”

“There’s no question that it’s of value to investigators,” said Ed Imwinkelried, a law professor at the University of California in Davis. Criminal profiling can be used like any other investigative tool to generate leads and help cast a wide but targeted net. But profilers have to be able to back up their statements with concrete evidence, and profiling should never be used to finger a specific individual, he said.

Safarik said good profilers are cautious, try not to overreach, and simply help to narrow down a field of potential suspects. “It’s not a science. But there are scientific aspects to it,” he said. Safarik said his expertise is in sexual murders of elderly women and that he’s studied that particular kind of murder enough that “I know empirically what type of offenders I’m looking for.”

Asked about successes, Safarik said he helped investigators who were stumped by two double homicides of elderly couples in 2006 in Iredell County, North Carolina. He took a look at the evidence and decided they were barking up the wrong tree. “I said, ‘I think you’re dealing with an older female offender and that she has mental-health problems,” Safarik said. Police ended up charging a 65-year-old woman.

But Safarik was also the subject of a recent Los Angeles Times story saying that his murder-trial testimony helped send an innocent man to jail. Safarik said he never would have written a report for investigators in that case if he knew, as he later found out, that the sheriff’s office had failed to interview all the witnesses at the crime scene. He also said his role in the prosecution was overblown, and that his testimony was thoroughly vetted and deemed acceptable in three trials.

Asked about how the validity of profiling should be measured, Safarik said that it comes down to how law enforcement views it. Proponents of profiling accurately point out that the technique remains popular among investigators, and its use has steadily increased. But, Kennedy argues, such surveys are likely biased by the fact that detectives have an interest in justifying the time and energy spent on the technique. He also highlights a study that examined 88 solved cases and found that while police reported that profiling was generally helpful 83 percent of the time, they also reported that the technique helped to actually identify the suspect in a far smaller percentage of the cases.

Many studies of criminal profiling end with a seemingly boilerplate conclusion remarking on the lack of empirical evidence that profiling works, followed by a call for more research on the topic. A 2015 study states that profilers are reluctant to cooperate with rigorous research of their practices, resulting in the “somewhat circular argument” that the proof of the technique’s validity lies in the fact that people keep requesting it. McGrath said the field also remains stymied by lack of data from the FBI: “There’s really not been any formal assessment of their successes and failures because they don’t turn over their files.” (In response to a request for comment for this article, an FBI spokesperson said the bureau would need weeks to connect me with an expert.)

Nevertheless, profiling’s grip on pop culture shows no sign of releasing. The reason may be very simple, Kennedy said: “It hangs around in the public mind because it’s cool. It’s TV land.”