ASPEN, Colorado — The head of the FBI has spent the last several months in something of a panic, warning anyone who will listen that terrorists are “going dark”—using encrypted communications to hide from the FBI—and insisting that the bureau needs some kind of electronic back door to get access to those chats.

It’s an argument that civil libertarians and technology industry executives have largely rejected. And now, members of the national security establishment—veterans of both the Obama and Bush administrations—are beginning to speak out publicly against FBI Director Jim Comey’s call to give the government a skeleton key to your private talks.

“I hope Comey’s right, and there’s a deus ex machina that comes on stage in the fifth act and makes the problem go away,” retired General Michael Hayden, the former head of the CIA and the NSA, told The Daily Beast. “If there isn’t, I think I come down on the side of industry. The downsides of a front or back door outweigh the very real public safety concerns.”

It’s a bit of a role reversal for Hayden and Comey. More than a decade ago, when Comey was the deputy attorney general, he stopped a controversial NSA program to electronically eavesdrop without a warrant. Hayden was the man responsible for that surveillance project—and many, many others. To hear the lifelong intelligence officer argue against an opportunity for the government to collect more data is odd, to say the least.

The encryption issue was also one of several small, but telling, ways in which Comey seemed out of sync with some of his fellow members of the national security establishment here at the Aspen Security Forum. (Full disclosure: I was a moderator at a panel at the forum, which paid for my travel costs.)

The FBI director called ISIS the biggest threat to the American “homeland”; the man in charge of coordinating the campaign against the terror group overseas said instead, “ISIS is losing.” Comey argued that it didn’t matter whether a terrorist sympathizer took orders directly from ISIS or was merely inspired by the so-called Islamic State; Homeland Security Secretary Jeh Johnson told the forum the following night: “I disagree a little bit with Jim in that I think that the distinction between terrorist-directed and terrorist-inspired is a significant one.”

Terrorist recruiters often start conversations over an open social network like Twitter—and then switch to WhatsApp or another encrypted system when the conversations get serious. When that happens, Comey explained to the audience at the forum, “you’ve no ability with a court order to intercept and look at that communication. So it’s the nature of the technology that's stopping us.”

Often, FBI agents won’t even bother trying to get that court order if they see a terror suspect has moved to a secure channel. They’re unlikely to get anything from the encrypted conversation, as many senior officials here at the forum noted, echoing Comey’s concerns.

“We need,” Comey said, “to be able to get access to the information in those targeted individual cases.”

Since last fall, the FBI chief has been asking for the technical and legal means to unlock encryption when necessary. “It’s the equivalent of a closet that can’t be opened. A safe that can’t be cracked. And my question is, at what cost?” he asked in a speech last October. “We aren’t seeking a back-door approach. We want to use the front door, with clarity and transparency, and with clear guidance provided by law.”

It’s an approach that Hayden and others contend could undermine the cybersecurity of everyone. The FBI might try to keep a door only for itself. In practice, it doesn’t work that way. Criminals or other governments can walk in the same portal.

“A hole is a hole,” Hayden said. “Given that reality, Americans are well-served by a high water level of security for everyone.”

Nor is it clear that a back door is all that necessary. Hayden’s old agency, the NSA, has come up with a host of ways to circumvent encryption. And when those tricks don’t work, the FBI has a team of hackers who can break into a target’s computer.

What’s more, it’s still possible for the government to look at the “metadata” of a conversation—the record of who is talking to whom—even if the specific contents of that chat are obscured. Often that information is more valuable to an FBI investigator or an intelligence agency targeteer than what was said. As Hayden once noted, “We kill people based on metadata.”

That metadata can be married to Facebook posts, Tweets, Seamless orders, and Amazon reviews for a fuller picture of a target. Like the rest of us, terrorist suspects have a hard time staying off the grid.

“‘Going dark’ is a real problem. At the same time, we need to get more creative about how we gather information,” said a former senior official in the Obama administration. With use of social media and smartphones skyrocketing, “people are leaving digital trails in ways they never did before.” Those trails can be picked up by the FBI, whether a conversation is encrypted or not.

Besides, it isn’t supposed to be easy for the FBI to listen in on what we do, said Michael Chertoff, the secretary of the Department of Homeland Security during the last Bush administration. “We do not historically organize our society to make it maximally easy for law enforcement, even with court orders, to get information,” he told the forum. “I don’t think socially we do that.”

This isn’t the first intra-government fight over encryption, Chertoff noted. The last time an administration insisted on a technological back door—in the 1990s—Congress shot down the idea. And despite cries of “going dark” back then, the government found all kinds of new ways to spy. “We collected more than ever. We found ways to deal with that issue,” Chertoff told the forum.

There are signs that Comey himself might be moving past his original request for a door into encrypted chats. When asked if he wanted “some sort of key” to gain access to private conversations, the FBI director answered, “The answer is I don’t know exactly. I can picture the end state we need. We need judges’ orders to be complied with.”