[eerie music] RAGAVAN: You know when a person is working on something, and it’s good, but it’s not perfect, and he just tries for perfection? ♪ ♪ That’s me in a nutshell. [audio muffled, distorted] RAY: I just want to re-create humans. DAVID: Okay, but—but why? RAY: I don’t know. I mean, it’s that feeling you get when you achieve something big. ♪ ♪ AUSTIN: [echoing] It’s really interesting. You hear these words coming out of your voice, but you never said them. RAGAVAN: Let’s try again. We’ve been working to make a convincing total deepfake. The bar we are setting is very high. HASH: So you can see it’s not perfect. - We’re trying to make it so the population would totally believe this video. RAY: Give this guy an Oscar. STEPHEN: There are definitely people doing it at Google, Samsung, Microsoft. The technology moves superfast. DAVID: Somebody else will beat you to it... - Someone else will, and that will hurt. RAGAVAN: Okay, let’s try again. Just make it natural, right? - It’s hard to be natural. - It’s hard to be natural when you’re faking. Go. What are you up to these days? AUSTIN: Today I’m announcing my candidacy for the presidency of the United States [laughter] And I would like to announce my very special running mate— the most famous chimp in the world, Bubbles Jackson. [laughter] RAY: People do not realize how close this is to happen. Fingers crossed. It’s gonna happen, like, in the upcoming months. Yeah, the world is gonna change. - If I squint my eyes. [laughter] HANY: Look, this is how we got into the mess we’re in today with technology, right? A bunch of idealistic young people thinking, “We’re gonna change the world.” PERSON: It’s weird to see his face... [laughter] - I wondered what you would say to these engineers. - I would say, “I hope you’re putting “as much thought into how we deal with the consequences “of this as you are into the realization of it. This is a Pandora’s box you’re opening.” [solemn music] ♪ ♪ [traffic buzzing, indistinct chatter] RAGAVAN: We’re not gonna make it, I believe. HASH: I think we could. Let’s try to hold the door open. RAGAVAN: Okay. PERSON: Let’s go, let’s go, let’s go, let’s go. [tram beeping] [man laughs] DAVID: There’s a group of artificial-intelligence whiz kids in Toronto. PERSON: The audio’s mostly there, right? PERSON: Yeah. DAVID: They’re racing toward a technological breakthrough that will shatter our trust in what we see and hear. RAGAVAN: What about him announcing... PERSON: Presidency? BOTH: Yeah, yes. RAGAVAN: He could throw the election. [laughter] PERSON: Yeah. PERSON: Hello. PERSON: Hello. - Hey, Lou. PERSON: So should we get started? DAVID: They work at a startup called Dessa, developing artificial-intelligence technology for banks and insurance companies. PERSON: I don’t think there’s anything more meaningful in life than being able to just constantly come into work every day, spend the majority of the time just making and creating good shit. [laughs] We have cake. PERSON: Oh, we have cake. DAVID: But it was their side project that caught my attention. RAGAVAN: It’s not perfect yet. We need perfection. DAVID: They were trying to create a perfect deepfake. [ominous music] ♪ ♪ From state propaganda to clever, viral ad campaigns, we’ve long been give reasons to be skeptical of what we see and hear. The more you express yourself, the more we own you. - We’re dying... DAVID: But deepfakes are different. - I genuinely love the process of manipulating people online for money. DAVID: They’re made with a rapidly evolving form of technology. - President Trump is a total and complete dipshit. DAVID: Artificial intelligence is used to manipulate video, to make people do and say things they didn’t do. - Well, when the President does it, that means that it is not illegal. DAVID: And that’s what brought me here to Dessa. [phone line trilling] RECEPTIONIST: Hello? - Hi, this is David Barstow from “The New York Times.” We’re back. Hey, guys, how are you? Working nights and weekends, this team had just created the audio equivalent of a deepfake video, using AI to clone a voice. Hash. What’s going on? There was Hash... and Joe, both machine-learning engineers. - We have two work streams... DAVID: Ray was still in engineering school. - We do have to be careful. DAVID: And Ragavan, a Dessa founder, sort of kept an eye on everyone. Now, guess whose voice they chose to fake. - It’s a hoax. DAVID: Trump... - Whoa. DAVID: Biden... - [vocalizing] DAVID: Beyoncé... Uh-uh. - What the fuck are you talking about? DAVID: Stand-up comic, MMA color commentator... - Cheers, by the way. - Cheers. Hey, this is great whiskey. DAVID: And one of the most prolific podcasters in the world... - What kind of gene pool are they drawing from? - Joe Rogan. - Joe Rogan. - Joe Rogan. DAVID: Joe Rogan. - We did a lot of experimentation. - Thousands of lines of code. - They might design some sort of organic propulsion system, like the way squid have and shit. DAVID: By training an algorithmic model on hours of Rogan podcasts... ROGAN: Have you seen that video? What are they doing? I just don’t know how you manage your time. DAVID: They came up with this. DEEPFAKE AUDIO: Hey, Joe Rogan. It’s me, Joe Rogan. Please come save me, man. These artificial-intelligence guys have trapped me in a machine. Check this out. DAVID: The Internet went wild. I was surprised to learn that it’s harder to fake a voice than video, and Dessa’s put way bigger companies to shame. DEEPFAKE: I’ve decided to sponsor a hockey team made up entirely of chimps. DAVID: Now they were on a quest for the holy grail of fakeness... - This is a deepfake test. HASH: So it’s really bad. DAVID: Trying to marry their synthesized voice with algorithmically manipulated video. And some of the largest tech companies in the world were racing to do the same. STEPHEN: Now that we’ve released the Rogan thing... DAVID: Right. - People have taken notice, and there are definitely people doing it at Google, Samsung, Microsoft. DAVID: Steve Piron, Dessa’s CEO, believes he can compete with the tech giants. He’s given his team three months to complete a total Rogan deepfake. STEPHEN: They do suffer from, like, a weird perfectionism. DAVID: Mm. - And they need to be pushed. I think they do well with deadlines. This space moves superfast... and the technology moves superfast. DAVID: Somebody else will beat you to it... - Someone else will, and that will hurt. DAVID: [chuckles] Okay. Dessa’s secret weapon is Ray, a 24-year-old student from Tunisia, whose work counts as credit toward his engineering degree. - I pretty much have to reformat everything. DAVID: He caught their attention with code for a machine-learning model that could convert text to speech light-years better than anything else online. It’s the key to their Rogan deepfake. PERSON: Okay. DAVID: With two months till deadline, he’s refining the model on David Attenborough, one of the world’s most recognizable voices. ATTENBOROUGH: From space, deserts are very conspicuous. With no cloak of vegetation to conceal them, strange formations are exposed in the naked rock. - We only needed 30 minutes of speech to do that. DAVID: The power of this machine-learning model is that, once trained, it can be made to say anything. So I could write a little something here, and you could have David Attenborough saying it? RAY: Yeah. - Do you mind if we do a little experiment? - Sure. DAVID: I’ve been having this argument with a friend of mine for years— who’s better, Brett Favre or Aaron Rodgers? And I’ve argued that Brett Favre is way better than Aaron Rodgers. So this is gonna be my way of having David Attenborough solve the fight once and for all. So I wrote... “In nature, there are lions and pussycats. Favre is a lion. Rodgers is a pussycat.” Let’s see what he does. [chuckles] Are we looking at your code? Is this the secret sauce? RAY: No, just maybe this command—let’s not... DAVID: Okay, let’s not focus that— Okay. [scattered chuckles] RAY: It will be a few minutes to synthesize. DAVID: Okay, that’s fine. All right. [tense music] ♪ ♪ RAGAVAN: There you go. - Ready? RAY: Press on the mouse. AUDIO DEEPFAKE: People have debated whether Aaron Rodgers is a better quarterback than Brett Favre. I can tell you from my study of nature that the answer is blindingly obvious. In nature, there are lions and pussycats. Favre is a lion. Rodgers is a pussycat. - [laughing] He couldn’t do “Favre.” That’s what hung him up. The fake David Attenborough fell a mere syllable short of perfection. It is possible to correct a misspelling, for example. When he got to, “Favre is a lion“... RAY: Ah, yeah. - He put this emphasis on it... AUDIO DEEPFAKE: Favre is a lion. DAVID: Which was really cool. It was like, that’s the way— - Yeah, I let the model— DAVID: You let the model predict how he would do it. - Yeah. HANY: They’re like kids. I mean, you know, they’re a bunch of 20-year-olds, right? And so—yeah, so I think they sort of came out of nowhere, but they obviously stumbled upon something. DAVID: Hany Farid is one of the world’s top authorities on synthetic media. But instead of making deepfakes, he develops tools to detect them. And Dessa’s deepfake Rogan voice caught his attention. HANY: The weak link has been audio. But if you can now synthesize a person’s voice, this is sort of, if you will, the missing link to create now the perfect fakes. - What worries you about that moment? - Yeah. Imagine a world where many people can create audio, video of a president, a candidate, a CEO, a general, a military leader saying and doing anything that you want them to. I can have a CEO saying, “Profits are down 20%.” That video goes viral, stock market crashes. I just need that to be true for ten minutes. Even if I can correct the record, it’s too late at that point. Think about personal fraud. Think about the national security implications. DAVID: Can you understand, though, why, you know, a group of really smart young people get sort of caught up in... - The act of discovery, right? HANY: Sure, of course I understand that, but that excuse is over. Look, this is how we got into the mess we’re in today with technology. A bunch of idealistic young people thinking, “We’re gonna change the world.” But I think we have to look at the landscape now and understand, there are consequences to what you are doing, and you have to think about those. DAVID: From the start, deepfakes took root in the dankest corners of the Internet. The term itself was coined by a Reddit user posting porn videos like these onto which celebrities’ faces were swapped without their consent. DANIELLE: Seeing the fake videos of celebrities morphed into porn... DAVID: Mm-hmm. - I felt it viscerally. It’s such a deep violation of your sense of sexual identity and integrity. DAVID: Danielle Citron is a legal scholar whose work on cyberstalking and privacy led her to deepfakes. DANIELLE: Anyone could be the subject of a deepfake sex video. - Like, what is the law’s response to putting a woman’s face into a porno film? - The law’s response is a failure, right? Around the globe, we lack laws that are well-tailored, sort of a criminal response to digital impersonations that are tantamount to identity theft. We don’t have them, country for country. And I think an important piece to note, too, is that in 1996, Congress granted online service providers an immunity from liability for all decisions related to user-generated content. - Mm-hmm. - And so sites can be engaged in the business of deepfake sex videos and say, “Because users post them, we have no liability,” and they would be right. [dramatic music] DAVID: Let me ask you another question. Have you been thinking about stamping in some way that this is fake? - Right. Thinking, maybe. Doing it, no. [laughter] - I think early on, some people in the company didn’t understand the gravity of it. They were like, “Oh, we should just tell people how it’s made.” RAY: Are you talking about me? - Was it—was it? [laughs] RAY: Me? DAVID: Part of the culture is to put stuff out, right? RAGAVAN: You guys got emails that were angry that we didn’t release it. - You mean from the machine-learning community? RAGAVAN: Yeah. - It’s looked down upon to not release how you do things. It’s almost like you’re stifling innovation in the field. - We can’t just say... “Here’s a demonstration of the nuclear bomb, “and here’s all the instructions on how to make a nuclear bomb.” - I think we’re still working through, like, what exactly this is... and what exactly... Our obligation is and our responsibility is. DAVID: Have you considered the possibility of whether maybe it’s just a really bad idea to sell this technology to anyone? - [groans] Yeah, I think we would... You know, we—we... Our commercialization efforts have not been— We’ve entertained phone calls. Uh... DAVID: Since their success cloning Rogan’s voice, Dessa has been contacted by groups, including Sinclair Broadcast Group... ALL: We are concerned about— DAVID: Renowned for making its anchors read identical scripts and exposed by Deadspin in this mashup video. ALL: This is extremely dangerous to our democracy. - As I understand it, they wanted to use the technology to kind of automate their anchors. You know, we didn’t walk down that path very long. - What about the CIA? - The CIA, uh... [tense music] There’s a bit of cool— it’s cool when the CIA calls. Um, it’s like, “We’ll pay you money, “and we can use it for whatever we want. And we’re the good guys,” which I think is true, but, you know, there’s, like— I think it was “whatever we want” that kind of stopped that conversation. DAVID: So you’re not gonna do business with the CIA? - We are not gonna do business with the CIA, yeah, yeah. DAVID: Sinclair confirmed that its digital-innovation team contacted Dessa about the voice technology but decided it wasn’t appropriate for Sinclair’s business. The CIA declined to comment. Well, maybe the bigger question is, have you had a conversation with someone who’s not a technologist, outside of this world, about this? I mean, you guys are in a bit of a bubble. - I have kind of a coach. DAVID: Uh-huh. - She’s, like, a psychologist. DAVID: Uh-huh. - Uh... DAVID: Like a life coach? - Well... like a life—yeah. DAVID: You know what I mean. - That sounds lame. Yeah, she is like a life— well, she is a life coach. I’m just embarrassed to admit that she’s a life coach. She’s a psychologist by training. DAVID: Okay. Uh-huh. - She was super freaked out... and that freaked me out. And she was— “Are you sure you’ve got the code locked down?” [dramatic music] REPORTER: Lawmakers are sounding the alarm on deepfake videos. REPORTER: Manipulated content could be the latest weapon in disinformation campaigns. REPORTER: And you can imagine if it was to be used in the 2020 election for a candidate to make it sound or appear that they said something... SCHIFF: I think the worst-case scenario is two or three weeks before an election, a video is released of one of the candidates that is completely fraudulent, that goes viral. That could easily be election altering. ♪ ♪ Where we’re probably farthest behind is, “How do we respond to it?” That’s an issue for the government, yes, but it’s an issue for private industry as well. DAVID: Social-media companies have already been put to the test on this front. - [slurring] So it’s really sad, and here’s the thing. DAVID: In May, a video of House Speaker Nancy Pelosi edited to make her sound drunk went viral. - [slurring] It was very, very, very strange. - [laughs] ADAM: I don’t think the companies are prepared. They certainly weren’t prepared for what the Russians did in 2016. I don’t think they’re fully prepared for what may happen at the 2020 election. DAVID: So far, we’ve not yet been confronted with a fully realized deepfake, and there’s a race to do that. They say it’s inevitable. I wondered what you would say to these engineers. - I would say, “This is a Pandora’s box you’re opening. “And when you get into that terrain, “where even with the best analysis, “you may not be able to disprove something, then we’re on very scary ground.” PERSON: Ready? Heh. [razor buzzing] DAVID: Two months in, the Dessa team was making their first serious attempt to combine the fake Rogan voice... RAGAVAN: Ray? RAY: Yeah? DAVID: With video manipulated to look like him. And they needed something surprisingly low-tech— RAY: Ready? DAVID: An actor who looked like Rogan. - Go. - It’s really interesting. You hear these words coming out of your voice, but you never said them. DAVID: Their coworker Austin read the script, an endless string of inside jokes, understood only by Rogan fanatics. RAGAVAN: What about the ethics of forcing chimps to play hockey? - Forcing them? These chimps have never had it better. We can hardly keep them off the ice. RAGAVAN: Let’s hear it. AUSTIN: Forcing them? These chimps have never had it better. - Cut. Thanks, Austin. DAVID: So you guys are going for the face swap? RAGAVAN: Yeah, yeah, that’s right. So, like, based on, you know, Austin’s facial expression. - Yeah. RAGAVAN: We use the AI to predict... DAVID: It will predict. RAGAVAN: Joe Rogan’s equivalent facial expression. - Okay. For the AI model to work its magic, Ray and Ragavan have trained it on an hour and a half of Rogan’s stand-up comedy. RAGAVAN: It’s relatively close. DEEPFAKE AUDIO: Forcing them? These chimps have never had it better. [laughter] - That’s awesome. DEEPFAKE AUDIO: We can hardly keep them off the ice. RAY: It’s weird to see his face with Rogan’s... DAVID: Can I come close? RAGAVAN: Yeah, yeah, yeah, sure, have a seat. - Okay. - Yeah. So Ray combine the audio, but— Well, I’ll let you see. DEEPFAKE AUDIO: Forcing them? These chimps have never had it better. We can hardly keep them off the ice. They just can’t wait to get back out there. We can hardly keep them off the— - Okay, so, like... AUDIO DEEPFAKE: They just can’t wait to get back... - ‘Cause the question is, can you get to something that would convincingly fool a bunch of people? RAGAVAN: Yeah, yeah. - And this wouldn’t do that. RAGAVAN: Yeah, yeah. - Right. Okay. [tense music] PERSON: I would say the face is still a work in progress. RAGAVAN: Is this the curry chicken? DAVID: While the Dessa guys struggled with their face swap, their competition announced breakthroughs that went viral. REPORTER: FaceApp has taken the social-media world by storm. REPORTER: Chinese face-swapping app Zao has become that country’s top smartphone app. DAVID: No one released a synthesized voice to rival Dessa’s, but Ray was still obsessed with improving it. RAY: I don’t know if I told you this, but, like, we are thinking about completely revolutionizing the way we think about text to speech. DAVID: What do you mean? RAY: If you pay very close attention, you see some noise artifacts. DAVID: You see artifacts. - Yes. - They’re, like, anomalies. - We want to get rid of those, make the audio completely real. DAVID: Then you’re eliminating the ability for someone to detect what’s real and what’s fake. RAY: For now, yes. - I mean, but why? - Sometimes you just start noticing the artifacts, and you don’t just—just... I aim for perfection. - And that irritates you? - Yes, I aim for perfection. I told you. DAVID: But deepfakes don’t need to be perfect to pose a threat. The very fact that they exist can be used to hide the truth. - That’s what I call the liar’s dividend. We have— DAVID: What does that mean? - So the liar’s dividend is the risk that liars will cry deepfakes to have us disbelieve the truth. DAVID: The deepfake is the dividend. - The deepfake is the dividend. And we saw it, right? Donald Trump has twice, different occasions, of the Access Hollywood tape... DONALD: And when you’re a star, they let you do it. You can do anything. BILLY BUSH: Whatever you want. DONALD: Grab ‘em by the pussy. DANIELLE: After admitting he said it and apologizing, he later said, “Oh, I don’t know if that was me. I don’t think that was me.” But don’t you think, honestly, here we are, 2020— I think he’s gonna come full-on with the liar’s dividend and say, “It’s just fake news. Come on, there’s this thing called deepfakes.” Oi. Right? [dramatic music] RAGAVAN: We’ll put in the fake audio afterwards and adjust it. Ready? RAY: Yeah. DAVID: As they push past their three-month deadline... PERSON: Two words— skates and bananas. RAGAVAN: Let’s do one more. DAVID: The Dessa team switched actors, but... - Some of the chimps are looking to buy their own houses now. DAVID: The script still didn’t make much sense. RAGAVAN: Okay, perfect. Thank you. DEEPFAKE AUDIO: Two words— skates and bananas. You’d be surprised how easy it is. Forcing them? These chimps have never had it better. - So go back to the... - These chimps have never had it better. RAGAVAN: When he says “better,” his mouth stays a little open. DAVID: Something’s not quite right. RAGAVAN: Yeah, yeah, yeah. - Yeah. RAGAVAN: So I think, uh... - Part of me feels... when I see the video, that we’re, like, just— there’s—there’s, like, a little bit more we could do. Like, you put in a little bit more effort, and it’d—it’d—it’d cross that magical threshold. And...and it’s... Um, yeah, something—something magical happens... or terrifying happens. DAVID: To me, we’re at an interesting moment in the story of deepfakes. The ones that people see so far, they’re all in that realm of, “Oh, that’s a cool thing.” RAY: Yeah. RAGAVAN: Yeah. - But no one’s actually— Like, the world thinks it knows about deepfakes... RAGAVAN: Yeah. RAY: But they don’t. - They don’t? RAY: That’s the thing about us humans, right? We only react after enough people suffered from something. That’s when we start building these defense mechanisms, right? I don’t like the idea either, but that’s how— what humans are. Like, “Problems are gonna happen out of this, and then we’re gonna build something to fix this.” DAVID: After six weeks of work... RAGAVAN: It’s just uploading so we can see it. DAVID: On their last attempt at a perfect deepfake, Team Dessa took a giant leap forward, with a fake script announcing the end of Joe Rogan’s podcast. - I’ve been thinking, and I realized that it’s been almost ten years since the first episode of the podcast. That’s fucking crazy. This show has become my life. That’s why I’ve decided to go out with a bang. So, on December 24th, we’ll be doing our last episode ever. That’s exactly ten years since the first episode came out. It’s been a good run, and we’re all going out on top. [tense music] - Holy camoly. RAY: Yeah. I looked at Rags, and I was like, “Holy shit. This could be the best one yet.” DAVID: ‘Cause I didn’t have this feeling the last time. And now you can see the teeth. RAGAVAN: The wrinkles under the eyes. DAVID: Yep. Whatever that difference is between 90% and 97%... - Right, right. - It’s been a good run, and we’re all going out on top. RAGAVAN: And, like... [stuttering] I’m sure if we show this to many people, they will be convinced that this is real. DAVID: Can you play it one more time? I just... DEEPFAKE: I’ve been thinking... DAVID: Watching these guys climb this engineering mountain... RAGAVAN: Watch the eyes. DAVID: Scrambling to be the first to the top, you could understand how smart people would see the danger of manipulated media... DEEPFAKE: It’s been a good run... DAVID: But go for it anyway. DEEPFAKE: And we’re all going out on top. STEPHEN: What was interesting is, maybe a month ago we released a deepfake detector. And I was surprised that it didn’t get as much traction. It got a little bit of traction. Folks downloaded it. But I... The—like, the—the disease is far more exciting and— than—than the cure. [bright music] DAVID: Standing on the edge of this new reality, I have to wonder, are the days of “seeing is believing” coming to an end? Will there be an app to help protect you from a post-truth world? Or will you have to ask, “Is this really my voice“? Ray’s optimistic. Are you?