Skip Straight to Trump vs. RoboTrump Quiz

So, that’s why I’ve been saying all along that, yes, I’d love to win. But, boy, do these guys want me to. These guys, they don’t talk about it. They say, “Donald Trump, please, please run.” Because they’ll take away your tax cuts, because they’ll take away your regulation cuts. They’ll take them away. And, frankly, really, really bad things will happen with our country. Our country would go down very quickly. Very quickly, very, very rapidly. The Democrats want to turn back the clock, which is, essentially, what they’ve done. They’ve turned back. We’ve gone much further left than anybody thought possible.

The above paragraph was written by an artificial intelligence. We call him “RoboTrump.”

Using this AI, it’s possible to generate an unlimited amount of convincing fake text. Literally millions of lines of text per hour can be generated.

What’s even scarier? Almost no one can tell it’s fake AI-generated text.

At Lawsuit.org, we asked 1,000 people to evaluate 20 paragraphs on 10 different political topics. Ten of the paragraphs were written by RoboTrump, and 10 were text from real Trump speeches.

On average, respondents guessed correctly only 40% of the time, or 10% worse than blindly guessing. This is proof of the massive danger presented by this new technology.

Thankfully, for most of our respondents, realizing their error sparked a deep change in their opinion on the dangers of AI-generated text. To understand their change in perspective, we asked them two questions. One before taking our quiz and one after.

How concerned are you that fake text, written by artificial intelligence, will be used to influence the 2020 election? (Asked before respondents took the quiz)

Half of the previous passages you read were written by an artificial intelligence trained on President Trump’s speeches. Given this, how concerned are you that fake text written by artificial intelligence will be used to influence the 2020 election? (Asked after the respondents took the quiz)

As you can see in the Sankey diagram below, many respondents changed their mind, moving from less concerned to significantly more concerned.

After taking the quiz, 432 of the 1,000 respondents (43%) said they were more concerned about the implications of AI-generated text on the 2020 election than prior to taking the quiz.

about the implications of AI-generated text on the 2020 election than prior to taking the quiz. Only 116 of the 1,000 respondents (12%) said they were less concerned about the implications of AI-generated text on the 2020 election than before taking the quiz.

After taking the quiz for yourself, it is our hope that you come to the same conclusion as many of our survey respondents:

Convincingly realistic text generated by AI on an unlimited scale is an existential risk to our democracy.

If you haven’t already, test your susceptibility to AI generated fake news with the quiz below:

</p>

<section>

<h2>

<h2><strong>Trump vs. RoboTrump</strong></h2>

</h2>

<p><p>A quiz to test your susceptibility to Artificial Intelligence generated fake text.</p>

</p>

</section>

<section>

<h2>

<p>The United States is leading a very broad coalition to deny terrorists control of their territory and populations, to cut off their funding, and to discredit their wicked ideology. I am pleased to report that the coalition to defeat ISIS has retaken almost 100 percent of the territory once held by these killers in Iraq and Syria. There is still more fighting and work to be done and to consolidate our gains. We are committed to ensuring that Afghanistan never again becomes a safe haven for terrorists who want to commit mass murder to our civilian populations. I want to thank those nations represented here today that have joined in these crucial efforts. You are not just securing your own citizens, but saving lives and restoring hope for millions and millions of people. When it comes to terrorism, we will do whatever is necessary to protect our nation. We will defend our citizens and our borders.</p>

</h2>

</section>

<section>

<h3>

<p> We defeated Isis and will fight terrorism. And we are going to defeat radical Islamic terrorism, which is a threat, believe me. Radical Islamic terrorism — you look at what’s happening in Europe, it’s crazy. It’s crazy what they’re doing. But we’re not letting them take over our country. We’re not allowing it. They’re trying, but they can’t because of us. You know, the other thing that I want people to know about me is, I’m so loyal to my friends on both sides of the aisle that I’ve never been disloyal to any group. So when it comes to loyalty, that’s for sure. Loyal to friends, loyal to family, loyal to country, loyalty to God, loyalty to your fellow citizens, and most importantly, loyalty to God. That’s why I say it, and I’ll tell you, this election has really shown that. We’ve had some very good results from last night. Some great results. The Democrats have become totally unhinged with their hatred and anger toward everything American. They don’t even try anymore. They’ve gone off the deep end. [Booing] This November, every American needs to vote Republican if they want to protect themselves and keep America safe. Because as long as Republicans control Congress, we’ll continue cutting your taxes, reducing regulations, keeping jobs here, protecting Medicare and Social Security, and defending America’s borders.</p>

</h3>

</section>

<section>

<h3>

<p> The economy is doing way better than anybody thought it was going to do. And I think we’re just about at the bottom of that slope now. But we have a long ways to go. We are not there yet. We’re only one point away from where you would like us to be. So if you’re in this room, and you’ve been watching — I’ve seen some incredible things happen over the last short period of time. We’ve created more jobs since Election Day than any administration has ever created. Think of that. Since election day, we’ve added nearly 4 million new jobs, including 600,000 manufacturing jobs. That’s something nobody even thought possible. It’s incredible. You look at what’s happening with companies moving back into our country. They’re coming back. Chrysler announced they’re opening up three plants in Michigan alone. General Motors said they were restarting production lines all over. They opened them up. Nucor announced plans for two brand-new plants. A lot of them. All sorts of things. Companies are pouring back, and that’s good. It means more products made right here in America. More people employed.</p>

</h3>

</section>

<section>

<h3>

<p> You know, I meet a lot of leaders. They’re great, you know, some of them. Some I like, some I don’t like. What the hell. But every one of them, they come in, they say, Mr. President, congratulations on what you’ve done for this country, what you’ve done with the economy is incredible. We have the hottest economy in the world. Think of it. And they all come in, congratulations, congratulations. African American unemployment has reached its lowest rate ever in recorded history. African American youth unemployment is at an all-time historic low. The Democrats have undermined jobs and wages in the African American community for years, for decades, for a century. Republicans are fighting and delivering every day for the African American community, and they know it. They see it. The support is incredible. Hispanic American and Asian American unemployment rates have also reached all-time historic lows. Thanks to our pledge to America’s workers, in just three months, we have secured commitments from over 160 of the biggest companies in our country to train and recruit more than 6 million workers, 5 percent of our force. Women’s unemployment just fell to 3.6 percent, which is, as you know, you’ve heard me say — I’m trying to get that historic number.</p>

</h3>

</section>

<section>

<h3>

<p>When the radical Democrats get control, they will try to erase every single thing that we’ve achieved. Democrats want to raise your taxes. They want to impose socialism. That’s what they’re doing. They want to take away your Second Amendment. Take away your strong health care that we’re making stronger and stronger every single day. By the way, nobody talks about, but we got rid of the individual mandate, the most unpopular aspect of Obamacare. Where you had the great privilege of paying a fortune in order to pay a fortune to not pay for bad health care. That’s not a very good — right? That’s not a very good thing. Now we got rid of the individual mandate. That’s big. And by the way, we’ve really radically changed that. And we had it beaten. But we didn’t get one Democrat. I always say, we didn’t get somebody vote, which is true. But we didn’t get one Democrat vote, either. Think of it. One Democrat would have repealed and replaced, but we’re essentially doing the same thing anyway. It’s been decimated by us. And now, if you notice, your premiums are going up at a much, much lower clip. </p>

</h3>

</section>

<section>

<h3>

<p>The Democrats are the party of crime. They want to abolish ICE. And they don’t like it. You know, you have these people that come in — I mean, it’s incredible. We had a case recently where we got MS-13 out of our country. They’re not coming back in. It’s amazing. But we’re getting them out. We’re doing things with ICE and Border Patrol and law enforcement. We’ve done more than anybody else. So this election is about safety for all Americans. This election is also about prosperity. For years, you’ve watched as your leaders apologized for America. Now you’re seeing America apologize for them. That’s why I’m asking every citizen from every party, every background and every race, color or creed to reject the Democrat politics of anger and division and embrace an agenda based on hope and optimism rooted deep within America’s heart. Because if there’s one thing we’ve learned over the last few months, it should be this: There really isn’t any place else I’d rather be. There’s no place. [Audience Chants USA] Thank you. God bless you, and god bless America.</p>

</h3>

</section>

<section>

<h3>

<p> Supreme Court Justice Brett Kavanaugh. And I have to say, he’s a great person. He’s an outstanding person, and he will be on the United States Supreme Court. He is going to be great. But we had one of those situations where you don’t want to do it. You just can’t do it, because you’re putting somebody in that could destroy your whole life. So they said, “”We’ll get him into line.”” We got him in. They said — it was like a big deal. It wasn’t even news. The Democrats are so obstructionist. It’s incredible. [Booing] They’ve been trying for years to stop Judge Gorsuch. Now, they’re doing everything they can to try to defeat him. They’re working hard. They’re not giving up. We’re up by one. We’ve never lost before. This election is about security, it’s about jobs, it’s about borders, it’s about taxes, it’s about everything. It’s all about security. That’s why we’re here tonight. That includes our military. Our military has done more than almost any other country in the world to help rebuild Iraq after ISIS. Rebuilding Iraq. In fact, we’ve rebuilt Iraq, as good as it is, better than most countries. Better than, frankly, some countries that were rebuilding their own country. Incredible. Incredible what they’ve done. And, really, if they’d let us alone, we’d probably still be there.</p>

</h3>

</section>

<section>

<h3>

<p>We proudly confirmed the newest member of the United States Supreme Court, Justice Brett Kavanaugh. Right? Wow, was he treated unfairly. That was brutal. That was brutal. I went up to him and I said — when I decided to — I chose him out of the finest, brilliant legal minds, scholars in the whole country. And for 10 years, I heard about Brett Kavanaugh. He’ll be a Supreme — young guy — he’ll be a Supreme Court judge some day. I didn’t know I was going to be president. I didn’t know. [Laughter] I heard he was going to be a Supreme Court judge. I didn’t know I was going to be president. But for years, they talked about him because of his great intellect. And I’ll never forget. Just shows you how life is. We’re in the White House. I say, Judge, because he was a judge — now he’s a justice — but I said, Judge, I’m going to choose you for the United States Supreme Court. Now, Judge, this is going to go so fast, so easy, you’re so perfect. [Laughter] You don’t have a thing wrong. You’re the best student, number one in his class all over the place. You’re the best — a great intellect, a great scholar, your family, your beautiful wife, your beautiful daughters, everyone is so perfect. </p>

</h3>

</section>

<section>

<h3>

<p>Hillary Clinton’s emails and her server. And I said, you know, this is a person that doesn’t know what the hell she’s doing. She was in charge of our nuclear codes. How does she get along with — how do we make it safe for our people to use it? So they gave me a list of 30,000 emails. They said there were 30,000, but there’s probably more. We have no idea what’s on them. But it’s all bad. It’s corrupt. The whole thing is corrupt. You look at it, folks. All these horrible things are going on. Our country has been so badly run. We’ve had one administration after another where they’ve done such an incredible job running things, and then boom, they’re gone. They’re gone forever. [Laughter] They’ve never come back. That’s why I’m here. To continue my great friendship with you, we’re honored tonight by your presence among us. Thank you. God bless you. And, yes, thank you, too, Melania, because you’re doing great. You’re doing great, too. Congratulations. Great job. Thank you. Thank you.</p>

</h3>

</section>

<section>

<h3>

<p>That was a Hillary Clinton deal. She said — no, no. She was secretary of state. [Audience chants Lock her up] She made a statement today, they’re doing a tour, her and her husband or something. And she made a statement, some ridiculous — I said, well, that’s why she lost the election. She just doesn’t get it. She’s never gotten it. But she never got it — so she said, we will create 250,000 jobs. And everybody was excited. Unfortunately, she was right, but it was created in South Korea, not in our country. Just another deal, check it off. Just another horrible deal. We will create 250,000 for — you know if she said for South Korea, at least it would have been honest, right? But we’ve totally renegotiated that. Under the guise of massive tariffs — I said, if you’re not going to do it, we’re just going to put tariffs in, that’s it. And they said, let’s do it. Come on. Let’s do it. Let’s make a decent deal. And we’ve opened it up for your farmers, for your manufacturers. We’ve taken a horrible deal and made it a good deal. </p>

</h3>

</section>

<section>

<h2>

<p>Trump vs. RoboTrump</p>

</h2>

<p><p>The average score is 4/10 (40% correct). This is strong evidence that we’ve now passed a critical point in media manipulation technology. The voice and style of any human can now be cloned. Huge volumes of believable text can be generated effortlessly. Literacy around these dangers is key. Please share this message.</p>

</p>

</section>

<section>

<h3>

<p>Trump vs. RoboTrump</p>

</h3>

<p><p>The average score is 4/10 (40% correct). This is strong evidence that we’ve now passed a critical point in media manipulation technology. The voice and style of any human can now be cloned. Huge volumes of believable text can be generated effortlessly. Literacy around these dangers is key. Please share this message.</p>

</p>

</section>

<section>

<h3>

<p>Trump vs. RoboTrump</p>

</h3>

<p><p>The average score is 4/10 (40% correct). This is strong evidence that we’ve now passed a critical point in media manipulation technology. The voice and style of any human can now be cloned. Huge volumes of believable text can be generated effortlessly. Literacy around these dangers is key. Please share this message.</p>

</p>

</section>

<section>

<h3>

<p>Trump vs. RoboTrump</p>

</h3>

<p><p>The average score is 4/10 (40% correct). This is strong evidence that we’ve now passed a critical point in media manipulation technology. The voice and style of any human can now be cloned. Huge volumes of believable text can be generated effortlessly. Literacy around these dangers is key. Please share this message.</p>

</p>

</section>

<section>

<h3>

<p>Trump vs. RoboTrump</p>

</h3>

<p><p>The average score is 4/10 (40% correct). This is strong evidence that we’ve now passed a critical point in media manipulation technology. The voice and style of any human can now be cloned. Huge volumes of believable text can be generated effortlessly. Literacy around these dangers is key. Please share this message.</p>

</p>

</section>

<section>

<h3>

<p>Trump vs. RoboTrump</p>

</h3>

<p><p>The average score is 4/10 (40% correct). This is strong evidence that we’ve now passed a critical point in media manipulation technology. The voice and style of any human can now be cloned. Huge volumes of believable text can be generated effortlessly. Literacy around these dangers is key. Please share this message.</p>

</p>

</section>

<section>

<h3>

<p>Trump vs. RoboTrump</p>

</h3>

<p><p>The average score is 4/10 (40% correct). This is strong evidence that we’ve now passed a critical point in media manipulation technology. The voice and style of any human can now be cloned. Huge volumes of believable text can be generated effortlessly. Literacy around these dangers is key. Please share this message.</p>

</p>

</section>

<section>

<h3>

<p>Trump vs. RoboTrump</p>

</h3>

<p><p>The average score is 4/10 (40% correct). This is strong evidence that we’ve now passed a critical point in media manipulation technology. The voice and style of any human can now be cloned. Huge volumes of believable text can be generated effortlessly. Literacy around these dangers is key. Please share this message.</p>

</p>

</section>

<section>

<h3>[copy] [copy] </p>

<p>Trump vs. RoboTrump</p>

</h3>

<p><p>The average score is 4/10 (40% correct). This is strong evidence that we’ve now passed a critical point in media manipulation technology. The voice and style of any human can now be cloned. Huge volumes of believable text can be generated effortlessly. Literacy around these dangers is key. Please share this message.</p>

</p>

</section>

<section>

<h3>

<p>Trump vs. RoboTrump</p>

</h3>

<p><p>The average score is 4/10 (40% correct). This is strong evidence that we’ve now passed a critical point in media manipulation technology. The voice and style of any human can now be cloned. Huge volumes of believable text can be generated effortlessly. Literacy around these dangers is key. Please share this message.</p>

</p>

</section>

<section>

<h3>

<p>Trump vs. RoboTrump</p>

</h3>

<p><p>The average score is 4/10 (40% correct). This is strong evidence that we’ve now passed a critical point in media manipulation technology. The voice and style of any human can now be cloned. Huge volumes of believable text can be generated effortlessly. Literacy around these dangers is key. Please share this message.</p>

</p>

</section>

<p>

Did you do poorly? Don’t feel too bad. Remember, on average, our 1,000 respondents did worse than chance, guessing correctly only about 40% of the time. The chart below shows the distribution of correct/incorrect guesses for each category of text, as well as respondents’ overall “score.” To clarify, the survey respondents were quizzed on 20 samples, one real and one AI generated across 10 topical categories: Obamacare, Fake News, Hillary Clinton, Brett Kavanaugh, Democrats, The Economy, Terrorism, North Korea, Putin, The Wall.

WTF – How Can an AI Possibly Write This Well?

So, what’s going on here? Has AI really gotten *that* good at generating realistic text, or does Trump speak in such unintelligible gibberish that he’s extremely easy to mimic?

The truth is likely a little bit of both. While Trump’s rambling style probably makes differentiating between real and fake more difficult than it would be for a more eloquent and talented speaker, today’s new natural language generation AI models have reached a tipping point in their ability to generate fake, real-sounding text.

In September, Salesforce released “CTRL,” a new, state-of-the-art natural language model. This model is larger (likely better), than the full-size GPT-2 model, which Elon Musk’s OpenAI said was “too dangerous to publicly release because of its potential for abuse.”



Then, a few weeks ago, the CTRL model code was updated to allow anyone to “fine-tune” the model. This enabled the ability to train the model on text written by an individual person, which the model could then mimic incredibly well when generating its own made-up text.

Some Populations Are More Vulnerable to Manipulation

Given that RoboTrump could crank out literally millions to billions of words per day, you can imagine how this technology could be used to manipulate opinion. Bots leveraging this tech could spam blogs, social media posts, etc. The volume of realistically generated text in Trump’s (or anyone else’s) style will enable massive disinformation campaigns of staggering volume and reach.

Digging deeper, we were curious if there were demographic or psychographic differences in the accuracy of the survey respondents we had take our quiz. Scarily, there were.

Trump supporters were also more likely to think that AI-generated text was real, showing less overall skepticism than those unsupportive of Trump.

Diving Deeper

Education also appeared to play a role in how believable respondents found the AI-generated text. Here, we see somewhat of a bi-modal distribution, with the least educated being the least skeptical (thinking most everything was real). There also seemed to be a slightly lower level of skepticism among the most educated as well (perhaps they overestimated their ability to differentiate).

Similarly, we see that age also has an impact, with younger folks being less skeptical overall (thinking most everything was real). This was a somewhat counter-intuitive result, since you might think younger people would be more likely to know about emerging AI tech. Still, the result stands, showing even tech-savvy younger people were highly susceptible.

Methodology

A survey and quiz were conducted with 1,000 respondents, where they were tested on their ability to correctly identify AI-generated text (RoboTrump) versus real text (from the real President Trump). Respondents were surveyed on Amazon’s Mechanical Turk Platform. Liberal or Conservative affiliation was based on self report within the categories given as was Trump approval.



The text corpus of real Trump text can be found here. This corpus was used to fine-tune the Salesforce CTRL model, which was then used to generate the fake Trump text.

Quiz respondents were shown each sample of text (both real and fake) in randomized order. They were not shown in a head-to-head fashion.

Bonus*: RoboTrump on Why He Will Win in 2020