The ZeroFOX researchers’ software, SNAP_R, can work in two ways. One uses the same artificial intelligence technique, deep learning, used by companies such as Google to make systems that can understand and translate language. It was trained on two million Twitter messages, allowing it to generate realistic-looking tweets of its own.

The system’s second mode is more targeted. It learns how to tweet by looking at an individual’s most recent tweets, and feeds them into an older technique called a Markov chain. It can then generate tweets similar to those written by the target, which a person might click thinking a message was written by a person with similar interests.

SNAP_R can also identify and target the most influential and active people talking about specific topics or using a specific hashtag. It looks for keywords such as “CEO” in a person’s profile, and indicators such as their number of followers. ZeroFOX is releasing a version of the software to help researchers think about the potential for these kinds of attacks and how to defend against them.

ZeroFox software generated these tweets to try to trick people on Twitter.

Mike Murray, vice president of security research at mobile security company Lookout, calls the prospect of using machine learning to automate the process of tricking people online “scary.” But he thinks it will take some time before that kind of approach is used to stage real attacks.

Despite recent progress, the best machine-learning techniques still require specialized expertise, and are far from perfect at generating language. Google is a leader in machine learning and language. But its Inbox app capable of generating responses to e-mails can only suggest short, one-sentence replies, says Murray. “If Google can’t generate more than a sentence, I probably can’t generate a really good phishing e-mail.”

ZeroFOX’s Tully isn’t predicting widespread criminal use of automated spearphishing tomorrow either. But he argues that machine-learning algorithms are getting easier to use, and needn’t perfectly master language to be successful on social media. People using Twitter are expecting to interact with strangers, and to see less-than-polished syntax, he says. “On Twitter the culture is so permissive and you don’t need to have perfect English or grammar.”