Shutterstock

Researchers from Stanford and Cornell Universities have developed an algorithm that only needs to observe five to ten online posts to predict whether a member of an online community needs banning.

The study, funded by Google, looked at the behaviour of internet trolls over time to see what eventually resulted in them being banned. Not only did trolls post more frequently and exhibit poor spelling and grammar, which degraded over time, but communities tended to serve as incubators that fostered their antisocial behaviour.


Three online communities with a combined 1.7 million users and 40 million posts were observed over the course of 18 months for the study. They were from the comment/forum sections of CNN, politics site Breitbart and games site IGN.

In order to see if it was possible to predict whether users would be permanently banned from communities in the future, the researchers compared the behaviour of users who were banned against the behaviour of those who were never banned, discarding the behaviour of those who faced temporary bans.

Read next Audi e-tron Sportback 55 quattro S Line review: a flashy EV light show Audi e-tron Sportback 55 quattro S Line review: a flashy EV light show

Users who ended up being permanently banned from sites demonstrated a very clear set of behaviours and characteristics. From the get-go, the quality of their posts was far worse than those by other users, and continued to degrade over time. As well being "harder to understand according to standard readability metrics", posts written by trolls tended to be more likely to contain "language that may stir conflict", including negative words and profanities.

The way trolls engaged within the communities was also characteristically different. They were far more likely to concentrate posts in individual threads and also posted far more frequently. Future banned users on CNN were found to post 264 times before they were banned. Over a similar time period the average community member only posted 22 posts. They also received more replies than the average user, which according to the researchers suggested "they might be successful in luring others into fruitless, time-consuming discussions".


Harsh community feedback to trolling was only likely to further exacerbate anti social behaviour, the study claimed. This led the researchers to believe "that communities may play a part in incubating antisocial behaviour".

Antisocial behaviour did vary between different communities. On Breitbart and IGN, trolls were more likely to reply to others' posts, while on CNN they were more likely to start new discussions.

The study noted that each of the platforms implemented mechanisms to try and discourage antisocial behaviour, including "community moderation, up- and down-voting, the ability to report posts, mute functionality, and more drastically, completely blocking users' ability to posts."


An algorithm that identifies abusive users may be useful to community moderators, but the study concluded that it could not be solely relied upon to troll-proof communities. One in five of the users picked out by the algorithm as a future troll were falsely identified as such and never banned. "A more fine-grained labelling of users (perhaps through crowdsourcing), may reveal a greater range of behaviour," the study noted.

The proposed system also does not tackle more deceptive forms of antisocial behaviour, as exhibited by those who surreptitiously provoke arguments while maintaining a normal appearance. Instead it focuses on the kind of overtly inflammatory behaviour that leads to users being banned.

Banning may not even be the most effective way to discourage trolls, the researchers added. "While we present effective mechanisms for identifying and potentially weeding antisocial users out of a community, taking extreme action against small infractions can exacerbate antisocial behaviour," they explained. "A better response may instead involve giving antisocial users a chance to redeem themselves."