Lastly, we hypothesized that the botnet is subdivided into various subgroups dedicated to retweeting specific accounts, thereby triggering different types of retweet cascades. To this end, we examined retweet activity to distinguish patterns of human and bot activity as well as interactions between them that could evince strategies of bot deployment (RO6). This last RO seeks to establish whether bots are deployed and operate in a concerted fashion; or, alternatively, whether competing strategies are employed to overcome the enduring risk that the botnet will be trapped in so-called echo chambers, that is, groups of bots self-referentially communicating with each other.

We subsequently inquired whether the accounts that were swept up in the retweet cascades were also bots themselves. We envisioned that the impact of a botnet may depend on whether it is embedded in a larger network of active users or, alternatively, restricted to a cluster of bots (RO5). Our hypothesis was that the more engagement with human agents the botnet generates, the more likely it is to widen cascades beyond the botnet. In other words, we would expect botnets to exhibit levels of human curation ( Howard & Kollanyi, 2016 ) that testify to their differentiated optimization and their fundamentally cyborg nature. We adopt the latter term to reflect on the close coupling of human agency and computer scripts characterizing bots that disruptively amplifies human communication ( Asenbaum, 2016 ).

RO3 and RO4 probe the impact of bot communication on the Brexit debate. We inspected our data set to determine whether bots could generate larger message cascades compared with active users (RO3). In close connection, we calculated the maximum, minimum, and mean cascade time to ascertain if bots triggered faster cascades than active users in the network (RO4). We thus scrutinized the impact of bots as an upshot of the intensity, reach, and speed of their activity, in addition to examining their network influence and the information dissemination patterns that characterized their actions during the last month of the EU referendum campaign. We relied on such metrics to contrast the activity patterns of bots with regular users, as well as of different types of bots operating within the same botnet.

With this study, we seek to identify a large network of bots that tweeted the Brexit debate and the type of content relayed by these accounts. We explore bot activity with insights into the prevalence of hyperpartisan and polarizing content ( Benkler et al., 2017 ), which constitutes our first RO (RO1). To this end, we began with an inspection of the web pages attached to tweets to identify the domain name of websites sourcing information to bots. Next, we hypothesized that bot activity would be marked by a high-volume posting signature followed by a drop in activity levels characteristic of the life cycle of bots (RO2). To this end, we conducted a time-series analysis and modeled the mean cascade time of bots and active users in the Twitter referendum data, thus distinguishing seasonal patterns and the posting behavior of political bots.

Notwithstanding, the bots recorded in the 2016 U.S. presidential elections were effective information disseminators. They were just as apt as humans at retweeting, republishing a similar volume of content to humans. Similarly, in the case of the EU referendum, the most prolific accounts did “not generate new content but simply retweeted content from other users” ( Howard & Kollanyi, 2016 , 4). While the latter scholars conceded that human agents could achieve similar levels of activity if they confined themselves exclusively to retweeting, Bessi and Ferrara (2016) cautioned that bots could have a debilitating effect on human communication because of their noted capability to disseminate content among human users.

However, the investigation by Bessi and Ferrara (2016) into the 2016 U.S. presidential elections found that overall humans posted more tweets than bots within the period under study. Furthermore, bots seemed inept at interacting with humans, replying (i.e., by quoting one’s Twitter handle preceded by the @ character) primarily to other bots. In their turn, humans were replying to humans more than to bots, another marker that humans and bots operate in largely disconnected subgraphs. These results are somewhat at odds with the political botnet studied by Metaxas and Mustafaraj (2010) , which directed replies at recipients purposefully selected for their partisan interest in the Massachusetts elections, a quarter of whom went on to retweet the automated message they received ( Mustafaraj & Metaxas, 2017 ). The variability of @-mentioning and retweeting practices indicates that bot masters are likely implementing a range of different strategies depending on the political objective set for the botnet.

Secondly, previous research has identified political bots to be tweeting at a rate of seven tweets per minute or 929 tweets in 138 min ( Metaxas & Mustafaraj, 2010 ). In that reported instance, a small botnet comprising nine Twitterbots was set up in 13 min to target accounts of interest by virtue of their previously expressed concern with the 2009 U.S. Massachusetts senate race. Those bots succeeded at starting cascades retweeted by posters whose political alignment resonated with the content of the message ( Metaxas & Mustafaraj, 2010 ). Other studies looking into the same senatorial elections have shown that botnets can raise retweeted URLs to the top of Google search results ( Ratkiewicz, Conover, Meiss, Gonçalves, Patil, et al., 2011 ).

Howard and Kollanyi (2016) approximated the use of political bots during the Brexit referendum by considering users that were extraordinarily active in the Twitter feed associated with the plebiscite. The authors reported that such users accounted for 32% of all Twitter traffic about Brexit. While acknowledging that there is no definitive way of telling which accounts are actual bots, they inferred that the top 10 accounts producing the highest volume of messages (north of 350 tweets) were likely automated. Indeed, other research has described bot activity levels as incessant which on Twitter, specifically, translated into “excessive amounts of tweets” ( Bessi & Ferrara, 2016 ). Nonetheless, user activity alone has been shown to be an unreliable metric for determining the presence of bots, as prolific Twitter posters can tweet abundantly by taking turns managing Twitter accounts and pushing several hundred tweets a day with little to no automation ( Bastos & Mercea, 2016 ; Mercea & Bastos, 2016 ).

The literature investigating bot activity is concerned with the imitation of human activity on social media by computer scripts ( Bessi & Ferrara, 2016 ). These algorithms, also referred to as “social bots,” have been shown to approximate ( Woolley & Howard, 2016 ) and upscale human conduct ( Bessi & Ferrara, 2016 ), often influencing communication exchanges on polarizing topics ( Howard & Kollanyi, 2016 ). Social bots can be deployed in a wide variety of contexts and constitute a growing subfield of communication and political science research, which cautions against their detrimental impact on electoral politics, policy discussions, and deliberation of contentious issues. Indeed, prominent political events such as the referendum on the UK’s membership of the EU or the 2016 U.S. presidential elections were shown to have been susceptible to such automated interference especially on Twitter ( Bessi & Ferrara, 2016 ; Howard & Kollanyi, 2016 ).

The overall tone of the messages was much in line with the context of disaffection with immigration and the cultural backlash spearheaded by older, traditional, and less educated readership of tabloids ( Boykoff, 2008 ). This cultural backlash was strategically leveraged and maximized by populist parties and leaders in order to promote “traditional cultural values and emphasize nationalistic and xenophobia appeals, rejecting outsiders and upholding old-fashioned gender roles” ( Inglehart & Norris, 2016 , p. 30). Our analysis makes no claim as to the veracity (or lack thereof) of the facts reported by the large network of users tweeting the Vote Leave campaign; instead, it seeks to shed light on this unusual user base tweeting the referendum by sourcing hyperpartisan content which is unlikely to fit a normative definition of fake news but that is likely to have played a role in the emerging and loosely defined fake news ecosystem ( Benkler, Faris, Roberts, & Zuckerman, 2017 ).

In what follows, we first ground this study in the literature on political bots—that is, bots deployed in a context of polarized political debate. We subsequently show that the group of Twitter accounts scrutinized in this article often tweeted in a concerted fashion and could well be described as a botnet or a supervised network of zombie agents—be they Internet-connected devices ( Paulson, 2006 ) or social media accounts ( Abokhodair, Yoo, & McDonald, 2015 ). As witnessed in other cases, this pool of accounts was used to automatically replicate posts on Twitter ( Woolley, 2016 ). In contrast to previous research, we identified how the botnet is subdivided into subnetworks dedicated to retweeting content generated either by bots or humans, thereby engineering different retweet cascades. As we show below, the sophistication of the operation deviates considerably from traditional Twitterbots. Common to most accounts in this botnet was the curated replication of content that was both user generated and a reproduction of tabloid journalism. Another important marker of this group was the overwhelming prominence of content associated with or authored by user accounts affiliated with the Vote Leave campaign.

Canvassers have strategically explored these cleavages. The successful Vote Leave campaign was repeatedly and intensely chastised by policy makers and media pundits for stoking anxiety about immigration by making misleading pronouncements about Turkey’s future EU membership. Equally, it was criticized for its disingenuous pledge to boost the National Health Service, a weakened pillar of the industrial welfare state in the UK, by redirecting Great Britain’s EU membership contribution into the service ( Doherty, 2016 ; Swinford, 2016 ). Despite these embattled claims, Vote Leave canvassers effectively relied on data analytics ( Cummings, 2016 ) to capitalize on this ostensible tension between the so-called circles of hardworking families and progressive elites, with a later analysis finding social media activity to be a positive predictor of the outcome of the vote ( Celli, Stepanov, Poesio, & Riccardi, 2016 ).

The political dealignment revealed by the UK EU membership referendum throws deeply engrained ideological leanings into question ( Kriesi & Frey, 2008 ). It also feeds into a context of polarization, alternative media, and hyperpartisanship consistent with emerging patterns of news consumption ( Bastos, 2016 ; Starbird, 2017 ). While the young and well educated are significantly more likely to access news via social media ( Ofcom, 2017 ), an older and less educated readership is traditionally associated with tabloids, which account for a substantive portion of the British press ( Boykoff, 2008 ) and are making inroads on social media platforms ( Newman, Richard Fletcher, Levy, & Nielsen, 2016 ). The partisan realignment observed in Britain, and the context of a highly polarized political climate underpinning the Referendum campaign, offered fertile ground for political actors interested in developing and deploying bots.

The referendum on Britain’s membership of the EU, specifically, was the fruit of more than four decades of efforts to extricate the country from the EU by political actors perennially suspicious of the supranational organization ( Becker, Fetzer, & Novy, 2016 ). The vote was portrayed as a milestone in the political life of the country ( Asthana, Quinn, & Mason, 2016 ) that opened up fault lines largely at odds with the traditional alignment of British political parties ( Becker et al., 2016 ). The observed political realignment foregrounds a sociocultural cleavage between young and well-educated sections of the population who embrace progressive postmaterialist values of equality, human rights, environmental protection, and a greater tolerance of immigrants; and on the other hand, an older, less educated demographic who witnessed both a decline in its material conditions and a gradual erosion of traditional values associated with industrial societies ( Inglehart & Norris, 2017 ).

Concerns about the activity of bots and sock puppets in the context of the United Kingdom (UK) European Union (EU) referendum were articulated in the press ( Silva, 2016 ) and academia ( Shorey & Howard, 2016 ), with researchers cautioning against the automation of political communication and the possible distortion of vital processes at the heart of contemporary liberal democracies, chief among which are competitive elections ( Woolley & Howard, 2016 ). The scale of bot deployment and its effect on information diffusion are topical concerns ( Bessi & Ferrara, 2016 ), with previous research reporting that bots are often deployed in contexts of polarized political discussion ( Ferrara, Varol, Davis, Menczer, & Flammini, 2016 ). We seek to contribute to this growing body of scholarship by scrutinizing a large network of bots that operated during the Brexit debate. We explored the tactics employed by bot masters deciding which tweets are retweeted and by which subgroup of accounts linked to the botnet.

A sock puppet account is a false online identity used to voice opinions and manipulate public opinion while pretending to be another person. The term draws from the manipulation of hand puppets using a sock and refers to the remote management of online identities to spread misinformation, promote the work of one individual, endorse a given opinion, target individuals, and challenge a community of users ( Zheng, Lai, Chow, Hui, & Yiu, 2011 ). Sock puppet accounts are often automatic posting protocols (i.e., bots) operating under a fictitious identity and as such they breach the terms of service of social networking sites like Facebook and Twitter. The administration and deployment of bots and sock puppet accounts are largely centralized and rely on trivial computational routines that allow users and organizations to control substantial subcommunities across any given social media platform ( Kumar, Cheng, Leskovec, & Subrahmanian, 2017 ).

The research data examined in this study were collected via the publicly accessible Twitter Streaming and REST APIs. Although the information collected for this study is public, there are important ethical issues associated with harvesting public Twitter accounts ( Zimmer, 2010 ). Twitter profiles set to private were removed from our pool of users, and no private information was examined in the analysis. While we have looked to preserve users’ rights and interests, we ultimately decided to disclose the Twitter handles examined in this study whenever there was a reasonable level of certainty that we were dealing with Twitterbots, to which ethical considerations of privacy are immaterial. We also considered the potential sensitivity of some of the tweets examined in this study, but anonymizing the seeding accounts would impinge on our ability to understand the scope of the botnet and the strategies adopted by bot masters. Lastly, we have considered the ethical obligation not to display deleted Tweets, but we believe the content analyzed in this study is of public and scholarly interest and that ethical considerations regarding users’ rights to not have their deleted tweet made public are immaterial in the context of large botnets participating in politically contentious debates.

Unfortunately, it is not currently possible to rebuild every step of the retweet cascade, as each retweet includes only a reference to the original message, so that if User C retweets User B who has previously retweeted User A, we can only establish that User A was retweeted by User C, with the intermediary steps of the cascade remaining unknown. As such, we cannot account for independent entry points that might have influenced the cascade ( Cheng, Adamic, Dow, Kleinberg, & Leskovec, 2014 ). However, given that each retweet includes a unique identifier arranged chronologically from the original tweet to the most recent retweet, we can rebuild cascades from the seed message to the retweets that have cascaded from that original content. Similarly, we rely on the time stamp attached to each tweet to estimate the variable time-to-retweet, calculated as the time elapsed between the original tweet and the i th retweet for cascade of size S .

The metrics used in this study to identify bot accounts are informed by the relevant literature and include detailed profile information, presence or absence of geographical metadata (or propensity to post using web clients), retweet to tweets ratio, @-mention to tweet ratio, activity level, followers to followees ratio, account creation date, and absence of known words in the username ( Table 2 ). Positive predictors of bot activity are shown in Table 2 and include tweets to user (tw2user), mean tweet to retweet (tw2rtMean), common words in the username (commonWords), use of web interface to relay content (webClient), ratio of outbound to inbound @-mentions (mentionOut2In), ratio of inbound to outbound retweets (retweetIn2Out), account creation date (newAccount), retweet reciprocity (rtReciprocity), and retweet cascade mean time (ccdMeanTime). For the purposes of this study, we contrast retweeting behavior observed in this group against the larger set of accounts we refer to as active users (as opposed to deactivated or recycled users, defined underneath). Retweet and @-mention behavior are defined as A → B when B retweets A and A → B when A mentions B (thus following the directionality of the information flow). While previous studies have explored Twitter cascades by tracking the diffusion of URLs ( Bakshy, Hofman, Mason, & Watts, 2011 ) and hashtags ( González-Bailón, Borge-Holthoefer, Rivero, & Moreno, 2011 ), we rely on retweets to inspect user-to-bot and bot-to-bot cascade composition.

Previous research found the frequentist approach to user activity alone to be an unreliable metric to determine the presence of bots, as prolific Twitter posters can tweet abundantly by taking turns and pushing several hundred tweets a day with little to no automation ( Bastos & Mercea, 2016 ; Mercea & Bastos, 2016 ). Consequently, in the attempt to differentiate between bots and high-volume posters, we analyzed several metrics of user activity in addition to the temporal posting patterns of potential bots. This composite analysis allowed us to ascertain whether their activity endured over time; or conversely, if there was a notable drop in activity levels that typifies the bot lifecycle in the wake of the EU referendum.

We queried the Twitter Streaming API to monitor 39 Twitter hashtags clearly associated with the referendum campaign from April to August 2016 (e.g., #voteleave, #voteremain, #votein, #voteout, #leaveeu, #bremain, #strongerin, #brexit, and #euref). For the purposes of this study, we focus on the 2-week period before and after the referendum vote, that is, June 10 to July 10, 2016. In this interval, we collected approximately 10M tweets associated with the referendum. We subsequently retrieved the profile of over 800K unique users that appeared in our data set and relied on thresholding and filtering approaches to disentangle real users from bots. The combination of methods reported in the literature ( Subrahmanian et al., 2016 ; Varol, Ferrara, Davis, Menczer, & Flammini, 2017 ) allowed us to identify a large group of bots whose accounts had been deactivated by the bot master or blocked/removed by Twitter in the aftermath of the referendum. We relied on the implementation of extended regular expression in R (2014) to identify the campaign associated with tweets and the libcurl implementation ( Temple Lang, 2016 ) to retrieve the web page title of URLs embedded in tweets (when available).

Results

From a total of 794,949 Twitter profiles that tweeted the Vote Leave and Vote Remain campaigns, we managed to identify the location of 60% of them (482,193) by triangulating information from geocoordinates embedded in tweets (i.e., reverse geocoding), geographic information tweeted by the users, and information that appeared in their profiles. From this cohort of users, only 30,122 users were identified as based in the UK, a smaller population than the set of 40,031 accounts that have been deactivated, removed, blocked, set to private, or whose username was altered after the referendum. This latter group of accounts represents 5% of all users that tweeted the referendum and is divided as follows: 66% or 26,538 were users who have changed their username since the referendum but remained active on Twitter (designated hereafter as repurposed or recycled accounts); 34% or 13,493 accounts were suddenly blocked or removed themselves from Twitter (deleted accounts).

Although repurposed/recycled accounts conspicuously interacted with deleted Twitterbot accounts, the focus of this study lies with the latter cohort. Notwithstanding, common to these two subgroups is the predominance of retweeted content that disappeared from the Internet shortly after the referendum. Another commonality is the notable support for the Leave campaign, measured by the relative frequency of key words and hashtags associated with each of the campaigns. While the ratio of messages using hashtags that supported the Leave and Remain campaigns was 31% and 11% for the entire network, recycled and removed accounts combined tweeted the referendum hashtags to a ratio of 37% and 17% (or 2,434,077 and 840,726 vs. 30,947 and 14,390 tweets for each of the campaigns, respectively). In what follows, we disentangle these groups to finally concentrate on a set of 13,493 accounts identified as bots.

Hyperpartisan and Perishable News By annotating tweets using textual markers such as hashtags and key words associated with the Leave and Remain campaigns, we found that the proportion of tweets supporting the Vote Leave campaign in the pool of removed accounts was yet higher, at 41% compared with 31% for active users, with the proportion of neutral tweets also being higher in the latter. Slogans associated with the Vote Leave campaign were also significantly more likely to have been tweeted by this pool of accounts in a ratio of 8:1. This subset of removed accounts was considerably more active in the period leading up to the referendum, with an average of 4.4 messages compared with 3.9 for the rest of the population ( x ¯ = 4.44; σ = 33.3 and x ¯ = 3.99; σ = 74.2, respectively); and also less active in the wake of the vote with an average of 2.4 tweets compared with 2.6 for the global population ( x ¯ = 2.42; σ = 9.0 and x ¯ = 2.61; σ = 63.2, respectively). Upon attempting to retrieve the web pages (RO1) tweeted by recycled and removed accounts, we found that most tweeted URLs (55%) no longer exist, cannot be resolved, or link to either a Twitter account or a web page that no longer exists. Nearly one third (29%) of the URLs link to Twitter statuses, pictures, or other multimedia content that is no longer available and whose original posting account has also been deleted or blocked, a marker of the perishable nature of digital content at the center of political issues (Walker, 2015). From this total, 1% of all links was directed to user @brndstr, one of the few accounts appearing in the communication network of recycled accounts that remains active under the same username. This account is managed by Dubai-based “Bot Studio for Brands,” a company specialized in providing bots for social media campaigns. A closer inspection of the accounts sourcing content to the pool of recycled and removed accounts reveals the markedly short shelf life of user-generated content. These are Twitter accounts invested in spreading dubious news stories sourced from a circuit of self-referencing blews (Gamon et al., 2008): a combination of far-right weblog such as WorldTribune.com and traditional tabloid media such as express.co.uk. However, the few web pages we managed to retrieve indicate that the content tweeted by this large pool of recycled and removed accounts does not conform with the notion of fake news designating news stories that are intentional, misleading half-truths, and/or outright lies (Benkler et al., 2017). Instead, the content is in line with a form of storytelling that blurs the line between traditional tabloid journalism and user-generated content, which is often anonymous, fact-free, and places a strong emphasis on simplification and spectacularization (Rowe, 2011). User-generated content takes the lion’s share of hyperlinks tweeted by recycled and removed accounts. The content is often presented as a professionally looking newspaper by resorting to content curation services such as paper.li and is likely to include Twitter multimedia (e.g., Twitter’s native multimedia sharing service twimg.com). Similarly, the few links that remained accessible 6 months after the referendum can hardly be described as fake news. The hyperlinked material is rich in rumors, unconfirmed events, and human-interest stories with an emotional and populist appeal that resembles tabloid journalism, except for the added complexity that audiences play a pivotal role in curating and distributing the content. The sources we managed to inspect, though not representative of the much larger universe of content tweeted by this population of users, and that unfortunately has mostly vanished from Twitter, is much akin to hyperpartisan tabloid journalism, with a topical emphasis on highly clickable, shareable, and human-interest driven stories (Bastos, 2016). Table 1 summarizes the URLs tweeted by this cohort of users. Table 1. Weblinks Tweeted by Deleted Accounts. View larger version Although 17% of weblinks point to Twitter accounts that are still active, a random sample shows that the original message is frequently no longer available, thus preventing any determination of the nature of the content originally tweeted. A notable example is the tweet ID 740138870092750848, which generated a cascade of several hundred retweets and whose posting user is still active. Although the user account seeding the cascade remains active, the original tweet has been removed (together with the relevant retweet cascade). With Internet Archive having no record of this specific tweet, it is no longer possible to know what the original image conveyed. The scale of deleted content applies both to weblinks tweeted by this population as well as to user accounts, a worrying development given the importance and contentious nature of the referendum (Walker, 2015).

Brexit Botnet Turning to the removed user accounts, we relied on metrics discussed in the relevant literature (Bessi & Ferrara, 2016; Ratkiewicz, Conover, Meiss, Gonçalves, Patil, et al., 2011) to determine whether the pool of deleted accounts comprised a large network of Twitterbots. Upon inspecting the account creation date, we found that 83% of accounts in the botnet had been created in the previous 2 years compared with 43% for the subset of active accounts and 48% for accounts that ended up being recycled. We interpreted the result as an important indication of bot activity. The highest rate of tweet to retweet was found in the campaign accounts @iVoteStay and @iVoteLeave, with a retweet to tweet ratio of 90% and a total number of retweets of nearly 60K messages. These accounts did author original content though, and we do not feel confident they can be classified as bots despite the extraordinary high levels of activity and the high likelihood that some form of automation was used to relay content. When analyzing retweet rate across groups, we found that the baseline for accounts that remained active after the referendum was of one retweet to each three tweets ( x ˜ = 0.33 and x ¯ = 0.45), while the ratio for accounts that changed their usernames is twice as high ( x ˜ = 0.61 and x ¯ = 0.54). The group that significantly deviates from this baseline is the set of accounts removed after the referendum (i.e., the botnet). For such accounts, the retweet rate is of 1 retweet for every tweet, with 54% of accounts never having authored any tweet related to the referendum (i.e., only retweets were registered for these accounts, another marker of bot activity). For this group of accounts, the median tweet to retweet rate is 1 ( x ¯ = 0.6302). Table 2 shows the metrics used to classify this subset of accounts as bots in a network of 13,493 Twitterbots that tweeted a total of 63,797 messages. The variables in the table indicate tweets per user, retweet to tweet ratio, incidence of known words in usernames, propensity to post using web clients, @-mention indegree to outdegree, retweet indegree to outdegree, account creation date, outdegree and indegree transitivity, retweet reciprocity, modularity score, mean and maximum cascade size, number of cascades triggered, share of triggered cascades, and cascade mean time. Table 2. Metrics of Automated Activity for Active Users, Recycled Accounts, and Bots. View larger version