New court documents reveal TikTok removed 48,000 videos promoting hate speech and violence in five months WIRED

At 8:00pm on February 22, 2019, 28-year-old Venkataraman walked into a south Indian police station and confessed to strangling his friend Vijay to death with a scarf. But police in the state of Tamil Nadu had already been looking for Venkataraman for eight hours. Not for murder, but for uploading a 56-second video to TikTok.

“Fight us if you are a real man, you Dalit dogs. You bastards are worthless in front of us. We’ll butcher you lowlifes,” a drunk Venkataraman rants in the selfie-style video, shot the day before the murder – a copy of which has been viewed by WIRED. He made the video after being encouraged by 18-year-old Vijay.


The video’s aim was to insult people belonging to the Dalit group – the lowest position in India’s Hindu class ladder that’s known as the caste system. Dalits make up 17 per cent of India’s population and a crime against them is committed every 15 minutes.

As Venkataraman’s video went viral, hundreds of angry people from the community that he abused blocked the roads and demanded action. Venkataraman, a bricklayer by profession, belongs to the Vanniyar community – a socially dominant group in Thalavedu.

Read next Back at work? So are burglars. Here's how to keep your home safe Back at work? So are burglars. Here's how to keep your home safe

To subdue the fast-growing crowd, police brought charges of criminal intimidation, intentional insults and inciting violence against Venkataraman. “This calmed the mob a bit,” says P Saravanan, the deputy inspector at Tiruttani police station, which dealt with the case.

Not expecting the massive backlash from the video, Venkataraman and Vijay went on the run. At 10:00am on February 22, still drunk on cheap local hooch, an argument broke out between the pair. They blamed each other for the mess, which quickly escalated into a fight.


“I used the red scarf I carry around my neck to strangle and kill him. I turned his body face down and buried his head,” Saravanan recalls Venkataraman saying. Saravanan was the police official who took Venkataraman’s statement on the night of his arrest. The police report blames Vijay for the creation of the inflammatory video. “He was the one who came up with the idea. He was the reason for the mess,” Venkataraman told Saravanan.

The town of Thalavedu, where Venkataraman and Vijay's video went viral and sparked an angry mob Nilesh Christopher

Vijay’s death went largely unnoticed. It took place in a remote part of India that most of the country’s 1.3 billion people wouldn’t be aware of. However, it demonstrates the rising tide of hate speech filled videos circulating on TikTok and the massive problems the company faces in the country.

Read next Huge fleets of Chinese boats have been hiding in North Korean waters Huge fleets of Chinese boats have been hiding in North Korean waters

During June and July, WIRED identified more than 500 examples of caste-based hate, threats, violence and ridicule attacking different communities within the Tamil language on TikiTok. Users extol the virtues of specific castes and verbally attack local caste-leaders, which can trigger hate crimes.


India’s caste structure is a feudal system of social division stratifying people into hierarchical groups based on their background and work. These include: priests, warriors, farmers/traders, labourers and outcasts. Dalits, formerly the ‘untouchables,’ fall outside the system and are widely persecuted.

Videos found on TikTok include casteist-hate speech posted by users identifying themselves from high castes while celebrating and singing the praises of their communities. These quickly spill into threats of physical violence with members of some communities claiming dominance over other castes.

“We must sever, not the fingers, but the heads of those who dare to lay their hands on us (our community),” one user says in a video, identifying himself as part of the Nadar community. Nadars have traditionally occupied a low status in the caste ladder but have risen through entrepreneurship. This video has been liked by thousands of users, and more than 89 videos have sprung up lip syncing to the speech.

Another user shared a 15-second video responding to posts against his caste. “You Paraiyar dogs stab people in the back. Our courageous lot meet people face to face. I’ll chase each one of you Paraiyar down and slay you,” he says, identifying himself as a member of the Vanniyar community. Paraiyar is often used as a derogatory term to refer to Dalits. The user has several posts threatening Dalits and calls for people from his community to come together on TikTok.

Read next How video hearings broke justice and stripped people of their rights How video hearings broke justice and stripped people of their rights

As WIRED sampled caste-based videos TikTok’s algorithm quickly caught on serving an endless trove of accounts advocating caste supremacy and casteist speech. Many of the videos were discoverable under caste hashtags like #nadar #vanniyar #thevar and #paraiyar – some had been viewed millions of times.

Alongside the hate-fuelled videos, under the caste-linked hashtags, are videos of users lip syncing to popular movies and songs about their pride for their caste. The hashtags are also used by people on TikTok who criticise caste-based conversation. Videos promoting a ‘casteless’ society sit alongside those propagating hate and caste-pride.

WIRED shared a small sample of the videos containing content that potentially breaks TikTok’s community guidelines with the company. “TikTok promotes creativity – hateful or violent content has no place on our platform," a spokesperson said. "Our moderation team flags and removes problematic videos 24/7." The spokesperson adds that the company currently moderates videos in 15 different Indian languages.

A spokesperson did not deny that the company faces challenges – like all other social platforms – with moderating hate speech but said its team had already seen some of the videos presented to it. “The team had identified the videos cited before WIRED contacted us and were in the removal process, but we continuously work to improve our capabilities to do even better,” the spokesperson said.

However, court documents obtained by WIRED, which have not been made public before, show the extent that TikTok’s problem with hate speech in India. During the five months between November 31, 2018 and April 19, 2019, the company removed 36,365 videos that breached its rules on hate speech and religion, and another 12,309 videos deemed to include dangerous behaviour and violence. The amount of hate speech and violence removed from TikTok outnumbers pornographic content by 20 times.

Read next Why defunding the police would actually make our streets safer Why defunding the police would actually make our streets safer

Over the five month period the documents reveal that 677,544 videos were reported to moderators and only one in ten of them (67,431) were removed for breaking TikTok’s guidelines. The number of videos reported accounted for just 0.00006 per cent of all those uploaded and only 1.01 per cent of the accounts reported were banned.

WIRED

TikTok’s wild growth has caused its problems. The app, which boomed in China before entering India in September 2018, allows users to upload 15 second videos set to catchy music, movie dialogues or voice-overs. It was originally called Musical.ly before ByteDance, one of the world's most highly valued startups purchased it for around $1 billion (£820 million) in November 2017 and rebranded it as TikTok.

In India clips and footage related to the country’s vast movie industry are prominent on TikTok alongside teenagers uploading homemade dance videos, comedy skits, and taking part in hashtag challenges – such as the #bottlecapchallenge. They sit alongside influencers who command hundreds of thousands of followers and have helped propel it to an estimated 200m users in India, making the country one of its largest markets.

The swell in user numbers has also made India one of TikTok’s most problematic markets. During the last six months there have been 18 incidents – ten deaths and eight arrests – linked either directly or indirectly to the use of the app.

Read next In the midst of a pandemic, sex workers have nowhere to turn In the midst of a pandemic, sex workers have nowhere to turn

Many of these have been accidents: a 22-year-old injured his spinal cord while attempting a backflip for a video; two people were killed in separate incidents after guns were fired; and one 20-year-old drowned while filming. Other serious incidents have been more disturbing. A mother of two recorded herself drinking poison and died of suicide after her husband claimed she had an additiction to the app. In another case, a man, separated from his wife is said to have stabbed her to death at her workplace for uploading videos on TikTok.

But these are just the gruesome tip of the TikTok iceberg – they are rare among the millions of short videos uploaded every single day. Where the app is facing some of its most difficult troubles is in how it is being used to exacerbate existing tensions in local communities.

Social divisions have run deep in India for centuries. But, the advent of easy-to-use video platforms, messaging apps and low mobile data limits, has seen hate speech targeting marginalised communities thrive. In April, police arrested a 21-year-old for uploading a caste-based video on TikTok that could have sparked communal unrest. And in another case, nine people were arrested for uploading a video inciting caste violence and potential public disorder.

The problems created by TikTok haven’t gone unnoticed by lawmakers either. In April, a court ruling in Tamil Nadu – the region where Vijay died – said the app was spreading ‘pornographic’ and other ‘inappropriate’ content. As a result, Google and Apple took the largely unprecedented step to remove TikTok from their app stores at the request of the Indian court system. TikTok was only reinstated after more than six million videos were purged from the app.

“TikTok is wreaking havoc on societies and villages in Tamil Nadu,” says lawyer K Neelamegam, who argued in favour of the TikTok ban in April. “It is extremely dangerous and playing an active part in disrupting peaceful functioning of our lives”. One Indian state information technology minister has also said TikTok is “degrading culture” and is “inimical to law and order”. And, India’s IT Ministry has threatened TikTok with a further ban for how it handles data and “anti-Indian” activity. It has demanded ByteDance answer a list of 24 questions about how it works.

Read next The 5G coronavirus conspiracy theory just took a really dark turn The 5G coronavirus conspiracy theory just took a really dark turn

TikTok's duet videos are often a basis for abuse videos TikTok

TikTok’s current position in India is inevitable. As the company’s user base has quickly expanded, it has had to react to more problematic videos being posted to its platform. The feeling of community can drive people to act on what they see online.

“We lip sync to songs and put up funny speeches of politicians,” says 19-year-old Arun Kumar, who lives in the Thalavedu village – where 18-year-old Vijay was murdered – and took part in the resulting protest. “But if we see a video dissing our community, we respond with a duet abusing them. We can’t be on the receiving side forever.”

One common video type is users making a ‘duet’ with a 15-second clip urinating into a toilet next to videos of popular political icons. In another split-screen duel, Indian civil rights icon Bhimrao Ramji Ambedkar is shown alongside a clip pointing towards a rack full of worn out footwear, and the user wriggling a slipper as an insult. The video response showing footwear, was to demean Dalit users on TikTok celebrating a national figure.

Duet’s have become an effective way of fanning the caste flames leading to outrage between entire caste groups. “Claims of being a part of a community and exerting superiority is the only thing that makes them feel powerful and popular,” says Kathir Vincent of Evidence, an organisation working to fight caste disrimination and promote social justice in India. “Most TikTok users in small towns work menial jobs and are bored. These videos help show the world (and the other castes) that they have done something heroic.”

Read next Researchers have finally cracked the secret paedophile code Researchers have finally cracked the secret paedophile code

“We have received reports of caste-based videos on TikTok sparking unrest,” says cyber crime cell inspector Senthil Elanthirayan, who is based in Madurai, a city in the Tamil Nadu region. Elanthirayan refused to disclose the number of cases, castes involved, or locations of the reports, as they are still being investigated.

TikTok says it takes hate speech seriously and has introduced safety centers to help educate users. The centers are websites, available in 10 Indian languages, that advise on the app's policies, provide external resources tackling difficult issues and provide guidance around account privacy settings. But critics suggest the company has yet to do enough to tackle some of the issues it faces.

“The problem with Tiktok is that they are not very open to advocacy or engaging with civil society. Not even to the standards of its American counterparts,” says Thenmozhi Soundararajan, executive director of Equality Labs, a South Asian human rights group. “I think they'd rather pay the fines and don't care.”

In many cases it is the lack of moderators with adequate understanding of the sensitivities of the region that lets such content thrive, she adds. TikTok did not answer questions on how many moderators in India it employees but stated it has staff moderating content in 15 different languages. Previous reports say parent company ByteDance employs more than 250 people in India.

The company does, however, point out that it is willing to work with civil society groups. In February, for Safer Internet Day, it worked with Indian NGO the Cyber Peace Foundation to host panels and talks around user safety and distributed posters advising on online safety to schools around the country. It has also introduced two in-app safety quizzes for its India users, the first of which it claimed has been seen by more than five million people.

Read next Coronavirus criminals are touting fake cures and fictitious masks Coronavirus criminals are touting fake cures and fictitious masks

At the moment though, it is unclear just how big TikTok’s hate speech problem is in India. While WIRED researched instances of hate speech in the Tamil language, Soudararajan says the issue is more pervasive. “We have actually reviewed casteist speech on TikTok in different north Indian languages as well.” She says videos relating to the Chamar community – a Dalit caste located in North India – are “very disturbing”. She and her team are currently researching and documenting casteist speech on TikTok in Hindi, Tamil and Telugu and plan to publish their results at the end of the year.

The problem of hateful casteist speech and calls to violence against minorities on social media is not unique to TikTok. As social networks have seen rapid growth in India and other countries where internet penetration has sharply increased, they have all faced similar moderation issues and problems with user behaviour.

A year-long research project conducted by Equality Labs found Facebook in India failed to remove 93 per cent of posts containing hate speech that violated its own rules. Other research has seen Facebook India being overrun by Islamophobic content.

But perhaps the most controversial tech growth in India so far has been WhatsApp. The Facebook-owned messaging app has 400m users in the country and has been intrinsically linked to fake news, misinformation and violence spreading through its network.

Rumours of child abductors have spread through WhatsApp and turned villages into lynch mobs. In one instance, verified by the BBC, a video produced by a child welfare group in Pakistan was mistakenly believed to show a kidnapping in India. It had been edited in such a way to remove the welfare group's message about keeping children safe. After the video was forwarded on WhatsApp, vigilante mobs formed and killed an estimated ten people.

Read next The mysterious final voyage of the Alta, Ireland’s doomed ghost ship The mysterious final voyage of the Alta, Ireland’s doomed ghost ship

In response to the troubles, WhatsApp has limited how messages are forwarded, ran education programmes in the country and appointed a grievance officer. In February, TikTok followed this approach by saying it would hire a ‘nodal officer’ to work in India. The person would work with regulators and handle disputes arising in the country and was appointed earlier this year.

Critics say it is crucial that technology companies expanding in India need to understand the complexities of the country. “All platforms including TikTok lack the cultural competency to enter our market with a clear understanding of the volatile nature of its internal dynamics,” says Soundararajan. “There is not a single platform that has cultural competencies related to caste and religious extremism.”

People in Thalavedu use TikTok Nilesh Christopher

Angry, hate-filled videos, can spill into real world repercussions. In Tamil Nadu, one teenager from Dharmapuri, travelled more than 200 kilometres to face-off with an internet troll speaking ill of his community on TikTok.

“This is for the coward Paraiyan who uses fake ID (to abuse me),” the user posted on June 27. “I have proved that I am a valourous Vanniyar. If you are courageous enough, come meet. I’ll be at the same spot,” he said. The user has built up a 12,000 plus following while uploading videos speaking about the supremacy of his community on TikTok.

Read next This startup is using AI to investigate crime on the high seas This startup is using AI to investigate crime on the high seas

Four hours later, he posted another video celebrating his ‘victory’ proclaiming he had honoured his caste and was leaving because of a no show from his opponent. The video had over a 1,000 likes and half a dozen downloads within minutes.

The pent up anger of these type of videos plays out in the comments where users share mobile numbers and addresses to take the fight offline; meet in person and settle scores. It is unknown how many times users confront each other offline. TikTok says users can report videos which breach its community guidelines through settings within the app – court documents says TikTok responds to requests within 15 minutes on average. It also has a private email address where government and law enforcement officials can make takedown requests for objectionable content.

“When you see so many casteist slurs, it makes it seem acceptable,” Soundararajan says. “Speech like this which normally be whispered in backroom, or in a direct threat in an atrocity prone area just moves it to the center and makes it more acceptable.''

TikTok needs to conduct a civil rights audit, she says, to understand how hate speech spreads on its platform. After severe scrutiny, Facebook committed to an audit in September 2018 to understand censorship and how it may discriminate against minority groups. “This is as dangerous as a snapped live wire dangling, waiting for something more grave to happen,” Vincent says.

During the court hearings surrounding TikTok’s app store ban, the company defended itself by saying it is crucial to free speech and people’s creativity. As the app continues to grow it is likely to face more scrutiny over its decisions.

Read next Forget prisons, the future of punishment will be virtual Forget prisons, the future of punishment will be virtual

Police officer Saravanan worries that applying a broad interpretation to everything on TikTok would provide a veneer that can lead people to overlook potentially harmful side effects. “If you leave a gun on a table, it is partially your (TikTok) responsibility,” he says. “What we have now is leaving a gun chest open.”

However, P Madhava Soma Sundaram, chairman of the Indian Society of Criminology begs to differ. “I will not blame Tiktok. This is a reflection of our society. The things that resonate with new internet users the most. It is very difficult to do adjudicate such content,” he says.

“You cannot defeat casteism and supremacy by meeting it halfway. You have to be clear sighted about it. There are people and politicians who want to push the clash of castes for political mileage. The division can be overcome only through education till then moderation on social media is only meeting the problem half way,” he says.

Updated August 13, 2019 14:06BST: TikTok has appointed a grievance office for India

More great stories from WIRED

💸 How the hell did Uber lose $5bn in three months?

♻️ The truth behind the UK's biggest recycling myths

🤷🏼 How is the internet still obsessed with Myers-Briggs?

🚬 England has an ambitious plan to eradicate smoking by 2030


🕵🏿 It's time you ditched Chrome for a privacy-first web browser

📧 Get the best tech deals and gadget news in your inbox