Almost four years ago, we started Discord to bring people together around games. Fast forward to today, over 200 million people have used Discord to do just that.

While our goals are focused on bringing people together around games, people also use Discord to talk about their open source projects, their favorite bands, tv shows, and more. Moreover, with all this growth comes some people doing bad things on Discord, which we don’t want in our community.

Two years ago, we began building a team focused on keeping our community safe on Discord (and since then, the Trust and Safety team has grown tremendously). Keeping you safe means making sure real-world harm doesn’t come to you, and no matter who you are, you don’t feel like someone else is intimidating you or harassing you away from being able to participate on Discord.

There’s been a lot of chatter recently from our community wondering how decisions are made on Trust and Safety at Discord. Let’s pull back the curtain and give you a look behind the scenes.

What goes into keeping you safe?

Our Trust and Safety Team currently reviews more than 800 reports every day for violations of our Terms of Service and Community Guidelines, in total handling more than 6,000 reports a week. Those reports vary greatly: sometimes the team may be investigating server raids and NSFW avatars; other times it’s removing deeply disturbing content like gore videos or revenge pornography. We also get reports where a person demands we ban another person for “calling them a poopyhead,” while other times someone is being doxxed or in danger of self-harm and a friend of theirs reaches out to us.

Further complicating things, we also get reports from people who use a combination of false information, edited screenshots, socially engineered situations, or mass reports in an attempt to get a person banned or server deleted. We don’t act without clear evidence, and we gather as much information as possible to make informed, evidence-based decisions that take into account the context behind anything that’s posted. We’ll talk about some of the hard decisions we face later in this blog.

Getting proactive

Answering user requests is usually our highest priority, and something that we do seven days a week. But it’s only part of the job. The Trust and Safety team also works with our engineering team to identify and proactively address potential harm before it ever gets on Discord (like making sure that you don’t get spammed just because you’ve joined a large server).

We also work with local and international agencies on law enforcement requests, many which involve life and death situations. Moreover, we spend a significant amount of time evaluating our policies within the Trust and Safety team, the company as a whole, and talking through potential changes with trusted NGOs to ensure our policies are fair.

That said, no system is perfect. Even if we’re 99.9% confident in the decisions we make, it means that we’ll mess up on 6 requests out of the 6000 every week. To make sure that we get as close to perfection as possible, most actions are reviewed by more than one person. We’re constantly reviewing our work so that we correct and learn from our mistakes and reduce that number even further.

Ultimately, nearly all of the work is done behind the scenes and the majority of people never make a report to Trust and Safety. Unlike many other parts of Discord where success is launching a feature and having millions of people celebrate it, success for Trust and Safety is having fewer people encounter a bad situation on Discord.

Questions that Trust and Safety deals with: case examples

In many situations, what happened is pretty obvious — a person has raided a server to post shock or gore content, they’re posting someone’s private information, or they’re directly threatening to harm someone in real life.

There are other cases where the situation is not so simple. Sometimes, parts of a conversation have been deleted, a slur is used as an act of reclamation, or someone is distributing a hack to a game on Discord that is generally used for cosmetic purposes — but could be used to cheat under certain conditions.

The Trust and Safety team seeks out context to best evaluate what’s going on even when things seem ambiguous. To illustrate the complexity of Trust and Safety’s decision making, see the three scenarios below and the accompanying considerations.

Situation 1: Bilateral Harassment

Two people report each other for bad behavior. One of them clearly started the harassment, while the other escalated it. It starts out with simple insults, but they’re not willing to block each other. Eventually, it escalates to where one threatens to shoot the other’s dog and the other responds by making a sexual threat towards the initial person’s boyfriend. Meanwhile, they’re doing this in a channel that has plenty of other people in it, some of whom are clearly uncomfortable with the escalation, and one of those bystanders in the server writes in too, asking us to do something about it. Finally, not only is the owner of the server not doing anything, they’re actually egging the two people on, further escalating the situation.

Situation 1: Questions Trust and Safety considers

Who should we take action on? Is it the person that started it? The person that escalated to a threat first? Or is it both people, even though each believes the other is at fault? Both people could’ve solved it by blocking the other — should we take any action at all?

How much do we believe each person felt threatened by the other person and thought the only reasonable thing to do was to keep engaging?

Should we also take action on the server owner in some way for egging them on instead of defusing the situation? If we do take action on the server owner, what should it be? A warning? What if one of the server members reports that the owner was privately messaging the two people in order to keep the feud going? Should we punish the server owner instead of the two people?

Situation 2: Ban Appeal Specificity

Someone is banned for messaging people across servers a combination of racial slurs and spam. This person contacts us to appeal their ban. First, we inform them of the specific Community Guideline they have violated. Then, the banned person asks which specific message led to the ban. They insist they’ve done nothing wrong and never violated the Community Guidelines. They claim they’ve been an upstanding citizen, are in twenty different Discord servers, and have a host of users that can speak on their behalf. They insist it’s a false ban based on false reporting. Finally, they enlist some of those people to write in and tell us that the person was maliciously reported. They demand we overturn the ban immediately.

Situation 2: Questions Trust and Safety considers

Is the banned person acting in good faith? Do they legitimately not understand how they violated our Community Guidelines?

Are they simply trying to identify the reporter? Should we provide vague information? Will the banned person continue arguing that whatever messages we have are insufficient?

How do we respond to the supporters that are writing in about this ban? How much information should they get about the situation?

Situation 3: DMs from ‘Discord’

Someone worriedly reaches out to us about a DM they received from another person claiming to be Discord staff. The DM is a warning that their messages are being monitored and that if they continued, the authorities will be contacted. They ask us if the message is real. While the DM isn’t from Discord, the person pretending to be Discord staff contacts us and admits to sending those messages from an alt. The impersonator claims they lied in order to dissuade the initial person from self-harming. When we investigate, it does appear to be true. The initial person was talking about some harmful activity and after receiving the impersonated warning, they’ve completely stopped.

Situation 3: Questions Trust and Safety considers

Impersonating Discord staff is a violation of our Community Guidelines. Most of the time, impersonators engage in extremely harmful behavior and will receive an immediate ban.

In this case, it appears the impersonator has good intentions. Should we take action on the impersonator? Do we just warn them not to do it again? Do we just let it go?

On the other side, should we confirm with the initial person that the message was not from Discord? If we do that, does this encourage them to continue to self-harm?