The social network has pledged to work harder to identify and remove disturbing content – but doing so can take a psychological toll

Facebook is hiring moderators. But is the job too gruesome to handle?

Ever wanted to work for Facebook? Mark Zuckerberg has just announced 3,000 new jobs. The catch? You’ll have to review objectionable content on the platform, which has recently hosted live-streamed footage of murder, suicide and rape.



In his announcement, Zuckerberg revealed that the company already has 4,500 people around the world working in its “community operations team” and that the new hires help improve the review process, which has come under fire for both inappropriately censoring content and failing to remove extreme content quickly enough. Just last week the company left footage of a Thai man killing his 11-month-old daughter on Facebook Live on the platform for a whole day.

Instead of scrutinizing content before it’s uploaded, Facebook relies on users of the social network to report inappropriate content. Moderators then review reported posts – hundreds every shift – and remove them if they fall foul of Facebook’s community standards. Facebook does not allow nudity (including female, but not male, nipples), hate speech or glorified violence.

Facebook refused to offer any further detail about the nature of the new roles, including the job description, whether they would be employees or contractors, or where they would be based. However, one thing is clear: such work is grueling and, experts say, can lead to psychological trauma.

Live and death: Facebook sorely needs a reality check about video Read more

“People can be highly affected and desensitized. It’s not clear that Facebook is even aware of the long term outcomes, never mind tracking the mental health of the workers,” said Sarah T Roberts, an information studies professor from UCLA who studies large-scale moderation of online platforms.

If these roles are outsourced to contractors, it’s going to be even harder for Facebook to monitor moderators’ working conditions and mental health, she said.

In January, moderators in similar roles at Microsoft sued the company, alleging that exposure to images of “indescribable sexual assaults” and “horrible brutality” resulted in severe post-traumatic stress disorder (PTSD). Microsoft disputes the claims.

Repeated exposure to extreme content can lead to “secondary trauma”, which is a condition similar to PTSD, but the witness is looking at images of what happened rather than being traumatized themselves.

“The symptoms are similar and include hypervigilance,” said Lisa Perez, associate professor of psychology at Minnesota State University, who has studied secondary traumatic stress and burnout among law enforcement investigators exposed to disturbing media images.

Investigators of child pornography, for example, will be extremely suspicious of people interacting with children, particularly their own. Viewing disturbing sexual images can also create problems in sexual and marital relationships.

For law enforcement, there’s a flip side: the knowledge that they are working to get bad people off the street. “While they have this emotional exhaustion and burnout, they tend to be very satisfied with their jobs and find them to be meaningful,” Perez said.

It is not clear how Facebook moderators would gain the same satisfaction. Although deeply disturbing to watch, it is not illegal to share beheading videos on the platform. Removing the content does little to address the perpetrator of the depicted act, although it does shield others from seeing it.

Moderators may also be required to make judgements about suicidal individuals. Facebook is testing artificial intelligence as a way of detecting comments indicative of suicidal thoughts, but they are still reviewed by humans.

“We’ve had situations where we’ve literally talked people off the edge of buildings,” said Peter Friedman, the chief executive of LiveWorld, which provides moderation services to major brands. Anyone handling such interactions needs to be highly trained, with clear processes to follow in order to escalate incidents in cases of emergency, he added.

I'm an ex-Facebook exec: don't believe what they tell you about ads Read more

Beyond the psychological toll moderators face, there’s an enormous burden of judgement: they have to distinguish between child pornography and iconic Vietnam war photos, between the glorification of violent acts and the exposure of human rights abuses. Decisions must be nuanced and culturally contextualized or Facebook will be accused of infringing freedom of speech.

“You can have a situation where the words that are being typed by the end user are exactly the same but one is a casual joke and the other is a serious thing that needs escalation,” said Friedman. “This requires intuition and human judgement. Algorithms can’t do that.”

Given the immense challenges associated with these roles, Facebook will need to keep a close eye on the process and ensure that moderators have the training and psychological support to handle disturbing content.

The Microsoft lawsuit outlines some specific recommendations for how moderators should be treated, including rotations out of the program, mandatory meetings with a psychologist with specialist training to recognize trauma and a spousal wellness program.

A Facebook spokeswoman said that the company recognizes that the work can often be difficult and that every person is offered psychological support and wellness resources. The company said it also has a program in place designed to support people in these roles, which is evaluated annually. Facebook offered no further details on the nature of the support or training.