Joint Ministerial foreword

The internet is an integral part of everyday life for so many people. Nearly nine in ten UK adults and 99% of 12 to 15 year olds are online. As the internet continues to grow and transform our lives, often for the better, we should not ignore the very real harms which people face online every day.

In the wrong hands the internet can be used to spread terrorist and other illegal or harmful content, undermine civil discourse, and abuse or bully other people. Online harms are widespread and can have serious consequences.

Two thirds of adults in the UK are concerned about content online, and close to half say they have seen hateful content in the past year. The tragic recent events in New Zealand show just how quickly horrific terrorist and extremist content can spread online.

We cannot allow these harmful behaviours and content to undermine the significant benefits that the digital revolution can offer. While some companies have taken steps to improve safety on their platforms, progress has been too slow and inconsistent overall. If we surrender our online spaces to those who spread hate, abuse, fear and vitriolic content, then we will all lose.

So our challenge as a society is to help shape an internet that is open and vibrant but also protects its users from harm. The UK is committed to a free, open and secure internet, and will continue to protect freedom of expression online. We must also take decisive action to make people safer online.

This White Paper therefore puts forward ambitious plans for a new system of accountability and oversight for tech companies, moving far beyond self-regulation. A new regulatory framework for online safety will make clear companies’ responsibilities to keep UK users, particularly children, safer online with the most robust action to counter illegal content and activity.

This will be overseen by an independent regulator which will set clear safety standards, backed up by reporting requirements and effective enforcement powers.

Although other countries have introduced regulation to address specific types of harm, this is the first attempt globally to address a comprehensive spectrum of online harms in a single and coherent way.

The UK’s future prosperity will depend heavily on having a vibrant technology sector. Innovation and safety online are not mutually exclusive; through building trust in the digital economy and in new technologies, this White Paper will build a firmer foundation for this vital sector.

As a world-leader in emerging technologies and innovative regulation, the UK is well placed to seize these opportunities. We want technology itself to be part of the solution, and this White Paper proposes measures to boost the tech-safety sector in the UK, as well as measures to help users manage their safety online.

We believe the approach in this White Paper can lead towards new, global approaches for online safety that support our democratic values, and promote a free, open and secure internet; and we will work with other countries to build an international consensus behind it.

Online safety is a shared responsibility between companies, the government and users. We would encourage everyone to take part in the consultation that accompanies this White Paper, and work with us to make Britain the safest place in the world to be online.

Rt Hon Jeremy Wright MP, Secretary of State for Digital, Culture, Media & Sport

Rt Hon Sajid Javid MP, Home Secretary

Executive summary

1. The government wants the UK to be the safest place in the world to go online, and the best place to start and grow a digital business. Given the prevalence of illegal and harmful content online, and the level of public concern about online harms, not just in the UK but worldwide, we believe that the digital economy urgently needs a new regulatory framework to improve our citizens’ safety online. This will rebuild public confidence and set clear expectations of companies, allowing our citizens to enjoy more safely the benefits that online services offer.

The problem

2. Illegal and unacceptable content and activity is widespread online, and UK users are concerned about what they see and experience on the internet. The prevalence of the most serious illegal content and activity, which threatens our national security or the physical safety of children, is unacceptable. Online platforms can be a tool for abuse and bullying, and they can be used to undermine our democratic values and debate. The impact of harmful content and activity can be particularly damaging for children, and there are growing concerns about the potential impact on their mental health and wellbeing.

3. Terrorist groups use the internet to spread propaganda designed to radicalise vulnerable people, and distribute material designed to aid or abet terrorist attacks. There are also examples of terrorists broadcasting attacks live on social media. Child sex offenders use the internet to view and share child sexual abuse material, groom children online, and even live stream the sexual abuse of children.

4. There is also a real danger that hostile actors use online disinformation to undermine our democratic values and principles. Social media platforms use algorithms which can lead to ‘echo chambers’ or ‘filter bubbles’, where a user is presented with only one type of content instead of seeing a range of voices and opinions. This can promote disinformation by ensuring that users do not see rebuttals or other sources that may disagree and can also mean that users perceive a story to be far more widely believed than it really is.

5. Rival criminal gangs use social media to promote gang culture and incite violence. This, alongside the illegal sale of weapons to young people online, is a contributing factor to senseless violence, such as knife crime, on British streets.

6. Other online behaviours or content, even if they may not be illegal in all circumstances, can also cause serious harm. The internet can be used to harass, bully or intimidate, especially people in vulnerable groups or in public life. Young adults or children may be exposed to harmful content that relates, for example, to self-harm or suicide. These experiences can have serious psychological and emotional impact. There are also emerging challenges about designed addiction to some digital services and excessive screen time.

Our response

7. This White Paper sets out a programme of action to tackle content or activity that harms individual users, particularly children, or threatens our way of life in the UK, either by undermining national security, or by undermining our shared rights, responsibilities and opportunities to foster integration.

8. There is currently a range of regulatory and voluntary initiatives aimed at addressing these problems, but these have not gone far or fast enough, or been consistent enough between different companies, to keep UK users safe online.

9. Many of our international partners are also developing new regulatory approaches to tackle online harms, but none has yet established a regulatory framework that tackles this range of online harms. The UK will be the first to do this, leading international efforts by setting a coherent, proportionate and effective approach that reflects our commitment to a free, open and secure internet.

10. As a world-leader in emerging technologies and innovative regulation, the UK is well placed to seize these opportunities. We want technology itself to be part of the solution, and we propose measures to boost the tech-safety sector in the UK, as well as measures to help users manage their safety online.

11. The UK has established a reputation for global leadership in advancing shared efforts to improve online safety. Tackling harmful content and activity online is one part of the UK’s wider ambition to develop rules and norms for the internet, including protecting personal data, supporting competition in digital markets and promoting responsible digital design.

12. Our vision is for:

● A free, open and secure internet

● Freedom of expression online

● An online environment where companies take effective steps to keep their users safe, and where criminal, terrorist and hostile foreign state activity is not left to contaminate the online space

● Rules and norms for the internet that discourage harmful behaviour

● The UK as a thriving digital economy, with a prosperous ecosystem of companies developing innovation in online safety

● Citizens who understand the risks of online activity, challenge unacceptable behaviours and know how to access help if they experience harm online, with children receiving extra protection

● A global coalition of countries all taking coordinated steps to keep their citizens safe online

● Renewed public confidence and trust in online companies and services

Clarity for companies

13. Increasing public concern about online harms has prompted calls for further action from governments and tech companies. In particular, as the power and influence of large companies has grown, and privately-run platforms have become akin to public spaces, some of these companies now acknowledge their responsibility to be guided by norms and rules developed by democratic societies.

14. The new regulatory framework this White Paper describes will set clear standards to help companies ensure safety of users while protecting freedom of expression, especially in the context of harmful content or activity that may not cross the criminal threshold but can be particularly damaging to children or other vulnerable users. It will promote a culture of continuous improvement among companies, and encourage them to develop and share new technological solutions rather than complying with minimum requirements.

15. It will also provide clarity for the wide range of businesses of all sizes that are in scope of the new regulatory framework but whose services present much lower risks of harm, helping them to understand and fulfil their obligations in a proportionate manner.

A new regulatory framework for online safety

16. The government will establish a new statutory duty of care to make companies take more responsibility for the safety of their users and tackle harm caused by content or activity on their services.

17. Compliance with this duty of care will be overseen and enforced by an independent regulator.

18. All companies in scope of the regulatory framework will need to be able to show that they are fulfilling their duty of care. Relevant terms and conditions will be required to be sufficiently clear and accessible, including to children and other vulnerable users. The regulator will assess how effectively these terms are enforced as part of any regulatory action.

19. The regulator will have a suite of powers to take effective enforcement action against companies that have breached their statutory duty of care. This may include the powers to issue substantial fines and to impose liability on individual members of senior management.

20. Companies must fulfil the new legal duty. The regulator will set out how to do this in codes of practice. If companies want to fulfil this duty in a manner not set out in the codes, they will have to explain and justify to the regulator how their alternative approach will effectively deliver the same or greater level of impact.

21. Reflecting the threat to national security or the physical safety of children, the government will have the power to direct the regulator in relation to codes of practice on terrorist activity or child sexual exploitation and abuse (CSEA) online, and these codes must be signed off by the Home Secretary.

22. For codes of practice relating to illegal harms, including incitement of violence and the sale of illegal goods and services such as weapons, there will be a clear expectation that the regulator will work with law enforcement to ensure the codes adequately keep pace with the threat.

23. Developing a culture of transparency, trust and accountability will be a critical element of the new regulatory framework. The regulator will have the power to require annual transparency reports from companies in scope, outlining the prevalence of harmful content on their platforms and what counter measures they are taking to address these. These reports will be published online by the regulator, so that users and parents can make informed decisions about internet use. The regulator will also have powers to require additional information, including about the impact of algorithms in selecting content for users and to ensure that companies proactively report on both emerging and known harms.

24. The regulator will encourage and oversee the fulfilment of companies’ existing commitments to improve the ability of independent researchers to access their data, subject to appropriate safeguards.

25. As part of the new duty of care, we will expect companies, where appropriate, to have effective and easy-to-access user complaints functions, which will be overseen by the regulator. Companies will need to respond to users’ complaints within an appropriate timeframe and to take action consistent with the expectations set out in the regulatory framework.

26. We also recognise the importance of an independent review mechanism to ensure that users have confidence that their concerns are being treated fairly. We are consulting on options, including allowing designated bodies to make ‘super complaints’ to the regulator in order to defend the needs of users.

27. Ahead of the implementation of the new regulatory framework, we will continue to encourage companies to take early action to address online harms. To assist this process, this White Paper sets out high-level expectations of companies, including some specific expectations in relation to certain harms. We expect the regulator to reflect these in future codes of practice.

28. For the most serious online offending such as CSEA and terrorism, we will expect companies to go much further and demonstrate the steps taken to combat the dissemination of associated content and illegal behaviours. We will publish interim codes of practice, providing guidance about tackling terrorist activity and online CSEA later this year.

The companies in scope of the regulatory framework

29. We propose that the regulatory framework should apply to companies that allow users to share or discover user-generated content or interact with each other online.

30. These services are offered by a very wide range of companies of all sizes, including social media platforms, file hosting sites, public discussion forums, messaging services and search engines.

31. The regulator will take a risk-based and proportionate approach across this broad range of business types. This will mean that the regulator’s initial focus will be on those companies that pose the biggest and clearest risk of harm to users, either because of the scale of the platforms or because of known issues with serious harms.

32. Every company within scope will need to fulfil their duty of care, particularly to counter illegal content and activity, comply with information requests from the regulator, and, where appropriate, establish and maintain a complaints and appeals function which meets the requirements to be set out by the regulator.

33. Reflecting the importance of privacy, any requirements to scan or monitor content for tightly defined categories of illegal content will not apply to private channels. We are consulting on definitions of private communications, and what measures should apply to these services.

An independent regulator for online safety

34. An independent regulator will implement, oversee and enforce the new regulatory framework. It will have sufficient resources and the right expertise and capability to perform its role effectively.

35. The regulator will take a risk-based approach, prioritising action to tackle activity or content where there is the greatest evidence or threat of harm, or where children or other vulnerable users are at risk. To support this, the regulator will work closely with UK Research and Innovation (UKRI) and other partners to improve the evidence base. The regulator will set out expectations for companies to do what is reasonably practicable to counter harmful activity or content, depending on the nature of the harm, the risk of the harm occurring on their services, and the resources and technology available to them.

36. The regulator will have a legal duty to pay due regard to innovation, and to protect users’ rights online, taking particular care not to infringe privacy or freedom of expression. We are clear that the regulator will not be responsible for policing truth and accuracy online.

37. The government is consulting on whether the regulator should be a new or existing body. The regulator will be funded by industry in the medium term, and the government is exploring options such as fees, charges or a levy to put it on a sustainable footing. This could fund the full range of the regulator’s activity, including producing codes of practice, enforcing the duty of care, preparing transparency reports, and any education and awareness activities undertaken by the regulator.

Enforcement of the regulatory framework

38. The regulator will have a range of enforcement powers, including the power to levy substantial fines, that will ensure that all companies in scope of the regulatory framework fulfil their duty of care.

39. We are consulting on which enforcement powers the regulator should have at its disposal, particularly to ensure a level playing field between companies that have a legal presence in the UK, and those which operate entirely from overseas.

40. In particular, we are consulting on powers that would enable the regulator to disrupt the business activities of a non-compliant company, measures to impose liability on individual members of senior management, and measures to block non-compliant services.

41. The new regulatory framework will increase the responsibility of online services in a way that is compatible with the EU’s e-Commerce Directive, which limits their liability for illegal content until they have knowledge of its existence, and have failed to remove it from their services in good time.

Technology as part of the solution

42. Companies should invest in the development of safety technologies to reduce the burden on users to stay safe online.

43. In November 2018, the Home Secretary co-hosted a hackathon with five major tech companies to develop a new tool to tackle online grooming, which will be licensed for free to other companies, but more of these innovative and collaborative efforts are needed.

44. The government and the regulator will work with leading industry bodies and other regulators to support innovation and growth in this area and encourage the adoption of safety technologies.

45. The government will also work with industry and civil society to develop a safety by design framework, linking up with existing legal obligations around data protection by design and secure by design principles, to make it easier for start-ups and small businesses to embed safety during the development or update of products and services.

Empowering users

46. Users want to be empowered to keep themselves and their children safe online, but currently there is insufficient support in place and many feel vulnerable online.

47. While companies are supporting a range of positive initiatives, there is insufficient transparency about the level of investment and the effectiveness of different interventions. The regulator will have oversight of this investment.

48. The government will develop a new online media literacy strategy. This will be developed in broad consultation with stakeholders, including major digital, broadcast and news media organisations, the education sector, researchers and civil society. This strategy will ensure a coordinated and strategic approach to online media literacy education and awareness for children, young people and adults.

Next steps

49. This is a complex and novel area for public policy. To this end, as well as setting out the government’s proposed approach, this White Paper poses a series of questions about the design of the new regulatory framework and non-legislative package. A full list of these questions is included at the end of this White Paper.

PART 1: Introduction

1. The challenge

Summary • Illegal and unacceptable content and activity is widespread online, and UK users are frequently concerned about what they have seen or experienced. • The prevalence of the most serious illegal content and activity on the internet, which threatens our national security or the physical safety of children, is unacceptable. The ease and extremity of the most serious online offending such as child sexual exploitation and abuse (CSEA) continues to increase. • The impact of harmful content and activity can be particularly damaging for children and young people, and there are growing concerns about the potential impact on their mental health and wellbeing. • Tackling illegal and harmful content and activity online is one part of the UK’s wider mission to develop rules and norms for the internet, including protecting personal data, supporting competition in digital markets and promoting responsible digital design.

1.1. The internet is an integral part of everyday life for so many people. Nearly nine in ten UK adults are online and adult users spend around one day a week on the internet1. This is also true for children and young people, with 99% of 12-15 year olds going online, spending an average of twenty and a half hours a week on the internet2.

1.2. The internet can be a powerful force for good. It serves humanity, spreads ideas and enhances freedom and opportunity across the world. Online services facilitate the exchange of information, goods and services. They match supply and demand with great efficiency, increase consumer choice and lower distance between participants.

1.3. However, there is growing evidence of the scale of harmful content and activity that people experience online. Online services can be used to spread terrorist propaganda and child abuse content, they can be a tool for abuse and bullying, and they can be used to undermine civil discourse. Despite the many benefits of the internet, more than one in four adult users in the UK have experienced some form of harm related either to content or interactions online3.

1.4. Social media platforms and other technology companies increasingly acknowledge that they have a greater responsibility to protect their users from harm. British citizens want to feel empowered to keep themselves and their children safe and secure online. Both the government and industry have a responsibility to ensure this is the case.

Online harms suffered by individuals

1.5. The most appalling and horrifying illegal content and activity remains prevalent on an unacceptable scale. Existing efforts to tackle this activity have not delivered the necessary improvements, creating an urgent need for government to intervene to drive online services to step up their response.

1.6. There is a growing threat presented by online CSEA. In 2018 there were over 18.4 million referrals of child sexual abuse material by US tech companies to the National Center for Missing and Exploited Children (NCMEC)4. Of those, there were 113, 948 UK-related referrals in 2018, up from 82,109 in 2017. In the third quarter of 2018, Facebook reported removing 8.7 million pieces of content globally for breaching policies on child nudity and sexual exploitation5.

1.7. Not only is the scale of this offending increasing, so is its severity. The internet Watch Foundation (IWF) estimates that 55% of the child sexual abuse material they find online contains children aged ten or under, and 33% of this imagery is in the most serious category of abuse6.

1.8. Terrorists also continue to use online services to spread their vile propaganda and mobilise support (see Box 2). Terrorist content online threatens the UK’s national security and the safety of the public.

1.9. All five terrorist attacks in the UK during 2017 had an online element, and online terrorist content remains a feature of contemporary radicalisation7. It is seen across terrorist investigations, including cases where suspects have become very quickly radicalised to the point of planning attacks. This is partly as a result of the continued availability and deliberately attractive format of the terrorist material they are accessing online.

1.10. Terrorist groups work to find new ways to spread their propaganda and evade government and law enforcement efforts to prevent this. These threats are not only restricted to the largest, best-known services, but are prevalent across the internet. Terrorist groups and their supporters constantly diversify their reliance on the online services they use to host their material online. While Facebook reported removing over 14 million pieces of content related to terrorism or violent extremism in 20188, the terrorist group Daesh used over 100 platforms in 2018, making use of a wider range of more permissive and smaller platforms.

1.11. We have also seen terrorists and their supporters adopting new techniques, with material being shared using hacked social media accounts, and propaganda videos being edited in an effort to avoid detection.

1.12. Terrorist groups place a huge premium on quickly reaching their audiences. A third of all links to Daesh propaganda, for example, are disseminated within an hour of upload, while in the immediate aftermath of the terrorist attack in Christchurch, there was a co-ordinated cross-platform effort to generate maximum reach of footage of the attack. It is therefore vital to ensure that there is the technology in place to automatically detect and remove terrorist content within an hour of upload, secure the prevention of re-upload and prevent, where possible, new content being made available to users at all.

1.13. The threat continues to evolve with terrorists’ relentless desire to seek out new ways to share their propaganda in an effort to radicalise and recruit. The most effective way to combat this adaptive threat is to have a consistent cross-platform response to ensure there are no safe spaces for terrorists to operate online.

1.14. Rival gangs use social media to glamourise weapons and gang life, as well as to directly depict or incite acts of violence. Alongside the illegal sale of weapons to young people online, this is a contributing factor to incidents of serious violence, including knife crime, in the UK. The latest police recorded crime figures, for the year ending September 2018, show an 8% increase in knife crime (to 39,818 offences) compared with the previous year. Homicide figures have risen by 14% (excluding terrorist attacks) over the same period9.

Harm: Child sexual exploitation and abuse online - Box 1 Threat: Child sex offenders use the internet to view and share Child Sexual Abuse Material (CSAM), groom children online, and live stream the sexual abuse of children. The sheer scale of CSEA online is horrifying. • In 2017, the IWF assessed 80,319 confirmed reports of websites hosting or linking to images of child sexual abuse. A total of 43% of the children in the images were aged 11-15 years old, and 57% were ten years old or younger. Two per cent were aged two or younger. • Sexual exploitation can happen to any young person – whatever their background, age, gender, race or sexuality or wherever they live. • In the most horrific cases, child sex offenders in developing countries are abusing children at the instigation of offenders in the UK who commission the abuse online and watch it over live stream for a fee. Impact: • Victims of abuse report ongoing trauma caused by the knowledge that images of their abuse are still being circulated and viewed by child sex offenders online. Victims also fear being recognised as a result of their images being available online. • Victims of online grooming suffer lasting harm after being blackmailed and coerced into sharing indecent images of themselves or live-streaming themselves to offenders, and live in fear that those images could be used against them.

Harm: Terrorist content online - Box 2 Threat: Terrorists, including Islamist groups such as Daesh and Al-Qaeda as well as far right terrorists, use the internet to spread propaganda designed to radicalise vulnerable people, and distribute material designed to aid and abet terrorist attacks. There are also examples of terrorists broadcasting attacks live on social media. Terrorist use of the internet poses a threat to national security and the safety of the public. • As larger platforms take more action against terrorist propaganda, terrorist groups have spread out to a wider range of more permissive and smaller platforms. • Terrorist groups are adopting new techniques to avoid detection, including sharing material via hacked social media accounts, and subtly altering propaganda videos. • Terrorist groups place a huge premium on quickly reaching their audiences. Impact: • The availability and spread of terrorist content online has been shown to contribute to terrorist attacks on UK soil. • All five of the terrorist attacks undertaken in the UK during 2017 had an online element to them (Speech at Digital Forum (2018). San Francisco by the Rt Hon Amber Rudd, 13 February 2018). • Online terrorist content is seen across terrorist investigations, including cases where suspects have become very quickly radicalised to the point of planning attacks.

Harm: Content illegally uploaded from prisons - Box 3 Threat: There are an increasing number of cases where online content originating from prisons is illegally uploaded by prisoners to social media. • Some prisoners transmit videos, images and messages from prisons using prohibited devices such as mobile phones. • They can use social media accounts to harass and intimidate their victims. Impact: • This can lead to victims of crime feeling that they have no escape from their tormentors, even when they have been imprisoned. • Prisoners openly uploading content from prisons can also undermine public confidence in the prison service.

Tackling serious violence online - Box 4 Rival gangs use social media to promote gang culture, taunt each other and incite violence. Content can also either directly depict or incite real world violence or glamourise gang life and the use of weapons. Government and law enforcement are taking action to tackle this threat: • We have provided £1.4 million to support a new national police capability to tackle gang related activity on social media. • This will bring together a dedicated team to take action against online material, focusing on investigative, disruption and enforcement work against specific gang targets, as well as making referrals to social media companies so illegal and harmful content can be taken down. • Prior to this, a new action group was established to bring together government, social media companies, police and community groups to tackle violent material available via social media.

Harm: The sale of opioids online - Box 5 Threat: Powerful and dangerous opioids are marketed and sold online. Fentanyl and its analogues (substances with similar but slightly altered chemical structures) are a group of powerful synthetic opioids. They have similar effects to other opioids such as morphine and heroin, but are significantly more potent. • Since December 2016, there have been at least 143 recorded deaths in the UK attributed to Fentanyl and its analogues (NCA analysis). Fentanyl has been sold on several well- known social media sites. The products are marketed as top-quality substances with fast and secure delivery, with mobile phone numbers provided for follow-up contact. • Of particular concern is that some social media groups and threads, including those used by vulnerable people, are being targeted. This includes people suffering from chronic pain, where there is a risk of accidental overdose and people dealing with depression, where there is a risk that the fentanyl may be used to assist suicide. Impact: • Whilst these products continue to be made available there is a risk that fatalities will increase. • There is also a risk that health professionals and other first responders will continue to be exposed to potentially harmful environments.

1.15. Beyond illegal activity, other behaviour online also causes harm. In 2017, one in five children aged 11 to 19 reported having experienced cyberbullying in the past year10; 21% of women have received misogynistic abuse online11, and half of girls aware of sexist abuse on social media say this has restricted what they do or aspire to in some way12. The House of Commons Petitions Committee has highlighted the extreme abuse experienced online by disabled people, which has forced some of them to leave social media13.

1.16. Victims have also described a qualitative difference between online and offline harms, particularly in reference to online abuse. The Law Commission noted the perceived anonymity of offenders as one of the characteristics of online abuse that may result in a different experience for victims – see Box 614. Many users also feel that the market currently offers them very few alternative, safer online services. For example, the 2018 Doteveryone Digital Attitudes15 report found that almost half of respondents felt they had no choice but to sign up to online services, even where they had concerns.

Tackling online anonymous abuse - Box 6 The internet can be used to harass, bully or intimidate. In many cases of harassment and other forms of abusive communications online, the offender will be unknown to the victim. In some instances, they will have taken technical steps to conceal their identity. Government and law enforcement are taking action to tackle this threat. • The police have a range of legal powers to identify individuals who attempt to use anonymity to escape sanctions for online abuse, where the activity is illegal. The government will work with law enforcement to review whether the current powers are sufficient to tackle anonymous abuse online. • We are enhancing law enforcement’s ability to tackle anonymous online abuse by investing in training that is designed to improve digital capability across policing. For example, as part of the £4.6 million Police Transformation Fund allocated by the Home Office, the Digital Investigation and Intelligence programme will build police capability to respond to the full range of digital crime types, through investment in technology and training. • We are also making it easier for the public to report online crimes. Through the Digital Public Contact programme, we will provide the public with a digitally accessible police force with a consistent set of online capabilities to use in engaging and transacting with police services through a single online channel. • We also expect companies to do substantially more to keep their users safe and counter online abuse, particularly where this is illegal. Companies need to take responsibility for tackling abusive behaviour on their services. More detail is set out in Chapter 3.

Online harms suffered by children and young people

1.17. Being online can be a hugely positive experience for children and young people – see Box 7. Recent research by internet Matters found that seven in ten parents think screen time is essential for their children’s learning development and two thirds of parents feel that devices give their children another outlet for creativity, particularly so for children aged 6-1016.

1.18 However, the impact of harmful content and activity can be particularly damaging for children, as set out in Box 1 above and Boxes 8-10 below. There is also growing concern about the relationship between social media and the mental health of children and young people. The Children’s Commissioner’s report published in November 2018 ‘Who knows what about me’ sets out the huge size and growth of children’s digital footprint and the associated risks and benefits17. Internet Matters reported in February 2019 that vulnerable young people are more likely to suffer online harms and less likely to receive online safety advice and education18.

The positive impact of being online for children and young people - Box 7 Most children have a positive experience online, using the internet for social networking and connecting with peers, as well as to access educational resources, information, and entertainment. The internet opens up new opportunities for learning, performance, creativity and expression. • A literature review by the UK Council for Child Internet Safety (2017) highlights evidence that young people recognise the positive role of the internet in relation to self-expression, developing understanding, bringing people together and respecting and celebrating differences. Research by UNICEF (2017) shows that use of technology is beneficial for children’s social relationships, enabling them to enhance existing relationships and build positive friendships online. • A report by The Royal Society for Public Health in 2017 found that young people reading blogs or watching vlogs on personal health issues helped improve their knowledge and understanding, prompted individuals to access health services, and enabled them to better explain their own health issues or make better choices. They also found that young people are increasingly turning to social media as a means of emotional support to prevent and address mental health issues. • More recently, research by Ofcom showed that nine in ten social media users aged 12-15 state that this use has made them feel happy or helped them feel closer to their friends. Two thirds of 12-15 year olds who use social media or messaging sites say they send support messages, comments or posts to friends if they are having a difficult time. One in eight support causes or organisations by sharing or commenting on posts. • In the 2019 UK Safer Internet Centre survey, 70% of young people surveyed said that being online helps them understand what’s happening in the world, with 60% noting they have only seen or heard about certain issues or news because they heard about them from the internet. 43% said they have been inspired to take action because of something they saw online, with 48% stating being online makes them feel that their voice or actions matter.

Harm: Cyberbullying - Box 8 Threat: In 2017, one in five children surveyed aged 11-19 reported having experienced cyberbullying in the past year. • The prevalence of cyberbullying is higher for some groups, such as women, religious minorities, LGBT+, BME and disabled individuals. Impact: • Cyberbullying has been shown to have psychological and emotional impact. In a large survey of young people who had been cyberbullied, 41% had developed social anxiety, 37% had developed depression, 26% had suicidal thoughts and 25% had self-harmed. • These figures are all higher than corresponding statistics for ‘offline’ bullying, and indicated the increased potential for harm of cyberbullying.

Harm: Self-harm and suicide - Box 9 Threat: In a survey of young adults, 22.5% reported self-harm and suicide-related internet use, including 8.2% and 7.5% who had actively searched for information about self-harm and suicide respectively. • Amongst those who had harmed with suicidal intent, 70% reported self-harm and suicide-related internet use. • The prevalence of using the internet to view related content has also been found to be higher in children than adults. One study of those presenting to hospital following self-harm found that 26% of children had viewed self-harm and suicide content, compared to 8.4% of adults. Impact: • The National Confidential Inquiry into Suicide and Safety in Mental Health (NCISH) analysed the characteristics of 595 children and young people (aged under 20) who had died by suicide in the UK between 2014 and 2016. • The NCISH found that suicide-related internet use (i.e. searching the internet for information on suicide methods) was reported for almost a quarter (23%) of these children and young people.

Harm: Underage sharing of sexual imagery - Box 10 Many children and young people take and share sexual images. Creating, possessing, copying or distributing sexual or indecent images of children and young people under the age of 18 is illegal, including those taken and shared by the subject of the image. • Surveys provide tentative evidence that between 26% and 38% of 14-17 year olds have sent sexual images to a partner, and between 12% and 49% have received a sexual image. • The proportion of young people sending images varies with age, with one study indicating that 26% of 14 year olds had sent and received sexual images, rising to 48% of 16 year olds. Impact: • Sharing sexual images can expose children and young people to bullying, humiliation, objectification and guilt. These images can be shared widely and appear on offender forums or adult pornography sites, or be used to extort further imagery. This puts children and young people in a vulnerable position and at risk of harm. It is a criminal offence to produce, possess or share sexual images of under 18 year olds. • The National Society for the Prevention of Cruelty to Children (NSPCC) reported that sexting was discussed in 1,392 counselling sessions with children and young people on their helplines that year, representing a 15% increase on the year before.

1.19. The UK Chief Medical Officers (UK CMOs) commissioned independent researchers to carry out a systematic evidence review on the impact of social media use on children and young people’s mental health. The review covered important and diverse issues including cyberbullying, online gaming, sleep problems and problematic internet use, which is also known as ‘internet addiction’.

1.20. Overall the research did not present evidence of a causal relationship between screen-based activities and mental health problems, but it did find some associations between screen-based activities and negative effects, such as increased risk of anxiety or depression19. It is important that parents and carers support their children to have positive experiences online.

1.21. While there is not yet sufficient evidence about the impact of screen time to support detailed guidelines for parents or requirements on companies, we will continue to support research in this area and ensure high quality advice is available to families. We also welcome efforts from the industry to develop tools to help individuals and families understand and manage how much time they spend online – more information on these is in Box 33.

Emerging challenge: Screen time - Box 11 Screen time and its impact on children is an issue of growing concern. Research by Internet Matters found that nearly half of parents (47%) are concerned about the amount of time their child spends online and 88% take measures to limit their child’s use of devices. • The UK CMOs recently conducted a systematic evidence review on children and young people’s screen and social media use. The CMO subsequently produced advice for parents and carers to encourage them to discuss boundaries with children around online behaviours and time spent using screens, and to lead by example. For example, the UK CMOs advised that: Sleep matters. Getting enough good quality sleep is very important. Leave phones outside the bedroom when it is bedtime.

Sharing sensibly. Talk about sharing photos and information online and how photos and words are sometimes manipulated. Parents and carers should never assume that children are happy for their photos to be shared. For everyone – when in doubt, don’t upload!

Education matters. Make sure you and your children are aware of, and abide by, their school’s policy on screen time.

Keep moving! Everyone should take a break after a couple of hours sitting or lying down using a screen. It’s good to get up and move about a bit. #sitlessmovemore

Safety when out and about. Advise children to put their screens away while crossing the road or doing an activity that needs their full attention.

Talking helps. Talk with children about using screens and what they are watching. A change in behaviour can be a sign they are distressed – make sure they know they can always speak to you or another responsible adult if they feel uncomfortable with screen or social media use.

Family time together. Screen-free meal times are a good idea – you can enjoy face-to-face conversation, with adults giving their full attention to children.

Use helpful phone features. Some devices and platforms have special features – try using these features to keep track of how much time you (and with their permission, your children) spend looking at screens or on social media. Future action – building our understanding: Given the amount of time many children spend online, and the level of parental concern on this issue, we urgently need to build a better understanding. • While we do not expect the regulator to set requirements around screen time, both government and the regulator will continue to support research in this area to inform future action in this space. • We need to develop a better understanding of not just of the impact of screen time as a whole, but also between different types of screen time and children’s development and wellbeing. • As part of this, we also expect companies to support the developing evidence base around screen time, for example by providing access to anonymised data to researchers as recommended by the CMOs • If the emerging evidence base demonstrates a strong link between different elements of screen time and damage to children’s wellbeing or development, companies will be expected to take appropriate action to fulfil their duty of care.

Threats to our way of life

1.22. The UK’s reputation and influence across the globe is founded upon our values and principles. Our society is built on confidence in public institutions, trust in electoral processes, a robust, lively and plural media, and hard-won democratic freedoms that allow different voices, views and opinions to freely and peacefully contribute to public discourse.

1.23. Inaccurate information, regardless of intent, can be harmful – for example the spread of inaccurate anti-vaccination messaging online poses a risk to public health. The government is particularly worried about disinformation (information which is created or disseminated with the deliberate intent to mislead; this could be to cause harm, or for personal, political or financial gain).

1.24. Disinformation threatens these values and principles, and can threaten public safety, undermine national security, fracture community cohesion and reduce trust.

1.25. These concerns have been well set out in the wide-ranging inquiry led by the Digital, Culture, Media and Sport (DCMS) Select Committee report on fake news and disinformation, published on 18 February 2019. This White Paper has benefited greatly from this analysis and takes forward a number of the recommendations. The government will be responding to the DCMS Select Committee report in full in due course. We also note the recent papers from the Electoral Commission and Information Commissioner’s Office on this and wider issues, and are considering these closely.

Harm: Online disinformation - Box 12 Threat: Online disinformation – spreading false information to deceive deliberately – is becoming more and more prevalent. Misinformation refers to the inadvertent sharing of false information. • A recent study from the University of Oxford’s Computational Propaganda Project has found evidence of organised social media manipulation campaigns in 48 countries in 2018. According to the Reuters Institute, 61% of people want the government to do more to separate what is real and fake on the internet (2018). • One of the major technological challenges in disinformation is the continued development of AI systems. AI techniques can be used to target and manipulate individual voters, with highly sophisticated micro-targeting based on individual psychology. • AI can be beneficial in the automatic detection of content, or automatically fact- checking articles. But developments in AI also make it possible to generate fake content (text, audio and video) which is difficult to detect by humans and algorithms – known as ‘deepfakes’. As a result, it is becoming even easier to create and disseminate false content and narratives. • The Russian State is a major source of disinformation. The Kremlin has used disinformation to obfuscate and confuse audiences around their illegal annexation of Crimea, intervention in eastern Ukraine and the shooting down of Malaysian Airlines flight MH17, which led to the deaths of 298 people including ten UK citizens. After the attempted murder of Sergei and Yulia Skripal in Salisbury in March 2018, the Russian State led a concerted disinformation campaign to distract from their culpability. This included the use of state media and covert social media accounts to sow over 40 different narratives as to what happened. Impact: • Most users are not always aware that much of the content they see is determined by sophisticated algorithms that draw on data about their online activity, such as their browsing history, their social media networks and what they post. • Research by Doteveryone suggests that 62% of people do not realise that their social networks can affect the news they see, while only three in ten adult online users questioned by Ofcom were aware of the ways in which companies can collect data about them online.

Harm: Online manipulation - Box 13 Threat: Propaganda and false information have long been used to persuade and mislead, but the internet, social media and AI provide ever more effective ways to manipulate opinion. • The tolerance of conflicting views and ideas are core facets of our democracy. However, these are inherently vulnerable to the efforts of a few to manipulate and confuse the information environment for nefarious purposes, including undermining trust. A combination of personal data collection, AI based algorithms and false or misleading information could be used to manipulate the public with unprecedented effectiveness. • The distinction between legitimate influence and illegitimate manipulation is not new. The government took action to prevent subliminal broadcast advertising in the Broadcasting Act 1990. The government gave the Independent Television Commission (replaced by Ofcom) a duty to ensure that licensed services complied with requirements not to include technical devices which convey messages or influence individuals without them being aware. We believe the government should make sure there are similar boundaries between legitimate and illegitimate practices online. The techniques and practices used are still emerging. We are developing a better understanding of the nature and scale of the potential problem and effective interventions.

Harm: Online abuse of public figures - Box 14 Threat: In recent years we have seen a worrying rise in the amount of abuse, harassment and intimidation directed at those in public life. Much of this abuse happens on social media. Impact: • An international survey of female journalists found two thirds (64%) had experienced online abuse – death or rape threats, sexist comments, cyberstalking, account impersonation, and obscene messages.46 Almost half (47%) did not report the abuse they had received, and two fifths (38%) admitted to self-censorship in the face of this abuse. • The Guardian’s research into the 70 million comments left on its site over a ten year period highlighted that of the ten most abused writers, eight were women and two were black men. This is in spite of the fact that the majority of the regular opinion writers for The Guardian are white men. This was then compared to the ten writers who received the least abuse – who were all men.48 There are too many stories of public figures closing their social media accounts following waves of abuse. • In December 2017 the Committee for Standards in Public Life, which was commissioned by the Prime Minister, published its report on intimidation in public life. The consultation sought views on a range of ideas including establishing a new offence of intimidation, and requiring imprints on electronic campaigning. The report included examples of the extent of intimidation of those in public life. • “It is hard to explain how it makes you feel. It is anonymous people that you’ve never met, true, but it has a genuinely detrimental effect on your mental health. You are constantly thinking about these people and the hatred and bile they are directing towards you.” – Rachel Maclean MP • “I spoke on a number of occasions in the House of Commons in different committees about the rights of women. To which I suffered daily attacks on Twitter, on my email system or endless online articles written about how people wished to see me raped.” – Jess Phillips MP • The report also makes a number of recommendations for actions that social media companies should take in relation to intimidatory content, including implementing tools to enhance the ability of users to tackle online intimidation and supporting users who become victims of this behaviour. These recommendations have helped to shape the indicative list of steps the regulator may want to include in codes of practice. • The government’s response to the Committee on Standards in Public Life’s Review of Intimidation in Public Life was published in March 2018 and set out a number of actions for government based on the Committee’s recommendations. As part of this work the government has undertaken a public consultation entitled Protecting the Debate: Intimidation, Influence and Information which closed in October 2018. The government’s response will be published in due course. This abuse is unacceptable – it goes beyond free speech and free debate, dissuades good people from going into public life, and corrodes the values on which our democracy rests. The new regulatory framework will make clear companies’ responsibility to address this harm.

Other online harms

1.26. There are other harms associated with the internet and online technology. For example, Ofcom and ICO’s report on Internet Regulation highlighted users’ concerns around privacy and hacking20. This White Paper is part of the government’s wider programme of work to establish the right norms and rules for the internet.

Responsible and ethical technology

1.27. The government takes both the protection of personal data and the right to privacy extremely seriously. The General Data Protection Regulation (GDPR) and the Data Protection Act 2018 (DPA), alongside the increased powers for the ICO to gather evidence, inspect artificial intelligence (AI) and levy significant fines on those who break the law, update our data protection laws fit for the digital age.

1.28. The DPA also includes an important new provision requiring the Information Commissioner to produce an age-appropriate design code. This provision breaks new ground by addressing the approach to the design of online services likely to be used by children. It ensures platforms and service providers put child user interests at the centre of the design process, and protects them from risks that arise from the use of their personal data online, including the algorithms and profiling that serves them with personalised content.

1.29. However, the increased use of data and AI is giving rise to complex, fast-moving and far-reaching ethical and economic issues that cannot be addressed by data protection laws alone. Increasingly sophisticated algorithms can glean powerful insights, which can be deployed in ways that influence the decisions we make and the services we receive. It is essential that we understand, and respond to, barriers to the ethical deployment of AI.

1.30. That is why the government has set up the Centre for Data Ethics and Innovation. The Centre will provide independent, impartial and expert advice on the ethical and innovative deployment of data and AI. The Centre will publish its first strategy document in spring 2019, setting out further details on its key priorities.

1.31. The way that technology is designed, who it is designed by and the outcomes it is trying to achieve also influence how it impacts its users and wider society. There is an increasing amount of evidence that social media platforms and other digital services can impact people’s habits, sleep patterns, productivity at work, attention spans and even voting preferences – see Box 15. We are looking carefully at how we can ensure that digital products and services are designed in a responsible way, with their users’ well-being in mind. Chapter 8 of this paper looks specifically at how we are working with companies to include considerations around safety in the design of their products.

Emerging challenge: Designed addiction - Box 15 Some online products have been designed to encourage continuous use. They include seemingly small but influential features, which incentivise people to keep using the app or platform for longer. One common example is the ‘infinite scroll’, in which information is loaded continuously as the user scrolls down the page, encouraging the user to keep scrolling. • A recent report by 5Rights highlighted other elements of ‘persuasive design’, such as ‘typing bubbles’, quantifying friends, and notifications. Even ‘likes’ can be powerful tools for keeping users online.52 • These techniques could exacerbate addictive behaviours. The number of people suffering from clinical addiction in this way has not been reliably quantified, but there are well-documented extreme cases of vulnerable individuals for whom addiction has got in the way of social lives, sleep, physical activity and other parts of a healthy, balanced lifestyle. • There is also evidence that a wider range of people experience less extreme forms of compulsive or habitual behaviour online. Some experts would stress that this compulsive behaviour is not clinical addiction. Future action: • The government shares concerns around designed addiction and is determined to ensure that we have sufficient evidence on this risk, and the right expectations of companies to design their products in safe ways. • In the future, we expect the regulator will continue to support research in this area to inform future action and, if necessary, set clear expectations for companies to prevent harm to their users. • We also expect companies to be transparent about design practices which encourage extended engagement, and to engage with researchers to understand the impact of these practices on their users. • DCMS is continuing to work with the Gambling Commission and the industry on player protections in the online sector. In May 2018, we published the response to the Consultation on Proposals for Changes to Gaming Machines and Social Responsibility Measures, which set out a clear plan to strengthen player protections. • Since then, a number of changes have been made to make gambling fairer and safer, including tightening advertising rules and launching GAMSTOP, the online self-exclusion scheme. Additionally, from May, the Gambling Commission will bring in changes that mean that age and identity must be verified before consumers can deposit money and gamble, and will require age verification before customers can access free-to-play demo games.

Thriving digital markets

1.32. The digital sector makes a huge contribution to our economy at £130.5 billion gross value added in 2017, equivalent to 7% of the UK gross value added. The sector has seen strong growth with an increase of 33% since 2010, compared to 29% for the total UK economy21. As the digital economy has grown, powerful new companies have emerged, often with very dominant market positions. This has raised questions about the competitiveness of digital markets and what this means for consumers.

1.33. The government’s Modernising Consumer Markets Green Paper sought views on how well equipped the UK’s competition regime is to manage emerging challenges, including the growth of fast-moving digital markets. We continue to consider policy options across the range of measures proposed in the green paper and are conducting a review of digital markets, due in summer 2019. This will be informed by the work of the independent Digital Competition Expert Panel, led by Professor Jason Furman which published its recommendations for government on 13 March 2019.

1.34. Professor Furman and the panel found that the digital economy has brought significant benefits but does not have enough competition. They called for a new digital markets unit to set and enforce a code of conduct so the largest digital companies know what are acceptable rules for competition. This new unit would give people more control over their data by enabling people to switch between platforms more easily. They also recommended changes to merger rules, and updating existing rules to improve enforcement over anticompetitive conduct. The government will consider these proposals and respond later in the year.

1.35. Thriving digital markets also rely on the innovative, efficient and fair use of data. In June 2018, the Secretary of State for DCMS announced that we would develop a National Data Strategy to ensure the UK is a world-leading data economy – unlocking the power of data across government and the wider economy, while building public trust and confidence in its use.

Online advertising

1.36. Online advertising plays a crucial role in the digital economy, with many free digital services, such as search engines or social networks, funded by advertising revenues. The online advertising ecosystem is highly complex, with much of the advertising space online bought through automated processes, and the velocity with which adverts are created and displayed is far higher than offline. Online advertising encourages and rewards the collection of user data (the more data a service has on a user the more effectively it can target adverts at them) and the holding of people’s attention (the longer they use a service the more adverts they see).

1.37. This combination of factors has given rise to a number of issues caused by or related to online advertising. Work is already underway to address some of these:

• A Home Office-led working group on CSEA and terrorist content linked to advertising met in March 2019, following an initial meeting in December 2018, comprising representatives from advertising trade bodies, agencies, brands, law enforcement and the IWF. The working group’s activity to date comprises actions to help ensure advertising is not supporting this kind of illegal activity.

• As part of the government’s Childhood Obesity Plan, DCMS and the Department of Health and Social Care launched a consultation on 18 March 2019 on introducing a 9pm watershed on TV advertising of products high in fat, salt or sugar, and similar protections for children viewing adverts online.

• The Competition and Markets Authority is considering further work on digital advertising, although this is dependent on the outcome of EU exit negotiations.

• In November 2018 the Advertising Standards Authority published its strategy More Impact Online, which aims to put the protection of consumers online at the heart of its work over the next five years, and makes commitments to explore, for example, the use of machine learning and AI to improve regulation.

• In 2018, the ICO conducted an investigation into data analytics and micro targeting of political advertising online. The report, ‘Democracy Disrupted?’, highlighted the risks of personal data being abused in digital campaigning and made a number of recommendations to improve transparency and data protection compliance. The ICO has also commenced a broader examination of the use of personal data in adtech22.

1.38. As announced in the DCMS Secretary of State’s immediate response to the Cairncross Review, DCMS will conduct a review of how online advertising is regulated in the UK.

2. The harms in scope

Summary • This White Paper sets out government action to tackle online content or activity that harms individual users, particularly children, or threatens our way of life in the UK, either by undermining national security, or by reducing trust and undermining our shared rights, responsibilities and opportunities to foster integration. It sets out an initial list of content and behaviour which will be in scope, as well as a list of harms which will be excluded. • There is currently a patchwork of regulation and voluntary initiatives aimed at addressing these problems, but these have not gone far or fast enough to keep UK users safe online. • Many of our international partners are also developing new regulatory approaches to tackle online harms, but the UK will be the first to tackle online harms in a coherent, single regulatory framework that reflects our commitment to a free, open and secure internet.

Harmful content or activity in scope of the White Paper

2.1. Table 1 below shows the initial list of online harmful content or activity in scope of the White Paper, based on an assessment of their prevalence and impact on individuals and society.

2.2. This list is, by design, neither exhaustive nor fixed. A static list could prevent swift regulatory action to address new forms of online harm, new technologies, content and new online activities.

Table 1: Online harms in scope

﻿ Harms with a clear definition Harms with a less clear definition Underage exposure to legal content Child sexual exploitation and abuse. Cyberbullying and trolling. Children accessing pornography. Terrorist content and activity. Extremist content and activity. Children accessing inappropriate material (including under 13s using social media and under 18s using dating apps; excessive screen time). Organised immigration crime. Coercive behaviour. Modern slavery. Intimidation. Extreme pornography. Disinformation. Revenge pornography. Violent content. Harassment and cyberstalking. Advocacy of self-harm. Hate crime. Promotion of Female Genital Mutilation (FGM). Encouraging or assisting suicide. Incitement of violence. Sale of illegal goods/ services, such as drugs and weapons (on the open internet). Content illegally uploaded from prisons. Sexting of indecent images by under 18s (creating, possessing, copying or distributing indecent or sexual images of children and young people under the age of 18).

2.3. There is already an effective response to some categories of harmful content or activity online. These will be excluded from the scope of the new regulatory framework to avoid duplication of existing government activity.

2.4. The following harms will be excluded from scope:

• All harms to organisations, such as companies, as opposed to harms suffered by individuals. This excludes harms relating to most aspects of competition law, most cases of intellectual property violation, and the organisational response to many cases of fraudulent activity. The government is leading separate initiatives to tackle these issues. For example, the Joint Fraud Taskforce is leading an ambitious programme of work to tackle fraud, including online fraud, through partnership between banks, law enforcement and government.

• All harms suffered by individuals that result directly from a breach of the data protection legislation, including distress arising from intrusion, harm from unfair processing, and any financial losses. Box 16 explains how the UK’s legal framework provides protection against online harms linked to data breaches.

• All harms suffered by individuals resulting directly from a breach of cyber security or hacking. These harms are addressed through the government’s National Cyber Security Strategy.

• All harms suffered by individuals on the dark web rather than the open internet. These harms are addressed in the government’s Serious and Organised Crime Strategy. A law enforcement response to criminality on the dark web is considered the most effective response to the threat. As set out in the strategy, the government continues to invest in specialist law enforcement skills and capability.

Stronger regulation of personal data online - Box 16 The UK already enjoys high standards of data protection law, that were modernised in 2018 with the introduction of the GDPR and the Data Protection Act 2018. The government chose to go further than other countries, by providing stronger powers to apply to the investigation and enforcement of specific online threats. Key protections for online harms involving personal data include: • An obligation to provide clear and accessible privacy information, tailored for children when they are the users of online services. • A legal obligation to accountability, making companies responsible for placing data protection at the centre of the design of online services in a way that mitigates the risk to users’ information. This also includes a requirement to undertake data protection impact assessments, and have them approved by the ICO where high risks persist. • A right to erasure of personal data online, with stronger provisions where data has been gathered from a child user. • An age-appropriate design code, which gives the design standards we will expect providers of online services and apps used by children to meet when they process their data. • A power to inspect algorithms in situ, to understand their use of personal data and whether this leads to bias or other detriment. • A power to require information to be handed over to the ICO wherever it is held, including on cloud servers.

Shortcomings of the current regulatory landscape

2.5. Currently there is a range of UK regulations aimed at specific online harms or services in scope of the White Paper, but this creates a fragmented regulatory environment which is insufficient to meet the full breadth of the challenges we face. The current regulatory framework includes:

• GDPR and the Data Protection Act enforced by the ICO. This includes collection and use of personal data, including when online. The GDPR also has extraterritorial scope and can be enforced against companies outside the UK who offer services to UK users23.

• The Electoral Commission’s oversight of the activity of political parties, and other campaigners, including activity on social media24.

• Forthcoming age verification requirements for online pornography25.

• The Equality and Human Rights Commission’s oversight of the Equality Act 2010 and Freedom of Expression26.

• Ofcom’s existing oversight of video-on-demand services27.

• The revised EU Audiovisual Media Services Directive, which will introduce new high-level requirements for video sharing platforms such as YouTube28.

• The Gambling Commission’s licensing and regulation of online gambling29. DCMS has been working with the Commission to tighten advertising rules on gambling and launched GAMSTOP, the online self-exclusion scheme. Additional age- verification requirements are expected to come take effect in from May this year30.

• The Competition and Markets Authority’s (CMA) enforcement of consumer protection law online. See Box 17 for further details.

Consumer enforcement by the Competition and Markets Authority - Box 17 Businesses risk breaching consumer protection law where their online behaviour misleads consumers or treats them unfairly. The CMA has undertaken a range of recent enforcement activity examining potentially unfair or misleading online behaviour, including: • Online gambling – the CMA worked with the Gambling Commission to sanction unfair online ‘bonus’ promotions by major gambling firms. The CMA was concerned that players’ money could effectively be trapped under the terms of these promotions, or that they could be caught out by unclear or imbalanced promotion rules. Changes were agreed with a number of firms, including William Hill and Ladbrokes. • Online reviews and endorsements – the CMA has an ongoing programme of work to tackle fake or misleading online reviews and endorsements. Most recently, 16 celebrities, reality stars and social media influencers committed to always be clear in their social media posts where they have been paid to post content online. The CMA is now examining the responsibility of social media platforms to ensure that paid-for content is always properly disclosed. • Secondary tickets – as a result of action by the CMA, including court proceedings against Viagogo, consumers will always receive essential information before they purchase a ticket from online resale platforms, in particular if there is a risk that the consumer will not be able to get into the event or venue. The court order secured against Viagogo also requires that ‘pressure selling’ messages are removed from their website. • Online hotel booking – the CMA recently agreed changes with companies in the Booking.com and Expedia corporate groups in relation to potentially misleading online practices. These include new requirements to be clear about the role that commission plays in the order of search results and that any claims about the limited availability of hotel rooms are accurate and do not risk misleading consumers.

2.6. Under the current liability regime, which is derived from the EU’s e-Commerce Directive, platforms are protected from legal liability for any illegal content they ‘host’ (rather than create) until they have either actual knowledge of it or are aware of facts or circumstances from which it would have been apparent that it was unlawful, and have failed to act ‘expeditiously’ to remove or disable access to it. In other words, they are not liable for a piece of user-generated illegal content until they have received a notification of its existence, or if their technology has identified such content, and have subsequently failed to remove it from their services in good time.

2.7. For illegal harms, it is also important to make sure that criminal law applies online in the same way as it applies offline. In February 2018 the Prime Minister announced a review by the Law Commission of the law in relation to abusive and offensive online communications, to highlight any gaps in the criminal law which cause problems in tackling this abuse. In its scoping report last year, the Law Commission concluded that behaviour is broadly criminalised to the same extent online as offline and recommended a clarification of existing communication offences. The government is now finalising the details of the second phase of the Law Commission work.

2.8. For legal harms, the same piece of content can be subject to different regulatory standards depending on the platform on which it appears. Ofcom’s report Addressing Harmful Content Online sets out how the same programme would be regulated to differing degrees depending on whether it is broadcast on TV, viewed on-demand, or on an online video sharing platform (see Box 18). This means that there are significant gaps in consumer protection31.

Regulation of the same content on different services - Box 18

Voluntary approaches

2.9. Beyond this range of regulatory requirements, the government’s Internet Safety Strategy Green Paper, published in 2017, focused on a voluntary approach to countering harmful behaviour and content online. The green paper recognised that government alone cannot keep citizens safe from online harms, and sought to work in close partnership with industry to put in place specific technical solutions to make social media platforms safer.

2.10. Voluntary initiatives between government, industry and civil society are promising in some areas, and the leading companies have taken a number of steps to improve their platforms, for example as set out in Boxes 19-21. We are clear that the progress made on terrorism and CSEA through this voluntary cooperation with the industry must continue, alongside the development of a new regulatory framework.

Existing initiatives to tackle online harms: Global Internet Forum to Counter Terrorism - Box 19 Following the Westminster terrorist attack in March 2017, the government convened a roundtable with major industry players, including Facebook, Twitter, Google and Microsoft to see what more could be done to tackle terrorist content online. This led to these companies setting up the Global Internet Forum to Counter Terrorism (GIFCT) in June 2017. The GIFCT is leading the cross-industry response to reduce the availability of terrorist content on the internet so that there are no safe spaces for terrorists online. Key objectives for the Forum are to increase the use of automation and machine learning technology to detect and remove terrorist content – ultimately preventing terrorist content being made available to users in the first place – and supporting smaller, less well- resourced companies to tackle these threats on their own platforms. The Forum has taken some positive steps since its establishment, but there is still much more to do. The government wants to see an ambitious and tangible plan for delivery. Our aims for the GIFCT in 2019 are for the Forum to: • Expand its membership, securing a greater range and quantity of companies to sign up as members of the Forum • Devote greater efforts to targeted interventions with priority platforms, including through the development and sharing of automated technology • Put in place a clear programme of activity, providing metrics against which success can be measured • Provide greater visibility to drive this agenda forward, including companies having a clearer public voice on the issue

Existing initiatives to tackle online harms: UK Council for Internet Safety - Box 20 The UK Council for Internet Safety (UKCIS) is a new collaborative forum through which government, the tech community and civil society work together to ensure the UK is the safest place in the world to be online. Expanding the scope of the former UK Council for Child Internet Safety (UKCCIS), UKCIS works to tackle online harms such as hate crime, extremism and violence against women and girls, in addition to maintaining a focus on the needs of children. Priority areas of delivery for UKCIS over the next year include: • Producing a landscape review of research around adult online harms, and regular concise summaries of emerging research. • Updated guidance to schools on sexting, and evaluation of online safety provision, and for Initial Teacher Training providers to help them upskill new teachers in online safety. • Promoting the Connected World framework, which describes the digital knowledge and skills that children should have the opportunity to develop at different stages of their lives. • A digital resilience framework and toolkit to help families, educators, policymakers, frontline service workers and the industry better support users online, across a wide range of harms.

Existing initiatives to tackle online harms: WePROTECT Global Alliance - Box 21 The WePROTECT Global Alliance (WPGA) was established in recognition that CSEA is a global crime requiring a global response. The UK government played a key role in establishing WPGA and is its sole financial donor. WPGA aims to protect more children, apprehend more perpetrators of abuse and make the internet free from child sexual exploitation. Eighty-five countries are members of WPGA, along with 20 global technology companies and 25 leading non-governmental organisations. The success of the UK government funded WPGA is that it has brought together government, law enforcement, industry and civil society to take a stand against online child sexual exploitation.

2.11. In the Government Response to the Internet Safety Strategy Green Paper consultation, we noted that only a relatively small group of the larger companies are engaged with the government’s work on online safety, even though online harms can and do occur across many websites. There is also a wide variation in the extent, efficacy and pace of actions by companies to tackle online harms. Some companies rely on user moderation to oversee reported violations of their terms and conditions, such as Reddit; others employ teams of moderators or deploy technology to monitor content, such as Facebook.

2.12. Many companies claim to hold a strong track record on online safety but there is limited transparency about how they implement or enforce their policies, and there is a persistent mismatch with users’ experiences – 70% of Britons believe that social media companies do not do enough to prevent illegal or unethical behaviours on their platforms32. 60% of respondents to our Internet Safety Strategy Green Paper consultation had witnessed inappropriate or harmful behaviour online; only 41% thought their reported concerns were taken seriously by social media companies33.

2.13. At present many online companies rely on using their terms and conditions as the basis by which to judge complaints. In practice however, companies’ terms and conditions are often difficult for users to understand, and safety policies are not consistent across different platforms, with take-down times, description of harms and reporting processes varying. A series of investigations have highlighted the risk of serious shortcomings in the training, working conditions and support provided for content moderators34.

2.14. There is no mechanism to hold companies to account when they fail to tackle breaches. There is no formal, wide-reaching industry forum to improve coordination on terms and conditions. The absence of clear standards for what companies should do to tackle harms on their services makes it difficult for users to understand or uphold their rights.

2.15. The government believes that voluntary efforts have not led to adequate or consistent steps to protect British citizens online. As highlighted above, users’ own experiences confirm a sense of vulnerability online.

An international approach

2.16. The threat posed by harmful and illegal content and activity online is a global one, and many of our international partners are also developing new regulatory approaches to tackle online harms. Box 22 sets out what some other countries are doing in this area.

International approaches to countering online harms - Box 22 Germany adopted its Network Enforcement Act (‘NetzDG’) in 2017. This law requires online platforms with more than two million registered users in Germany to remove ‘manifestly unlawful’ content, which contravenes specific elements of the German criminal code, such as holocaust denial and hate speech, within 24 hours of receiving a notification or complaint, and to remove all other ‘unlawful’ content within seven days of notification. Non-compliance risks a fine of up to €50 million. This law also seeks to increase platform responsibility through imposing greater transparency and significant reporting obligations. Australia established an eSafety Commissioner through its Enhancing Online Safety for Children Act in 2015. The eSafety Commissioner is responsible for promoting online safety for all Australians. As well as offering a complaints service for young people who experience serious cyber bullying, its remit includes identifying and removing illegal online content and tackling image-based abuse. The European Commission, led by DG JUST, published in September 2018 a proposal on preventing the dissemination of terrorist content online – Member States agreed a Council version of the text in December 2018. The aim of the proposal is to ensure a consistent approach across industry to the removal of online terrorist content by Hosting Service Providers, for example social media platforms and video sharing sites. There are similarities in the approach taken to the framework proposed in this White Paper – as currently drafted it looks to take a proportionate approach to setting requirements, introduce duties of care on companies, and implementing a transparency framework. Over 2018, the EU Commission, led by DG CNECT, also published its Action Plan against Disinformation. The Commission collaborated with companies including Facebook, Google and Twitter to produce a code of practice against disinformation. This resulted in commitments to improve the transparency of political advertising, prevent the misuse of automated bots and to invest in tools to amplify diverse perspectives. The UK government has previously indicated its support for the measures and will continue to collaborate internationally on this issue.

2.17. The government is working closely with international partners as we develop our own approach that reflects our shared values and commitment to a free, open and secure internet. The approach proposed in this White Paper is the first attempt globally to tackle this range of online harms in a coherent, single regulatory framework. We will continue to share experiences and seek to work with international partners. Further details are set out in Chapter 6.

Existing initiatives to tackle online harms: Project Arachnid - Box 23 The government has invested £600,000 into Project Arachnid, a groundbreaking project that trawls the web to identify web pages with suspected child sexual abuse material. The technology can be deployed across websites, forums, chat services and newsgroups to instantaneously detect illegal content, before sending a take-down notice to service providers so they can quickly protect children from further exploitation. Project Arachnid is the product of a partnership with the Canadian Centre for Child Protection and the US National Center for Missing and Exploited Children, demonstrating the UK’s determination to work with international partners to tackle harmful activity online. To date, Arachnid has trawled 1.5 billion webpages, detected 7.5 million suspected images of child sexual abuse and issued more than 1 million take-down notices for the removal of child abuse material on the open web (live dashboard data, data taken on 21 March 2019).

PART 2: Regulatory model

3. A new regulatory framework

Summary • The government will establish a new statutory duty of care to make companies take more responsibility for the safety of their users and tackle harm caused by content or activity on their services. Compliance with this duty of care will be overseen and enforced by an independent regulator. • Companies must fulfil their new legal duties. The regulator will set out how to do this in codes of practice. If companies want to fulfil these duties in a manner not set out in the codes, they will have to explain and justify to the regulator how their alternative approach will effectively deliver the same or greater level of impact. • Regarding the threat to national security or the physical safety of children, the government will have the power to direct the regulator in relation to codes of practice relating to terrorist activity or CSEA online, and these codes must be signed off by the Home Secretary. • For all codes of practice relating to illegal harms, including incitement of violence and the sale of illegal goods and weapons, there will be a clear expectation that the regulator will work with law enforcement and other relevant government agencies to ensure the codes adequately keep pace with the threat. • Developing a culture of transparency, trust and accountability will be a critical element of the new regulatory framework. The regulator will have the power to require annual transparency reports from companies in scope, outlining the prevalence of harmful content on their platforms and what measures they are taking to address this. These reports will be published online by the regulator, so that users and parents can make informed decisions about online use. The regulator will also have powers to require additional information, including about the operation of algorithms. • The regulator will encourage and oversee the fulfilment of companies’ commitments to improve the ability of independent researchers to access their data, subject to appropriate safeguards. • As part of the new duty of care, we will expect companies, where appropriate, to have an effective and easy-to-access user complaints function. The regulator will require companies to respond to user complaints within an appropriate timeframe and to take action consistent with the expectations set out in the regulatory framework. • But we also recognise the importance of an independent review mechanism to ensure that users have confidence that their concerns are being treated fairly. We are consulting on allowing designated bodies to make ‘super complaints’ to defend the needs of users. • Ahead of the implementation of the new regulatory framework, we will encourage companies to take early action to address online harms. To assist this, the White Paper sets out high-level expectations of companies, including some specific expectations in relation to certain harms. We expect the regulator to reflect these in future codes of practice. • Where there is a threat to national security or the physical safety of children, such as CSEA and terrorism, we will expect companies to go much further and demonstrate the steps taken to combat the dissemination of associated content and illegal behaviours. We will publish interim codes of practice providing guidance about tackling terrorist activity and online CSEA later this year.

3.1. The government will establish a new statutory duty of care on relevant companies to take reasonable steps to keep their users safe and tackle illegal and harmful activity on their services.

3.2. The fulfilment of this duty will be overseen and enforced by an independent regulator.

3.3. This statutory duty of care will require companies to take reasonable steps to keep users safe, and prevent other persons coming to harm as a direct consequence of activity on their services. This broader application of the duty, beyond simply users of a particular service, recognises that in some cases the victims of harmful activity – victims of the sharing of non- consensual images, for example – may not themselves be users of the service where the harmful activity took place. This duty will apply to all of the harms included in the scope of the White Paper, as set out below.

3.4. A key element of the regulator’s approach will be the principle of proportionality. Companies will be required to take action proportionate to the severity and scale of the harm in question. The regulator will be required to assess the action of companies according to their size and resources, and the age of their users.

3.5. The regulatory approach will impose more specific and stringent requirements for those harms which are clearly illegal, than for those harms which may be legal but harmful, depending on the context.

3.6. Companies must fulfil their new legal duties. The regulator will set out how to do this in codes of practice. The codes will outline the systems, procedures, technologies and investment, including in staffing, training and support of human moderators, that companies need to adopt to help demonstrate that they have fulfilled their duty of care to their users.

Companies will still need to be compliant with the overarching duty of care even where a specific code does not exist, for example assessing and responding to the risk associated with emerging harms or technology.

3.7. There will be a strong expectation that companies follow the guidance set out in these codes. If they choose not to do so, companies will have to explain and justify to the regulator how their alternative approach will effectively deliver the same or greater level of impact. This approach is familiar to companies, for example the UK Corporate Governance Code35 and the ICO’s code of practice on data sharing. Though these codes will be developed with the companies and other stakeholders in an open and transparent way, the regulator will ultimately decide on their content.

3.8. The regulator will assess whether companies have fulfilled their duty of care, including by reference to relevant codes of practice, and compliance with the company’s own relevant terms and conditions. Failure to meet these obligations may result in enforcement action by the regulator. Further details on enforcement are in Chapter 6.

3.9. The regulator will also expect companies to make clear how they are fulfilling their statutory duty of care. Relevant terms and conditions will be required to be sufficiently clear and accessible, including to children and other vulnerable users. The regulator will assess how effectively these terms are enforced as part of any regulatory action.

Terrorism and CSEA online

3.10. Companies will be required to take particularly robust action to tackle terrorist use of the internet and online CSEA. The government will have the power to issue directions to the regulator regarding the content of the codes of practice for these harms, and will also approve the draft codes before they are brought into effect. Similarly, the regulator will not normally agree to companies adopting proposals which diverge from these two codes of practice, and will require a high burden of proof that alternative proposals will be effective.

3.11. Between the publication of this White Paper and the establishment of a regulator, the government will work with law enforcement and other relevant bodies to produce interim codes of practice for online terrorist content and CSEA. These codes will be published later this year.

General monitoring

3.12. The regulator will not compel companies to undertake general monitoring of all communications on their online services, as this would be a disproportionate burden on companies and would raise concerns about user privacy. The government believes that there is however, a strong case for mandating specific monitoring that targets where there is a threat to national security or the physical safety of children, such as CSEA and terrorism.

Transparency, trust and accountability

3.13. Developing a culture of transparency, trust and accountability, and consistent standards of transparency, will be a critical element of the new regulatory framework.

3.14. In May 2018, the Government Response to the Internet Safety Strategy Green Paper consultation set out the role transparency and reporting must play in building our understanding of the extent of online harms and how effectively companies are tackling breaches in their terms and conditions.

3.15. Alongside this response, we published a draft transparency reporting template and began a series of engagements with industry. This process, which has included discussion with over 20 companies, has provided some helpful insights into current industry action. It is encouraging that more companies have since started publishing their own global transparency reports. We will publish the government’s first annual transparency report later this year.

3.16. At the same time, we indicated that transparency reporting was one of the potential areas for new legislation. Greater transparency will ensure:

• The regulator can gain an understanding of the level of harms on online platforms and the mitigating action being taken by companies. This will inform its regulatory priorities and determine the effectiveness of, and compliance with, different regulatory measures.

• Users can gain a greater understanding and awareness of whether and to what extent companies are taking positive steps to keep their users safe, and the processes different companies have in place to prevent harms.

• Companies take responsibility for the impacts of their platforms and products on their users. It will incentivise accountability within the industry.

3.17. To inform its reports and to guide its regulatory action, the regulator will have the power to require annual reports from companies covering the following areas:

• Evidence of effective enforcement of the company’s own relevant terms and conditions, which should reflect guidance issued by the regulator in its codes of practice.

• Processes that the company has in place for reporting illegal and harmful content and behaviour, the number of reports received and how many of those reports led to action.

• Proactive use of technological tools, where appropriate, to identify, flag, block or remove illegal or harmful content.

• Measures and safeguards in place to uphold and protect fundamental rights, ensuring decisions to remove content, block and/or delete accounts are well- founded, especially when automated tools are used and that users have an effective route of appeal.

• Where relevant, evidence of cooperation with UK law enforcement and other relevant government agencies, regulatory bodies and public agencies.

• Details of investment to support user education and awareness of online harms, including through collaboration with civil society, small and medium sized enterprises (SMEs) and other companies.

3.18. The regulator will produce and publish an annual transparency report outlining key data on companies’ performance against their duty of care and the prevalence of harms on different platforms. It will also publish companies’ transparency reports on its website, ensuring these are easily accessible to the public so that users and parents can make informed decisions about online use. Where the regulator has required companies to produce transparency reports, it will be mandatory to provide them; failure to do so will result in enforcement action (as set out in Chapter 6).

3.19. The regulator will use insight from users, civil society, government, law enforcement and other relevant government agencies, and other regulators to inform its understanding of the prevalence and impact of online harms, and the effectiveness of companies’ responses.

3.20. As well as the power to require annual reports from companies, the regulator will have the power to require additional information from them to inform its oversight or enforcement activity, and to establish requirements to disclose information. It may also undertake thematic reviews of areas of concern, for example a review into the treatment of self-harm or suicide related content. The regulator will have the power to require companies to share research that they hold or have commissioned that shows that their activities may cause harm.

3.21. The regulator will build on government engagement with companies to understand how best to establish comparable data-points and reporting between platforms.

3.22. As part of a movement towards greater transparency, companies should also work in conjunction with the regulator to build a shared understanding of the mechanics of their associated platforms or services. Where necessary, to establish that companies are adequately fulfilling the duty of care, the regulator will have the power to request explanations about the way algorithms operate. The regulator may, for example, require companies to demonstrate how algorithms select content for children, and to provide the means for testing the operation of these algorithms.

3.23. In determining where such explanations will be appropriate and what form they should take, the regulator will work closely with the Centre for Data Ethics and Innovation, the expert body that has been set up to advise government on the