Abstract

Today, companies aggregate, trade, and utilize personal information at unprecedented levels. Their unilateral and extensive access to data about the characteristics, behaviors, and lives of billions allows them to constantly monitor, follow, judge, sort, rate, and rank people as they see fit. Our previous report documented the massive scale and scope of today’s networks of digital tracking and profiling. It investigated relevant industries, business models, platforms, services, devices, technologies, and data flows, focusing on their implications for people – whether as individuals, consumers, or citizens – and society at large.

This working paper examines how the corporate use of personal information can affect individuals, groups of people, and society at large, particularly in the context of automated decisions, personalization and data-driven persuasion. After briefly reviewing our previous research’s findings and key developments in recent years, this paper explores their potential to be used against people in detail.

Systems that make decisions about people based on their data produce substantial adverse effects that can massively limit their choices, opportunities, and life-chances. These systems are largely opaque, nontransparent, arbitrary, biased, unfair, and unaccountable – even in areas such as credit rating that have long been regulated in some way. Through data-driven personalization, companies and other institutions can easily utilize information asymmetries in order to exploit personal weaknesses with calculated efficiency. Personalized persuasion strategies provide the means to effectively influence behavior at scale. As companies increasingly and unilaterally shape the networked environments and experiences that underlie and determine everyday life, manipulative, misleading, deceptive, or even coercive strategies can be automated and customized down to the individual level.

Based on the examination of business practices and their implications we conclude that, in their current state, today’s commercial networks of digital tracking and profiling show a massive potential to limit personal agency, autonomy, and human dignity. This not only deeply affects individuals, but also society at large. By improving the ability to exclude or precisely target already disadvantaged groups, current corporate practices utilizing personal information tend toward disproportionally affecting these groups and therefore increase social and economic inequality. Especially when combined with influencing strategies derived from neuroeconomics and behavioral economics, data-driven persuasion undermines the concept of rational choice and thus the basic foundation of market economy. When used in political campaigns or in other efforts to shape public policy, it may undermine democracy at large.

While this working paper does not directly offer solutions, it examines, documents, structures, and contextualizes today’s commercial personal data industries and their implications; further research will build on this basis. Hopefully, it will also encourage and contribute to further work by others.

The production of this report was supported by the Open Society Foundations.