In a short parable titled "A Message from the Emperor," the early 20th century German-language writer Franz Kafka describes the process by which a herald relays a message from a dying Chinese emperor to the outer provinces of the empire. It turns out that the empire is so huge that the herald could spend his entire life trying to make his way to the distant hinterland.

Throughout history, China's rulers have struggled to get reliable data about its vast and diverse population. The late Chinese historian Ray Huang noted that classical China failed to develop the administrative institutions necessary to make it "mathematically manageable." Local leaders fabricated and reported rosy facts to the capital for personal ends. The inability of the state to gather reliable data precluded the development of capitalism and China's modernization.

Modern China inherited these problems. The central government is acutely aware of how little it knows about the country. A vast array of fragmented local authorities gathers data for the government, but in the bottom-up reporting process through the bureaucratic hierarchies the data inevitably become distorted for political and personal ends.

One example is false reports on food production during the so-called Great Leap Forward, instigated by Mao Zedong. In contemporary China, inflated economic data is another example. According to WikiLeaks, Li Keqiang, now China's premier, told a U.S. official in 2007 that China's gross domestic product figures are "man-made," and therefore unreliable. In January 2017, Liaoning Province Governor Chen Qiufa admitted that economic growth figures from 2011 to 2014 had been fabricated, with fiscal revenues inflated by at least 20%.

China's quest for an effective mechanism for mathematical management remains unfinished. But technology is changing that, allowing a wide range of data to be collected directly without distortions. This is made possible by the digital revolution that is in full swing in China. Every minute on the Chinese internet almost 400,000 people log onto Tencent Holdings' messaging app WeChat, 4 million search queries are made on the search engine Baidu, and $1 million is spent in Alibaba Group Holding's online marketplaces. These actions contain a wealth of information that reveals what people think, feel and consume.

Current discussions on big data in China center on its business impacts, but its potential for development and governance is beginning to be recognized. To embrace the digital era, Li and Chinese President Xi Jinping head the government's Central Leading Group for Internet Security and Informatization. China's 13th five-year plan (2016-2020) includes a national big data strategy to advance social and economic development. The real-time information streams now available offer insights on emerging concerns that are highly relevant to China's development, such as better understanding of socioeconomic trends, waste management and air pollution.

In November 2013, China's National Bureau of Statistics signed agreements with 11 major enterprises, including Alibaba and Baidu, to build long-term partnerships on big data. The collaboration with Baidu will improve the prediction model of the macroeconomy by combining big data from the internet. In the words of NBS director Ma Jiantang, official statistics should "sincerely embrace and take efforts in using big data."

The United Nations Development Program is also working with Baidu to develop a smartphone app to promote sustainable e-waste disposal practices by quantifying the environmental damage caused by incorrect disposal of electronic equipment. In 2016, the Beijing city government deployed air pollution forecasting tools incorporating more widespread sources, such as traffic systems, economic data, and even social media. These provide more accurate forecasts, and estimates of the economic consequences of interventions such as factory closures.

Digital dictatorship

Perhaps the biggest opportunity lies in the emerging internet of things -- the proposed connectivity of everyday objects. The market research firm Gartner forecasts that more than 20 billion appliances and devices will be connected to the internet globally by 2020. China is not missing the opportunity. In the lead-up to the World Telecommunication Standardization Assembly in October 2016, China advocated a new navigational and addressing technology called Digital Object Architecture, which could enable the real-time tracking of each connected device and individual.

China's big data push has triggered concerns about invasive tracking and social control. In 2015, the Chinese government proposed the creation of a comprehensive "social credit system" that would rate citizens in financial, legal and civic terms. While the details are vague, some commentators have already jumped to the conclusion that this is the onset of a digital dictatorship.

While legitimate concerns about the further coercion of citizens' rights need to be addressed, aspects of the system could improve the functioning of the market economy. For example, financial credit scoring information helps individuals and small businesses to obtain financing more easily. Information on legal compliance also acts as a proxy for the underdeveloped legal system.

This brings us to the multitude of challenges that needs to be addressed before big data's potential can be realized in China. The most important is privacy and transparency. The social credit system debate raises questions such as the extent to which individuals will be rated on the basis of information they cannot access, and what criteria will be used to decide the relevance of the information.

In addition, most valuable data is held by private companies. They need assurances before making their client data available for development purposes. A crackdown on private measurement initiatives, such as property price and air quality indexes, would discourage data-driven insights and public-private partnerships.

National data policies need to define the conditions for data gathering and sharing, establishing a regulatory environment that guarantees the legitimate interests of consumers. Admittedly, this could be difficult in China, where the rule of law is not well established. As data becomes less fragmented and more centralized, it is also important to promote safety and security.

Effective application of big data to improve governance also requires a future-proof approach to government. The government will need to process more information and respond more quickly, and traditional boundaries between government departments will blur as data from different fields combine to create new insights. This calls for an adaptive strategy that emphasizes experimentation, flexibility, and adjustment. Decision makers will need to become more sophisticated data users in identifying and understanding biases in data sources.

Big data poses both challenges and opportunities for China. If managed correctly, however, it could transform the country's Kafkaesque bureaucracy and offer a solution to a data governance problem that has lasted for thousands of years.

Andy Yee is public policy director for Visa for greater China, and previously worked for Google as an Asia-Pacific public policy analyst. This article expresses the author's personal views.