Bullfighting in Piazza San Marco by Canaletto and Cimaroli/DeAgostini/Getty; zetter/Istock/Getty

Only metres away from the tourist throngs that bustle through Venice's crowded piazzas, the silence inside Santa Maria Gloriosa dei Frari is so profound it hurts the ears. State archivists long ago took over this fourteenth-century friary, but they are just as studious as the Franciscan brothers who once lived here, as they tend the historical records that fill some 80 kilometres of shelving within. Now, a crew of scientists laden with high-tech equipment is stirring things up in these hallowed stacks.

History hangs heavy at the Frari, and computer scientist Frédéric Kaplan likes it that way. He has an ambition to capture well over 1,000 years of records in dynamic digital form, encompassing the glorious era of the Most Serene Republic of Venice. The project, which he calls the Venice Time Machine, will scan documents including maps, monographs, manuscripts and sheet music. It promises not only to open up reams of hidden history to scholars, but also to enable the researchers to search and cross-reference the information, thanks to advances in machine-learning technologies.

If it succeeds, it will pave the way for an even more ambitious project to link similar time machines in Europe’s historic centres of culture and commerce, revealing in unprecedented detail how social networks, trade and knowledge have developed over centuries across the continent. It would serve as a Google and Facebook for generations long past, says Kaplan, who directs the Digital Humanities Laboratory at the Swiss Federal Institute of Technology in Lausanne (EPFL).

Although the previous decade has seen many digital-humanities projects that scan, annotate and index manuscripts, this one stands out because of its ambitious scale and the new technologies it hopes to use: from state-of-the-art scanners that could even read unopened books, to adaptable algorithms that will turn handwritten documents into digital, searchable text.

The boon for scholarship should extend well beyond historians. Economists and epidemiologists, for example, are eager to access the written records left by tens of thousands of ordinary citizens, which could reveal how financial markets developed or how diseases such as the plague spread. “We are in a state of electrified excitement about the possibilities,” says Lorraine Daston, a director of the Max Planck Institute for the History of Science in Berlin. “I am practically salivating.”

The Serene Republic

Venice is the perfect city for the experiment because of its wealth of historically important, well-ordered documentation. It was founded in the fifth century AD by citizens of the Roman empire escaping barbarian invaders from the north. Its inhospitable lagoons provided much-needed protection, and its location at the north end of the Adriatic Sea also had strategic advantages. It soon became the most important trading post between Western Europe and the east, bringing it riches and power.

As Venice’s empire grew, it developed administrative systems that recorded vast amounts of information: who lived where, the details of every boat that entered or left the harbour, every alteration made to buildings or canals. Modern banking was invented in the Rialto, one of Venice’s oldest quarters, and notaries there recorded all trading exchanges and financial transactions.

Crucially, those records survived through turbulent centuries. While the rest of Europe was roiled by its perpetually warring monarchs, from the eighth century onwards Venice began to develop into a stable republic that provided the peace and order required for trade to flourish. In many ways it was a model democracy. The people elected a leader — the doge — supported by various councils, whose members were also usually elected. Governance was secular, but for the most part co-existed tolerantly with religion.

“We are in a state of electrified excitement about the possibilities.”

French emperor Napoleon Bonaparte put an end to the Serene Republic in 1797. En route to Vienna during his attempt to conquer the Austro-Hungarian Empire, he declared Venice’s secular and democratic governance to be a form of autocracy, and the city to be an enemy of the revolution. He forced the republic to dissolve itself. In 1815, the old Frari was turned into the State Archives of Venice.

Over the next decades, all state administrative documents, including death registers, were transferred there, along with medical records, notary records, maps and architectural plans, patent registers and a miscellany of other documentation, some from elsewhere in Italy. Particularly significant are ambassadors' reports from wider Europe and the Ottoman Empire, providing a unique source of detailed information about daily life. “Venetian ambassadors were the most observant travellers, trained to find out things like what was being unloaded at the docks, or what a prince or other high-up was like as a person,” says Daston. “Their reports were full of gossip and intrigue.”

Most of the archive, predominantly written in Latin or the Venetian dialect, has never been read by modern historians. Now it will all be systematically fed into the Venice Time Machine, along with more unconventional sources of data, such as paintings and travellers’ logs.

Birth of a career

Kaplan has spent his career applying artificial intelligence (AI) in the humanities, mostly in linguistics. He has modelled the evolution of language, for example, by using AI to search centuries of newspaper reports for patterns of words and phrases. But he had always yearned to apply these techniques to building a time machine in a European city with a couple of centuries' worth of archives. His thoughts first turned to Paris, Amsterdam or Geneva, Switzerland. But when the rectors of EPFL and the Ca’Foscari University of Venice decided to collaborate and called for ideas, he immediately offered to develop his time-machine idea for Venice. He vividly recalls the first time he entered the archives, in 2012. Time stands still in the warren of more than 300 rooms, which are neither air-conditioned nor heated. In winter they are bitingly cold, in summer stiflingly hot. The fragile documents are stacked floor to ceiling, and occasional flakes of yellowing paper drift down from their edges. “I felt completely overwhelmed,” he says. “Seeing what a thousand-year archive looks like, knowing that most of it was not available — I knew we needed to do it.”

When the project officially launched in 2012, Kaplan knew that it demanded much more than his computational prowess. It would need historians to annotate the manuscripts, to provide the necessary context for data handling. They might note the role of each person mentioned in a contract to clarify exactly who was the recipient, for example, or assess the reliability of a particular source of information. Archivists, too, would be needed, for their deep knowledge of the immense collection of documents.

Hillary Sanctuary/EPFL/Archivio di Stato

So he engaged as his co-director Isabella di Lenardo, a Venice-trained historian who is now at EPFL. She had no hesitation: “It was what I had been waiting for all my life.” Venice state archivists, used to the old ways of guardianship, took a little longer to buy in to the idea, but within a year they were full partners.

The interdisciplinary collaboration immediately started to harvest the sort of obscure archival knowledge that tends not to penetrate to the outside world. For example, although ambassadorial reports are a particularly rich source of detail, they were often written in code to keep the messages secret, a source of frustration for historians. Yet one casual conversation among team members led to the serendipitous discovery of a small, sixteenth-century book called Libro de le cifre, which provided the encryption code for some Venetian ambassadors’ reports. Historians are now eagerly preparing to decode its secrets.

Scanning history

Even before the Venice Time Machine arrived, the State Archives had started a digitization project funded by the Italian Ministry of Cultural Heritage. In 2006, a huge, purpose-built scanner began to digitize the archive’s precious store of more than 3,000 maps of Italian towns, including many commissioned by Napoleon. These ‘cadastral’ maps delineate property boundaries and record the ownership of small parcels of land; some of the documents are as large as 4 metres by 7 metres.

The Venice Time Machine has shifted this process into overdrive, bringing in other state-of-the-art, high-speed scanners specially adapted for the project. They include one with a robotic arm to turn the pages of books and an imposing rotary scanner with a 2-metre-wide turntable that allows technicians standing on opposite sides to feed it multiple A3-sized documents at the same time. These scanners now form a pipeline that produces several thousand high-definition images per hour, feeding terabytes of information to servers in Venice for long-term storage, and to Lausanne, where high-performance computers transform the images into digital text ready for annotation.

The automatic reading of old handwritten manuscripts is a major challenge. Standard character-recognition software allows printed books to be read letter by letter despite variations in fonts, and thus rendered searchable. But this doesn’t work for handwritten manuscripts, where shapes of individual letters can vary enormously between scribes, and can evolve over time. Various approaches to solving the problem are being developed in a European Union collaboration called Recognition and Enrichment of Archival Documents (READ). Kaplan, a member of the collaboration, is currently applying his preferred approach to the Venice Time Machine, using machine learning to recognize the shapes of whole words.

Machine learning relies on algorithms that modify their own rules and behaviour as they harvest examples from data sets, honing their skills with every new experience. The time machine’s algorithms are designed to analyse the structure of written text and pull out graphical shapes that look similar, forming a link between them (see ‘Hacking history’). That allows a user to find a name in one document, and then ask the system to reveal where the same name appears in all the other manuscripts in the database.

In the next decade, these scanners could be joined by an instrument that reads books without even opening them. Now being developed at EPFL, the concept is based on computed tomography (CT) scanning techniques used in medicine, where X-ray images taken at different angles build up a 3D picture of the inside of a body, slice by slice. EPFL scientists are researching the composition of ancient inks to identify molecules that could act as X-ray contrast agents. “It may need more than five years before the tomography scanner can be put into operation,” says Kaplan. But it would offer huge advantages: it could scan books much faster, probe delicate volumes without damage, and access the hundreds of thousands of fragile sealed wills in the Venice archives that would be destroyed if opened.

Social networks

Even as these technologies are being developed and refined, the Venice Time Machine is already demonstrating how it can help to reshape scholars’ understanding of the past. The narratives that fill history textbooks are usually built around famous people, because so much more is known about them. Yet the time machine will bulge with the sort of mundane records that state administrators everywhere routinely gather to keep track of their populations. This will enable historians to reconstruct the lives of hundreds of thousands of ordinary people — artisans and shopkeepers, envoys and traders — and build much more rounded historical narratives.

Hillary Sanctuary/EPFL/Archivio di Stato

Napoleon’s efficient approach to state administration has been particularly valuable to the project. One cadastral map of Venice that he commissioned in 1808 has provided a backbone of reliable data, allowing historians to add geographical context to a 1740 census that lists citizens who owned and rented property in the city. By combining this with 3D information about buildings from paintings such as those of Canaletto, the time-machine team has produced an animated tour through Venice, showing which businesses were active in each building at the time. “Napoleon may have brought the Republic of Venice to an end,” says Kaplan, “but for us, he was the starting point for the recovery of its history.”

Kaplan and di Lenardo have also made a series of other animations of Venice over space and time, which will be updated and enriched as more data feed into the machine. One is a dynamic video of the development of the Rialto from AD 950 onwards, using diverse sources of information at different time points. The simulation shows how the buildings — and the iconic Rialto Bridge — sprung up among the salt marshes, along with the area’s periodic destruction by fires and subsequent reconstructions.

Other simulations tag buildings in the Rialto with the names of family businesses, or depict the social networks that formed between Venetians and others throughout Europe. The Venice Time Machine assumes that there is a connection between people whose names appear in the same document, and this allows it to show each person as a node in a web of connections. When the same individuals crop up in other documents, the web begins to grow into a giant network — just as scientists draw up social networks from Facebook or Twitter data. This network should allow historians to discover details about the lives of large numbers of previously unknown people in Venice and beyond, and their place in society.

Units, banking and plague

Daston thinks that the time machine could help to answer an almost endless list of historical questions. For example, it could show how language developed to describe the strange animal species brought to the Venice docks from newly discovered countries, or it could track the trajectories of scholars and scientists as they wandered across Europe.

Her personal passion is the epistemology of measurement. “Everyone was crazy about measuring the world in the seventeenth century, yet units of measurement were barely mentioned in the fifteenth and sixteenth centuries,” she says. “Being able to do keyword searches over the centuries could help us understand how the science of measuring became established.”

That enthusiasm spills over from history to other fields. Economic historian Joan Rosés of the London School of Economics and Political Science says that centuries of searchable data from notaries in a city as important to economic history as Venice “could help change our understanding of how financial markets work”. Much of economic theory was developed without hard data, he says, and economists seeking a more robust evidence base are stymied by a lack of suitable data sets on things such as transactions and the flow of money. Modern records, including those from banks, are of limited value: the data have already been processed according to the economic theory to which the institution subscribes. Historical data sets are cleaner because they record raw, intuitive behaviour — simply who sold what, for how much. But the large financial archives of Europe, such as the Notarial Archive of Catalonia in Barcelona, are not online. “When I go to the Barcelona archive for research, I may read just three documents a day,” says Rosés, “so the Venice Time Machine will be a game-changer.” And there is much to be learnt from people who were economic failures. “You can deduce a lot of stupid things if you only study successful, famous people — the only people that we know a lot about,” he says.

“The Venice Time Machine will be a game-changer.”

Epidemiologist Marcel Salathé of EPFL is already collaborating with the Venice Time Machine, peering into records that reveal the names and locations of people who died, often with details about the circumstances of their demise. “It is like a primitive electronic health record,” he says. Plague wiped out one-third of the population of Venice in the mid-seventeenth century, and Salathé hopes to discover more about how the disease spreads. Outbreaks still occur around the world, but there are big gaps in data about its transmission. Animal research alone cannot fill them, and modern human data sets are too small to help, he says.

Kaplan hopes that Venice is just a starting point. The Venice Time Machine has applied, with partners around Europe, to become one of the next billion-euro flagship programmes funded by the European Union. If it wins, it will create time machines in other cities with similarly important archives, and link them together. Earlier this year a consortium of Dutch academics launched the Amsterdam Time Machine, although it has yet to secure funding. Its coordinator — Julia Noordegraaf of the University of Amsterdam, who studies the history of the creative industries — says it is “a great opportunity to study the cultural traffic between Amsterdam and Venice during its golden age in the seventeenth century”. A Paris Time Machine is also under discussion.

The unbridled ambitions of the time-machine project are a concern for some researchers, not least because many of its core technologies are still being developed. “The vision of extending digital representation into different time slots is absolutely, self-evidently right — but it might be better to develop things more in a lot of different, small projects,” says Jürgen Renn, a digital-humanities pioneer and a director at the Max Planck Institute for the History of Science.

Nevertheless, Daston suspects that the time machine heralds a new era of historical study. “We historians were baptized with the dust of archives,” she says. “The future may be different.”