To unlock access to multitudes of unheard stories, IU Libraries will work extensively with project partners at University of Texas at Austin, New York Public Library and information management consultant AVP. Together the team will develop and test a scalable Audiovisual Metadata Platform, known as AMP, to generate searchable time-stamped descriptions for audiovisual content.

BLOOMINGTON, Ind. -- Current access to text-based data and knowledge is unprecedented in human history. Now, enabled by a $1.2 million grant from The Andrew W. Mellon Foundation, Indiana University Libraries is setting its sights on the next research frontier: hundreds of millions of hours of audiovisual content.

"With any sort of mass digitization, the item does not speak for itself," said Mary Kidd, coordinator of systems and operations in the Department of Preservation and Collections Processing at New York Public Library. In recent years, the library has preserved nearly 80,000 audiovisual items through digitization.

The barriers created by the time-consuming, expensive process of describing audio and video solely through manual means create a major impediment to discovery and use, even as nationwide preservation efforts exponentially expand available content. In fact, Indiana University is home to one of the country's most ambitious digitization efforts, the Media and Digitization Preservation Initiative . Library leaders at IU and across the country are actively seeking ways to accelerate access to the hundreds of thousands of newly digitized resources.

Online aggregations of library, museum and archival collections, such as the Digital Public Library of America, are primarily composed of images and text, while less than half a percent of available content is video or sound, Dunn and his partners said.

"Discovery opportunities for recorded audio or moving images are extremely limited today," said Jon Dunn, principal investigator and assistant dean for library technologies at IU. "Our work will generate descriptions, or what is known as metadata, through a combination of expert human labor and new uses of automated processing technology."

Traditional approaches to metadata generation for audiovisual objects must rely almost exclusively on labor by experts who type information into a database or create a written finding aid.

"The ultimate result of the AMP project will be a metadata-generation interface made openly available to collection managers around the world," said Chris Lacinak, president and founder of AVP, the consulting and software development firm working with grant partners on AMP development.

Kidd sees great potential in a tool that supports human experts.

"What is so exciting about this project, and not reflected in work anywhere else, is that AMP uses a variety of automation tools that create a dynamic set of data we can then synthesize into our records," she said.

The automation will include artificial intelligence and machine-learning-based functionality such as natural language processing, speech-to-text conversion, facial recognition, scene and music detection, object recognition and more. The AMP team plans to incorporate automated tools from both the research community and from commercial providers such as Amazon and Google.

Over the 27-month grant period, different forms of audiovisual content will be used to test and refine the technology. Examples of potential collection use cases include the Joffrey Ballet Archive at New York Public Library and documentation of field collections at IU Libraries' Archives of Traditional Music.

The project kicks off by learning from prior efforts to automate audiovisual metadata though consultation with the Fraunhofer Institute for Digital Media Technology IDMT in Germany, as well as the foundational work of the High Performance Sound Technologies for Access and Scholarship project at the University of Texas, led by AMP partner Tanya Clement. Previous Mellon Foundation support for AMP instigated these connections through community development activities and documentation of workshops and assessment activities.

"From early in the 20th century and continuing through the present, audiovisual media contain vital and irreplaceable evidence that documents contemporary society and culture," Mellon Foundation senior program officer Donald J. Waters said. "Future understanding of the 20th century and beyond requires an aggressive investment today in tools and methods for preserving and providing research access to these media."

Additional quotes

"Lack of description and rights management documentation stand in the way and severely limit the value of audiovisual collections to our researchers. AMP addresses discoverability in a way that will open up this emerging field of study." -- Carolyn Walters, Ruth Lilly Dean of University Libraries at Indiana University

"Scholars, teachers and the general public tend to engage with what they can easily find online. AMP has the potential to change how and if we use audio and video from these important humanities audio and video collections in our research, classrooms and everyday understanding of our culture." -- Tanya Clement, associate professor, Department of English, University of Texas at Austin

"It is key to start a project in this domain with a detailed understanding of the content and the queries to be supported, to consider the user perspective and to have this understanding as early as possible." -- Uwe Kühhirt, head of Metadata Department, Fraunhofer Institute for Digital Media Technology IDMT



About IU Libraries

Indiana University Libraries, one of the nation's largest public academic research libraries, partners with every academic department on the IU Bloomington campus. Materials are digital, visual, audio and print. Over 60,000 journals are offered electronically, and the libraries holds over 8 million print volumes in 450 languages, and 120,000 films in its Moving Image Archive. As co-leaders of one of the country's most ambitious digitization efforts, IU's Media and Digitization Preservation Initiative, IU Libraries is celebrating digital preservation of 308,706 total items as of summer 2018, including 1,700 hours of moving images from film.

About The Andrew W. Mellon Foundation

The Andrew W. Mellon Foundation endeavors to strengthen, promote and, where necessary, defend the contributions of the humanities and the arts to human flourishing and to the well-being of diverse and democratic societies. To this end, it supports exemplary institutions of higher education and culture as they renew and provide access to an invaluable heritage of ambitious, path-breaking work.