The event was livestreamed.

Organizers:

The workshop was organized by Sanjeev Arora (IAS/Princeton University), Joan Bruna (IAS/NYU), Rong Ge (IAS/Duke), Suriya Gunasekar(IAS/Toyota Technical Institute), Jason Lee (IAS/USC), Bin Yu (IAS/UC Berkeley)

This workshop sought to bring together deep learning practitioners and theorists to discuss progress that has been made on deep learning theory, and to identify promising avenues where theory is possible and useful. There were several invited talks each day and also spotlight talks by young researchers.

The workshop was free of charge thanks to the support from the Institute for Advanced Study and the Schwab Charitable Fund made possible by the generosity of Eric and Wendy Schmidt.

Invited Speakers who confirmed participation:

Anima Anandkumar, Raman Arora, Sanjeev Arora, Mikhail Belkin, Léon Bottou, Joan Bruna, Michael Collins, Simon Du, Gintare Karolina Dziugaite, Surya Ganguli, Rong Ge, Suriya Gunasekar, Stefanie Jegelka, Chi Jin, Sham Kakade, Yann LeCun, Jason Lee, Ke Li, Tengyu Ma, Aleksander Madry, Chris Manning, Behnam Neyshabur, Dan Roy, Nathan Sbrero, Rachel Ward, Bin Yu

Registration: was required

If you have any problems with registration or other practical questions, please write to Michelle Huguenin (huguenin@ias.edu).

Registration for the meals and workshop are required. (Speakers do not need to register)

Although there is no registration fee, because seating is limited both in the seminar room and in our dining hall, you are required to register if you wish to attend all or part of the workshop. In addition, in order for our chef to prepare enough food, we need to have a headcount for the meals. Please note the workshop attendees are expected to pay for their lunch. Approximate range of prices are 4.00 for a salad to 9.00 for an entree. Vegetarian lunches will be available. Only cash is accepted at the cash registers. Credit cards and personal checks are not accepted.

Agenda:(all talks were in Wolfensohn Hall)

October 15, 2019

REGISTRATION: Opening at 8:30 am - 9:30 am in Wolfensohn Hall and will remain open until 3 pm

Workshop on Theory of Deep Learning: Where next? Topic: Emergent linguistic structure in deep contextual neural word representations slides video Speaker: Chris Manning, Stanford University Time/Room: 9:30am - 10:10am/Wolfensohn Hall Topic: Explaining landscape connectivity of low-cost solutions for multilayer nets video Speaker: Rong Ge, Duke University; Member, School of Mathematics Time/Room: 10:10am - 10:40am/Wolfensohn Hall Topic: Fixing GAN optimization through competitive gradient descent video Speaker: Anima Anandkumar, Caltech Time/Room: 11:10am - 11:50am/Wolfensohn Hall Topic: Tightening information-theoretic generalization bounds with data-dependent estimates with an application to SGLD video Speaker: Daniel Roy, University of Toronto Time/Room: 11:50am - 12:20pm/Wolfensohn Hall Topic: Spotlight Talks: Yuanzhi Li, Soham De, Mahyar Fazlyab, Maithra Raghu, Valentin Thomas video Speaker: Various Time/Room: 12:20pm - 1:00 pm Topic: Is optimization the right language to understand deep learning? video Speaker: Sanjeev Arora, Princeton University; Distinguished Visiting Professor, School of Mathematics Time/Room: 2:30pm - 3:10pm/Wolfensohn Hall Topic: Spotlight Talks: Amir Asadi, Dimitris Kalimeris video Speaker: Various Time/Room: 3:10pm - 3:40pm/Wolfensohn Hall Topic: PAC-Bayesian approaches to understanding generalization in deep learning video slides Speaker: Gintare Karolina Dziugaite, Simons Institute for the Theory of Computing Time/Room: 4:00pm - 4:30pm/Wolfensohn Hall Topic: Overcoming the Curse of Dimensionality and Mode Collapse video slides Speaker: Ke Li, University of California, Berkeley Time/Room: 4:30pm - 5:00pm/Wolfensohn Hall Topic: Are All Features Created Equal? video slides Speaker: Aleksander Madry, Massachusetts Institute of Technology Time/Room: 5:00pm - 5:40pm/Wolfensohn Hall

October 16, 2019

Topic: Energy-based Approaches to Representation Learning slides video Speaker: Yann LeCun, NYU and Facebook AI Time/Room: 9:30am - 10:10am/Wolfensohn Hall Topic: On Large Deviation Principles for Large Neural Networks video slides Speaker: Joan Bruna, New York University Time/Room: 10:10am - 10:40am/Wolfensohn Hall Topic: Neural Models for Speech and Language: Successes, Challenges, and the Relationship to Computational Models of the Brain video Speaker: Michael Collins, Columbia University Time/Room: 11:10am - 11:50am/Wolfensohn Hall Topic: On the Connection between Neural Networks and Kernels: a Modern Perspective video Speaker: Simon Du, Member, School of Mathematics Time/Room: 11:50am - 12:20pm/Wolfensohn Hall Topic: Hike in Institute Woods 12:20 pm-1:00 pm Topic: From Classical Statistics to Modern ML: the Lessons of Deep Learning video slides Speaker: Mikhail Belkin, Ohio State University Time/Room: 2:30pm - 3:10pm/Wolfensohn Hall Topic: Spotlight Talks: Vaishnavh Nagarajan, Preetum Nakkiran, Xiaowu Dai, Weijie Su video Speaker: Various Time/Room: 3:10pm-3:40pm/Wolfensohn Hall Topic: Towards a theoretical foundation of neural networks video Speaker: Jason Lee, Princeton University; Member, School of Mathematics Time/Room: 4:00pm - 4:30pm/Wolfensohn Hall Topic: Panel Session Time/Room: 4:30pm - 5:30pm/Wolfensohn Hall

October 17, 2019

Topic: Learning Representations Using Causal Invariance video Speaker: Leon Bottou, Facebook AI Research Time/Room: 9:30am - 10:10am/Wolfensohn Hall Topic: Understanding the inductive bias due to dropout video Speaker: Raman Arora, Johns Hopkins University; Member, School of Mathematics Time/Room: 10:10am - 10:40am/Wolfensohn Hall Topic: Interpreting Deep Neural Networks video slides Speaker: Bin Yu, University of California, Berkeley Time/Room: 11:10am - 11:50am/Wolfensohn Hall Topic: Designing explicit regularizers for deep models video slides Speaker: Tengyu Ma Time/Room: 11:50am - 12:20pm/Wolfensohn Hall Topic: Spotlight Talks: Arjun Nitin Bhagoji, Jiaoyang Huang, Rosemary Ke, Or Sharir, Omar Shehab Speaker: Various video Time/Room: 12:20pm - 1:00pm/Wolfensohn Hall Topic: Kernel and Rich Regimes in Deep Learning video Speaker: Nati Srebro, TTIC Time/Room: 2:30pm - 3:10pm/Wolfensohn Hall Topic: Spotlight Talks: Sebastian Goldt video Speaker: Various Time/Room: 3:10pm - 3:40pm/Wolfensohn Hall Topic: Provably Efficient Reinforcement Learning with Linear Function Approximation video Speaker: Chi Jin, Member, School of Mathematics Time/Room: 4:00pm - 4:30pm/Wolfensohn Hall Topic: Poster Session Speaker: Various Time/Room: 4:30pm - 5:30pm/Wolfensohn Hall

October 18, 2019

Topic: Reinforcement Learning, Deep Learning, and the Role of Policy Gradient Methods video Speaker: Sham Kakade, University of Washington Time/Room: 9:30am - 10:10 am/Wolfensohn Hall Topic: Statistical Mechanics of Machine Learning video Speaker: Surya Ganguli, Stanford University Time/Room: 10:10am - 10:40am/Wolfensohn Hall Topic: Concentration inequalities for random matrix products Speaker: Rachel Ward, The University of Texas at Austin; von Neumann Fellow, School of Mathematics Time/Room: 11:10am - 11:50am/Wolfensohn Hall Topic: Representational Power of Graph Neural Networks video slides Speaker: Stefanie Jegelka, Massachusetts Institute of Technology Time/Room: 11:50am - 12:20pm/Wolfensohn Hall Topic: Spotlight Talks: Zhiyuan Li, John Zarka, Stanislav Fort video Speaker: Various Time/Room: 12:20pm - 1:00pm/Wolfensohn Hall Topic: Toward a Causal Analysis of Generalization in Deep Learning video Speaker: Behnam Neyshabur, Google Time/Room: 2:30pm - 3:00pm/Wolfensohn Hall Topic: Spotlight Talks: Zhifeng Kong, Daniel Paul Kunin, Omar Montasser video Speaker: Various Time/Room: 3:10pm - 3:40pm/Wolfensohn Hall Topic: Informal discussion sessions Time/Room: 3:40pm - 5:30pm/Wolfensohn Hall

~end

Contributed talks: Deadline was Sept 2

A few shorter slots were given to showcase late-breaking results and work by young researchers (grads and postdocs).

Selected papers were invited as either talks or posters. Notification deadline was Sept 10.

Travel Information:

Directions to IAS (driving, train and plane): https://www.ias.edu/about/maps-directions

Places to Stay:

Please inquire with the hotel if they offer shuttle service to the local Princeton area.