The authors thank D. Tank, S. Lewallen and R. Low for insightful discussions, and F. Caccuci, T. Wills, S. Deneve, Y. Burak, J. Murray, J. Pillow, L. Paninski and M. Sahani for comments on the work or manuscript. I.F. is grateful to G. Prasad for pointing her to the field of topological data analysis several years ago, and to W. Bialek for raising the possibility of unsupervised discovery of encoded variables from neural data, also several years ago. This work was supported in part by grants from the NIH (U01-NS094330-03), the Simons Foundation (SCGB and the International Brain Laboratory) and the Howard Hughes Medical Institute through the Faculty Scholars Program to I.F., and by the Canadian Research Chair in Systems Neuroscience (245716), a CIHR Project Grant (155957), a NSERC Discovery Grant (RGPIN-2018-04600) and the IRDC (108877-001) to A.P. Part of this work was performed by R.C. and I.F. in residence at the Simons Institute for the Theory of Computing at UC Berkeley, where R.C. was a Google Research Fellow.