WASHINGTON—Newly declassified court documents show one of the National Security Agency's key surveillance programs was plagued by years of "systemic overcollection'' of private Internet communications.

A 117-page decision by Judge John Bates of the Foreign Intelligence Surveillance Court offers a scathing assessment of the NSA's ability to manage its own top-secret electronic surveillance of Internet metadata—a program the NSA scrapped after a 2011 review found it wasn't fulfilling its mission.

...