WASHINGTON—Law-enforcement agencies are increasingly finding ways to unmask users of a popular Web service designed to hide identities and allow individuals to exist online anonymously.

To keep their identities secret, users and administrators of a recently shuttered child-pornography website used a network called Tor that obscures the source of Web traffic, authorities said in March. Agents from Homeland Security Investigations tracked many of them down anyway, largely because of mistakes that even some of the most sophisticated...