A trio of British-born researchers working in the U.S. won the Nobel Prize in physics for what one of them called a curious mathematical “toy” that to his surprise revolutionized the study of exotic matter suitable for quantum computers, new superconductors, and advanced designer materials.

Working separately, the three laureates conceived a new way to understand the topology of materials, or the study of shapes that change in increments. At its simplest, a sheet of paper can have many sides, depending upon the topology of...