A fossil jaw found in the Himalayan highlands of Tibet belongs to a vanished human species called Denisovans, deepening the mystery of human evolution in Asia, scientists said Wednesday in a new study probing the roots of humankind.

Discovered by a local Buddhist monk, the fossil shows these archaic human relatives lived on the roof of the world in the rarefied air at almost 11,000 feet—an altitude that would leave many people today starved for oxygen. They settled at these frigid heights at least 160,000 years ago, more than...