SYDNEY—When shares in BHP Billiton Ltd.’s spinoff company South32 start trading Monday it will complete one of the largest corporate breakups in mining history and, for Australia, the biggest market listing this century.

So much for the superlatives: Investors’ main question now is what the new company is worth.

BHP shareholders this month...