CANBERRA—Australia approved plans to dump vast amounts of mud and rock into oceans surrounding the Great Barrier Reef, paving the way for developers to expand a coal port on the country's eastern coast.

The government agency that manages the reef, a designated World Heritage site, said it would allow as much as 3 million cubic meters of dredged-up material from the sea floor to be dumped.

The...