TOKYO—After weeks of criticism over the cost of Tokyo’s new Olympic Stadium, Prime Minister Shinzo Abe said Friday that his government would scrap the current design for the central venue of the 2020 Summer Games and start from scratch.

“We decided that we will start over from the beginning,” Mr. Abe said, in a reversal that came less than three weeks after the government said it was committed to building a $2.1 billion stadium designed by Iraqi-born British architect Zaha Hadid.

What changed the prime minister’s mind, officials here acknowledged, was the criticism that the stadium was far too expensive for a country that had promised a frugal approach to the 2020 Games. Polls showed the public overwhelmingly opposed, and even members of Mr. Abe’s ruling party publicly objected.

The about-face means the stadium won’t be available for the 2019 Rugby World Cup in Japan as originally intended. It is also an embarrassment for Mr. Abe, who had won the Olympics for Tokyo with a presentation that included the Hadid design, characterized by two soaring 400-meter (1,300-foot) arches to support the roof.

Ms. Hadid’s firm, Zaha Hadid Architects, issued a statement after Mr. Abe’s announcement defending the stadium’s design, which the company said wasn’t responsible for the increased cost.