Former Prime Minister Junichiro Koizumi said Saturday that scandal-hit Shinzo Abe is unlikely to win a third term in September’s Liberal Democratic Party presidential election in light of the public’s growing distrust in him.

“Winning a third term will be tough as he is losing public trust,” Koizumi told reporters in Mito, Ibaraki Prefecture.

Abe needs to hold onto the LDP presidency to stay on as prime minister.

Touching on the Moritomo Gakuen scandal, in which state land was sold at a steep discount to a nationalist school operator linked to Abe and his wife, Koizumi said the troubles stem from Abe’s pledge to quit as prime minister and lawmaker if he or his wife, Akie, are found to have been involved in the deal.

Noting that Abe’s wife briefly served as honorary principal of the elementary school Moritomo planned to open on the land, the outspoken former leader said, “How can he say that he is unrelated (to the matter)?”

Abe returned to office in December 2012 after the LDP wrested power from the Democratic Party of Japan. He is the third-longest-serving postwar prime minister behind Eisaku Sato and Shigeru Yoshida.

While prime ministers tend to hold power for relatively short periods compared with leaders in other major developed countries, the popular Koizumi managed to stay in the top job from 2001 to 2006.

Besides Koizumi, sharp criticism from the LDP’s factions and coalition partner Komeito have made things more difficult for Abe’s government, adding to his problems.

“The situation is getting worse and worse,” said a source close to the prime minister.

An LDP source said that one scenario likely to unfold will see Abe’s disapproval rating climb, leading him to give up plans to hold on for a third term. The source even pointed to the possibility of Abe quitting before his term as party head expires.

“Even a firmly guarded castle may collapse if there is laxness or arrogance,” Akiko Santo warned at Thursday’s meeting of the faction led by Finance Minister and Deputy Prime Minister Taro Aso, in her capacity as acting head of the group.