(CNN) Japan's deputy prime minister has come under fire after apparently blaming childless women for Japan's demographic woes.

Taro Aso, 78, denied that the elderly were the cause of the country's declining population and increasing social security costs during a speech in Fukuoka prefecture in southern Japan on Sunday.

"There are lots of strange people who say the elderly are to blame, but that is wrong," said Aso, who also holds the role of finance minister. "The problem is with those who didn't give birth."

After being grilled by an opposition lawmaker during a budget committee session, Aso said Monday that he would retract the remark "if it caused misunderstanding."

"It gave a false impression without conveying the meaning of my original remark," Aso explained, without clarifying what he had actually wanted to say.

Read More