(CNN) New South Korean President Moon Jae-in has cast doubt on a landmark deal with Japan over wartime sex slaves.

"The reality is the majority of our people cannot emotionally accept the comfort women agreement," Moon told Japanese Prime Minister Shinzo Abe Thursday.

Moon emphasized "the two sides should work together based on understanding of the emotions and reality of the people," according to a statement from the presidential office.

However, he was clear that he thought the issue shouldn't affect the wider South Korea-Japan relationship, saying the two countries should deal with the "comfort women" dispute and "work independently in order to respond to the North Korean nuclear and missiles issues."

The issue of so-called "comfort women" has caused rifts between the two countries for decades. An agreement signed by the governments of South Korea and Tokyo in 2015 was intended to settle the matter, but victims' groups objected to it and the issue remains highly controversial.

Read More