(CNN) The organizers of Japan's 2020 Summer Olympics have refused to ban a controversial flag that South Koreans say stirs memories of the violent, brutal life endured under Japanese rule during the early to mid-1900s.

South Korea's parliamentary committee for sports wants the so-called Rising Sun flag to be banned from venues at next year's Tokyo Olympics as it is viewed in South Korea as a symbol of Japanese "imperialism and militarism."

But this week, Tokyo 2020 -- the organizing committee of the Tokyo Olympics -- said that the flag wouldn't be banned as it was widely used in Japan.

"The flag itself is not considered to be a political statement, so it is not viewed as a prohibited item," Tokyo 2020 said in a statement.

That decision has caused controversy back in South Korea, where An Min-suk, the chair of the parliamentary committee for sports, condemned the decision.

"A flag symbolizing war is not suitable for peaceful Olympic Games," An said in a press conference Tuesday.

"The Rising Sun flag is akin to a symbol of the devil to Asians and Koreans, just like how the swastika is a symbol of Nazis which reminds Europeans of invasion and horror."

The Korean Sports & Olympic Committee told CNN Friday that it had not yet made an official request to Tokyo 2020 or the International Olympic Committee (IOC) to ban the flag. In a response to CNN, an IOC spokesperson said sports stadiums should be free of any political demonstration. "When concerns arise at games time we look at them on a case by case basis," the spokesperson said.

South Korea's Foreign Ministry said Thursday it had expressed regret over the decision via diplomatic channels. Earlier in the week, South Korea's Foreign Ministry spokesperson Kim In-chul said the Japanese government needed to acknowledge its history.

"I believe the Japanese government is well aware that the Rising Sun flag is viewed by neighboring nations as a symbol of (Japanese) imperialism and militarism," Kim said on Tuesday, adding that the ministry planned to continue to work to change Tokyo 2020's decision.

The flag disagreement comes amid an ongoing trade spat between the two countries, which has been raging since July.

JUST WATCHED This is what the Tokyo 2020 Olympics medals will look like Replay More Videos ... MUST WATCH This is what the Tokyo 2020 Olympics medals will look like 01:02

Why the flag is offensive to South Koreans

To Japan, the Rising Sun flag is part of its culture . The flag was used by the Imperial Japanese Army, and the country's navy and defense forces still use the flag to this day.

However, South Korea sees the Rising Sun flag as a symbol of Japanese imperialism and war atrocities. Those who use the flag often draw the ire of South Koreans. Last month, for instance, Sean Ono Lennon -- the son of Yoko Ono and John Lennon -- found himself at the center of controversy after defending his girlfriend for wearing a T-shirt that featured the flag.

"We may have differing opinions on what the rising sun pattern represents, but that does not mean we should be enemies," he said in an Instagram post responding to the online outrage.

In the first half of the 20th century, the Korean Peninsula was colonized by Japan, and many Koreans were brutalized, murdered and enslaved. That period is living memory for elderly Koreans and continues to be a highly emotive subject in both North and South Korea.

Japan and South Korea remain locked in heated debate over statues depicting " comfort women " -- Korean women forced into providing sexual services for Japanese soldiers before and during World War II -- and decisions by South Korea's Supreme Court allowing citizens to sue Japanese corporations for reparations for forced labor.

Although South Korea and Japan signed a treaty in 1965 to settle lingering wartime issues, many Koreans feel the deal was unfair and believe there is still room for individuals to make legal claims for reparations.

Another landmark deal reached in 2015 saw Japan offer another apology and pledge $8 million for a foundation to support the surviving "comfort women." However, the former "comfort women" rejected it , saying they had not been consulted.

In recent months, tensions between the two countries have turned into an ongoing trade dispute. The standoff began in July when Tokyo placed controls on exports to South Korea of three chemical materials which are used to make computer chips, among other things.

Last month, Japan officially removed South Korea from its list of preferred trading partners, prompting South Korea's Ministry of Foreign Affairs to release a statement calling the measure "unjust" and saying it was being treated as a "hostile nation" by Japan. South Korea removed Japan from a list of trusted trading partners and announced it was ending its military sharing agreement with Japan which was signed in 2016.