Japan’s deputy prime minister has said the Tokyo Olympics are “cursed”, as speculation mounts that the Games will have to be postponed due to the coronavirus pandemic.

Taro Aso, who has a history of making gaffes, told a parliamentary committee on Wednesday that the Olympics appeared to be blighted by world events every 40 years.

Japan had planned to host the summer and winter Olympics in 1940, but the second world war forced the cancellation of both Games.

Forty years later, many countries, including the US, China and Japan, boycotted the Moscow Olympics in protest at the Soviet Union’s invasion of Afghanistan.

“It’s a problem that’s happened every 40 years – it’s the cursed Olympics, and that’s a fact,” Aso said.

As Japanese officials and International Olympic Committee again insisted the Games would go ahead as planned, it emerged that the Tokyo 2020 organising committee’s chief, Yoshiro Mori, had recently attended a meeting with a senior sports official who had since tested positive for the virus.

Mori, a former prime minister, was at the same 10 March meeting, held to discuss last year’s Rugby World Cup, as the deputy head of the Japanese Olympic Committee, Kozo Tashima, who tested positive on Tuesday.

Mori, who is 82 and has lung cancer, has not shown symptoms and does not meet the requirements for a test, an official from his office told Reuters.

About 60 people attended the meeting, with Mori seated about 10 metres away from Tashima on the opposite side of the table, according to Jun Kusumoto, a spokesman for the Rugby World Cup organising committee. Health authorities have contacted other attendees who are thought to be at risk.

“[Mori] goes to hospital three times a week for dialysis, so if he develops a fever or has other symptoms, a doctor will be able to test for it,” the official from Mori’s office said.

The chief cabinet secretary, Yoshihide Suga, told reporters that the prime minister, Shinzo Abe, had met Mori on Monday, but did not directly address a question about whether Abe would be tested.

Aso, who doubles as finance minister, said holding the Games this summer would “not make sense” if other countries were unable to send their athletes.

“As the prime minister said, it’s desirable to hold the Olympics in an environment where everyone feels safe and happy. But that’s not something Japan alone can decide.”

Tokyo 2020 organisers said a little-known Japanese swimmer who competed in the 1996 Atlanta Games would receive the Olympic torch during a scaled-back handover ceremony in Athens later on Thursday.

Naoko Imoto, who works in Greece for Unicef, had been approached by organisers after virus-related travel restrictions prevented a Japanese delegation from flying to Athens to receive the symbolic flame, which is due to arrive in Japan on Friday.

“We decided yesterday that we felt it was necessary for a Japanese person to undertake this role,” the organising committee’s chief executive, Toshiro Muto, told reporters.