Nine months after Shinzo Abe's Liberal Democratic party was elected to power there are widespread rumours that supernatural phenomena are preventing him from moving into his official residence in central Tokyo.

Rather than move into the presidential home, Abe has preferred to make the 15-minute commute from his family's house to the office and has shown no indication that he is preparing to move into the art deco mansion in the heart of Nagatacho, the capital's political district, where he lived during his previous stint as prime minister in 2006-07.

While officials deny Abe has a fear of the supernatural – and there was no discussion of the matter in 2007 when he resigned from office due to ill health – rumours abound that the official residence is haunted by figures from Japan's bloody prewar history.

In 1932, a group of naval officers assassinated the then prime minister, Tsuyoshi Inukai, at the residence during a failed coup. Four years later, it was the scene of another attempted coup by a radical faction of the imperial Japanese army.

Bullet holes and traces of fire damage near the entrance to the mansion property have left physical reminders of the coup attempts. But there have also been frequent reported sightings of men in military uniform in a nearby garden. Several former first ladies have refused to move into the building over fears it was haunted.

The building, completed in 1929 in the style of the US architect Frank Lloyd Wright, did not become the prime minister's official residence until 2005.

In May, media speculation about the spirits of Japan's prewar political instability prompted an opposition MP to ask in parliament if Abe was too frightened to move in.

The government said it was "not aware" of the ghost tales although Abe, who said the 11-room property was simply too big for his needs, has since acknowledged hearing speculation about the building's supernatural guests.

Japanese media claim that Abe recently broached the subject with colleagues during a dinner at the official residence.

"I don't feel like living here because there are ghosts," the Asahi Shimbun quoted him as saying. He reportedly suggested to the assembled LDP executives: "Why don't we all live here together?"