Goldfinger was a man who thought big, a champion of communism, an eccentric, a bully who put people in fear. And that was just the architect.

The story of the Erno Goldfinger's vehement reaction when the author Ian Fleming appropriated his name - and aspects of his character - with deliberate savagery for the villain and title of the James Bond novel was disclosed to the Guardian Hay festival yesterday.

The dispute led to legal action. When the film Goldfinger came out, the architect was afflicted by spoof calls in the middle of the night. Callers would intone in bad Sean Connery accents, "Goldfinger? This is agent 007," or sing the film's theme tune, "an irritation still endured by members of the family who list their names in the telephone directory," Nigel Warburton, of the Open University, told a breakfast-time audience.

Fleming turned the dominating, 6ft 2in Erno into the 5ft imperious megalomaniac Auric Goldfinger, who nearly succeeds in stealing the US gold reserves at Fort Knox for the Soviet Union.

Erno - like Auric - was a British-naturalised foreigner and a Marxist who spent much of the second world war raising money for the Soviet cause. Otherwise there were differences between the two, as Dr Warburton noted, discussing his new book Erno Goldfinger: The Life of an Architect, the first biography to be published.

But when Erno's business associate Jacob Blacker was asked for his opinion of a proof copy of the Bond story, he told Erno ironically that he could find only one substantial difference: "You're called Erno and he's called Auric."

Erno Goldfinger was one of the 20th century's prime advocates of London tower blocks. He designed the often reviled Alexander Fleming House at the Elephant and Castle, Trellick Tower in Ladbroke Grove and Balfron Tower in Tower Hamlets.

One story explaining Fleming's animosity is that he lived for a time in Hampstead and disliked Erno's design for terraced houses in Willow Road, according to Dr Warburton. Fleming knew of Erno through a golfing friend who was related to Erno's wife.

The friend appears in the novel - but his woman relative has been transformed into a heroin addict. Erno somehow heard about the novel when it was in the publisher Jonathan Cape's presses in 1959. His response was, "Shall we sue?"

After hearing Blacker's view, Erno ordered solicitors to act. Cape agreed to pay his costs and agreed out of court to make clear in advertising and in future editions that all characters were fictitious.

Fleming, in turn, was livid. He asked Cape to insert an erratum slip in the first edition changing the character's name to Goldprick, a name suggested by the critic Cyril Connolly. Luckily for the film posters and theme tune of the future, sung by Shirley Bassey, Cape demurred.

Dr Warburton said the clarification did not appear in the novel's current edition.

The real-life Goldfinger, however, deserved to be remembered as a visionary architect who wrote in 1941: "Cities can become centres of civilisation where men and women can live happy lives. The technical means exist to satisfy human needs. The will to plan must be aroused. There is no obstacle but ignorance and wickedness."