The estate of JRR Tolkien is embroiled in a fierce legal battle over an American novel that uses the author of The Lord of the Rings as a central character.

The dispute comes only months after Tolkien's heirs settled a multimillion-pound lawsuit over royalties from the Lord of the Rings films. Tolkien's family claimed that the New Line studio behind the $3bn-grossing trilogy failed to pass on any money to the estate, showing "insatiable greed".

Now the estate, registered in Oxford, where Tolkien was a university professor, is demanding the destruction of all copies of Steve Hillard's Mirkwood: A Novel About JRR Tolkien. The 450-page work recounts a young woman's quest to find her grandfather after discovering documents given to him by Tolkien. The estate is demanding an immediate halt to further sales, and threatens legal action to obtain damages.

In a letter to Hillard, the estate's lawyers, Manches, said: "At no time have our clients granted permission to use the name and personality of JRR Tolkien in the novel, nor would they in any foreseeable circumstances." It claims "unlawful commercial advantage" has been taken of the estate's "valuable rights", and argues that Hillard's book "trivialises the name, personality and reputation of the late professor".

Hillard admits to using a quote from a published Tolkien letter, but says other conversations are imaginary and that he has produced a respectful portrait. He also claims that the author of The Hobbit, who died in 1973, would have been on his side in the argument. "His stories were unearthed from his research," said Hillard. "He would be somewhat concerned about attempts to stifle works that borrow from history."

Intellectual property rights and the powers of an estate to control names and reputations are now set to be challenged in the US courts. Hillard is in turn suing the Tolkien estate in what looks set to be a test case. In his lawsuit, filed in a Texas district court, Hillard argues that the novel is one of "innumerous fictional works that contain fictional accounts involving real people".

He cites Joyce Carol Oates's Blonde, which features Marilyn Monroe; Michael Cunningham's The Hours, with Virginia Woolf; and Don DeLillo's Underworld, in which Frank Sinatra appears.

Hillard, 62, from Austin, Texas, said: "Imagine if you couldn't use Winston Churchill as a character in a book about the second world war. How many movies have used a fictional treatment of Churchill? The implications of this assertion by the estate would be that you couldn't do that."

His lawyer, Daniel Scardino, said: "Just imagine a world where you can't talk about celebrities, where you can't put celebrities in works of authorship, whether fiction, non-fiction, literary criticism or otherwise, where somehow their celebrity status insulates them from criticism … That's the real concern." The estate's demands were "wholly without legal basis", he added

Hillard's legal papers state that Mirkwood "has not in any way … violated the rights of the estate under UK or US law", and that "Hillard is within his rights to author a fictional novel with a celebrity as a character".

Mirkwood is Hillard's first novel. He self-published it and Amazon has been selling it worldwide since January. About 900 copies have been sold. In his day job, he heads a private equity fund assisting minority groups and women in the acquisition and running of radio and TV companies. Having studied philosophy and law he has studied Tolkien over two decades, inspired by his two daughters' love of Tolkien's classics.

According to the book's jacket: "Mirkwood reinvents JRR Tolkien as a man haunted by the very myths he rewove into his famous works … In 1970, [he] sets in motion elvish powers embodied in a cache of archaic documents."

The estate claims that the book jacket's design – a tree illuminated by rays of light above three figures – is "strikingly similar" to Tolkien publications.

The Tolkien estate is headed by the author's son, Christopher, as literary executor. Its lawyer, Steven Maier, said: "I can't comment on the present case in too much detail … However, the Tolkien estate will always take action to protect its intellectual property rights.

"The estate understands that there is a balance … between freedom of expression and the protection of legal interests… This particular use of Tolkien's name has crossed the line of what is fair."