Towards the end of his life, in the depths of introspective melancholy, Ludwig van Beethoven created some of the world’s most intensely emotional music. The ninth symphony, the Missa Solemnis and some of his greatest piano sonatas are works that still communicate a uniquely concentrated darkness of thought. Now a British Beethoven scholar believes she can explain the German composer’s motivations – and help solve a puzzle that has troubled musicologists and biographers for almost two centuries.

The pieces, Susan Lund believes, were written in response to the grief of separation from a secret, stricken son, and as consolation for the boy’s beloved mother.

In her new book, Beethoven: Life of an Artist, Lund, a veteran Beethoven expert and novelist, has put together what she regards as conclusive musical arguments to show – as she has claimed since the late 1980s – that it was sorrow over separation from his only child that inspired late masterpieces such as his renowned choral setting of the mass and the Hammerklavier sonata, Opus 106.

“Everything mysterious about the path of Beethoven musically, if you put this boy into the equation, then it is all explained,” Lund, 72, said. “It seems clear he loved a woman called Antonie Brentano, an aristocrat who had been married to a man she did not love at the age of 17. I am convinced her son Karl Josef was Beethoven’s child.”

The boy, born in 1813 and never seen by the composer, became ill aged four with a condition that limited his movements and mental capacity. It was news of this devastating illness, Lund argues, that caused a notoriously unexplained barren period in Beethoven’s creative life in 1817. Karl Josef died childless in his late 30s.

Lund’s case fits in with a theory established almost four decades ago in Maynard Solomon’s groundbreaking study, Beethoven, and it also answers the question that has prompted perhaps more speculation than any other in the lives of the great composers: the identify of the “immortal beloved”, the woman addressed in the passionate, three-part love letter found among Beethoven’s personal papers after his death in 1827.

The first page of Beethoven’s letter to his “immortal beloved”. Photograph: Universal History Archive/UIG via Getty Images

In the summer of 1812, the 41-year-old composer had travelled to the Bohemian spa town of Teplitz overnight, following a probable love tryst in Prague. The next morning he wrote a letter to “My angel, my all, my own self”, asking: “Can you alter the fact that you are not entirely mine and I am not entirely yours?”

The following evening he continued the letter and added a final page, using a phrase that has become his most famous: “Already in bed my thoughts go out to you, my immortal beloved.” Beethoven concluded: “I can either live wholly with you, or not at all.” His letter, which was unsent or had been returned, survived many changes of lodgings in the 15 years until his death and was clearly dear to him.

Since the publication of Solomon’s book in 1977, a school of thought has supported his claim that Brentano, a cultured patron who had met Beethoven in Vienna in 1810, is the likeliest object of his love. Like Lund, Solomon holds that Brentano is the candidate who best fits with all the historical dates and locations. Solomon established, from hotel registration documents, that she and her husband spent the summer of 1812 at Bohemian spas and that Brentano had once said Beethoven was “even greater as a man than as an artist”.

According to Lund, Solomon was aware of her claim that Brentano’s son was Beethoven’s lost child. But he did not include the suggestion in his biography, she said, because he feared it would distract from his own interpretation of the key events of the composer’s late life.

Now Lund, who has already written both a novel and a screenplay exploring the putative relationship between Beethoven and Brentano, believes she has pieced together the background to some of his most moving works.

“I claim that Beethoven wrote the Missa Solemnis for Karl Josef, and that is colossally significant,’’ she said. While the composer was atheist, at least according to Haydn, in a huge effort of empathy with the Catholic Brentano he wrote the work he called “his greatest” as solace for a mother with a son who, as she thought, was unable to secure his path to heaven.

“This is what this woman he loved believed in,” said Lund. “All the other candidates for the role of his mistress were not as religious as Antonie, who had been raised by nuns. What is more, when he became ill, Karl Josef could only be soothed by being played to on the piano.” She cites as evidence the composer’s short piano trio movement, written for Antonie’s eldest daughter, Maximiliane, just before he arrived in Teplitz. Years later, he dedicated the first in his final group of three piano sonatas to her and had intended to dedicate the next two, Opus 110 and 111, to her mother. His last major piano work, the Diabelli Variations, was also dedicated to Brentano.

Gary Oldman as Beethoven in the film Immortal Beloved. Photograph: Sportsphoto Ltd./Allstar

Many rival camps on the identity of Beethoven’s “immortal beloved” have developed over the years. Some suggest the composer’s nephew, over whom he fought a custody battle, was the product of a romantic relationship with his sister-in-law, Johanna. This view, put forward by director Bernard Rose in the 1994 biopic Immortal Beloved, starring Gary Oldman, has little standing in the music world. Others point to the unhappily married Hungarian aristocrat Josephine von Brunsvik, to whom Beethoven wrote lovingly in 1805. But she remarried in 1810. A third candidate is Countess Giulietta Guicciardi, a former pupil to whom the composer dedicated the piece known as thee Moonlight Sonata. Two other women, Therese Malfatti and Amalie Sebald, may also fit the bill. Malfatti could have been the “Elise” of Beethoven’s famous tune “Für Elise”, while Sebald, a singer, has been placed in Teplitz in 1812.

The detective work is a mere distraction for many music lovers. Observer critic Fiona Maddocks says solving such a personal riddle is only relevant if it affects our understanding of the work: “It might be of interest, in human terms, to find that Beethoven had a son, but musically any such discovery is only of substance if we can prove how it made a difference to the notes he wrote.”

According to Lund, that is exactly what she has done.