In the 19th century, the English branch of the powerful and immensely rich Rothschild family built the most famous of their country houses in the Vale of Aylesbury, which is why, one misty morning in late March, I find myself at Waddesdon Manor, a picture-perfect Victorian replica of a French chateau. "I think this house will give you a sense of how the family used to live," says Hannah Rothschild, my host. "The blinds and curtains drawn to protect the art, the panelling and drapes creating a deadening effect. These were houses that killed noise, even the noise of children." Overflowing with servants – at Tring Park, down the road, footmen were required to carry cherry trees to the table, that diners might pick their fruit straight from the branch – and run to a routine as immutable as marble, growing up in such a house was like living in a gilded cage.

Hannah wants me to soak up this atmosphere, airless and introspective, because she believes that it explains, at least in part, the extraordinary life of her great aunt, Pannonica, aka Nica, the subject of her startling new book, The Baroness. Nica, who was born in 1913, grew up at Tring Park (Tring is now a school; Waddesdon Manor, though administered by a trust under the chairmanship of Hannah's father, Jacob, was bequeathed to the National Trust in 1957). There, she wiled away her young days in a starched white dress, sewing and playing the piano; her parents did not approve of education for girls and running and hiding were forbidden lest her frock be ruined. Life was monotonous and dull but, knowing nothing else, she did not think to kick against it. In 1934, she was duly presented at court and her marriage in 1935, to Baron Jules de Koenigswarter, a handsome French diplomat, was predictable, if not the soaring match her ambitious mother had dreamed of. If he was controlling, well, she was used to that.

In 1948, however, something happened. On her way to the airport after a visit to New York, Nica stopped to visit a friend, the jazz pianist Teddy Wilson, who played her a recording of "Round Midnight" by a then unknown jazz pianist, Thelonious Monk. Unable to believe her ears, she listened to it 20 times in a row and was bewitched. Having missed her plane, she never went home again. Abandoning her husband and five children, she moved into a suite at the Stanhope hotel and set about trying to meet the man who had made this extraordinary record. Naturally, it took a while to track the erratic Monk down. It wasn't until 1954 that she finally laid eyes on him, having flown to Paris for the privilege. Did he live up to her dreams? Oh, yes. He was, she said, "the most beautiful man she had ever seen". From that moment, there was no going back. For the next 28 years, Nica devoted her life to Thelonious Monk. In her eyes, he could do no wrong. He was a genius, pure and simple, and there was nothing she would not do – no money she would not spend, no place she would not go – to make his life easier.

It has taken Hannah almost as long to write about Nica. She first heard about this unlikely relative when she was 11, from her grandfather Victor, another extraordinary Rothschild (Victor famously liked to water-ski in a Schiaparelli dressing gown). "You're like my sister," he said, as Hannah struggled to master the 12-bar blues. "You love jazz, but can't be arsed to learn to play it." Hannah knew her other great aunts, Liberty and Miriam, but this Pannonica was a mystery. When she asked her father about her, all he would say was: "No one ever talks about her." When she asked Miriam, she said: "She's the Peggy Guggenheim of jazz" and: "She is vulgar."

Growing up, though, the whispers Hannah heard were tantalising. She's known as the Jazz Baroness. Charlie Parker died in her apartment. She lived with 306 cats. Twenty-four songs were written for her. She raced Miles Davis down Fifth Avenue. She went to prison so he wouldn't have to… So when, in 1984, Hannah went to New York for the first time, she decided to telephone her. "Would you like to meet up?" she said, nervously. "Wild," said her great aunt, who was then 71. "Come to the club downtown after midnight." She informed Hannah that she would know the spot by the sight of her car – a large pale-blue Bentley – parked outside.

Thelonious Monk and Baroness Nica de Koenigswarter get into her Bentley outside the Five Spot cafe, New York, 1964. Photograph: Ben Martin/Time & Life Pictures/Getty Images

Hannah Rothschild was 22 and "failing to live up to the expectations, real or imagined, of my distinguished family"; Pannonica, a success only in her own terms, seemed to throw her a kind of lifeline. Nica's very existence suggested that perhaps one could escape one's past. "I looked across the table at this newly discovered great aunt," she writes in her book, "and felt a sudden, inexplicable surge of hope. A stranger walking into the club would merely have seen an old lady sucking on a cigarette, listening to a pianist. They might have wondered what this fur-coated, pearl-wearing dame was doing, swaying to the music, nodding appreciatively at a particular solo. I saw a woman who seemed at home and who knew where she belonged." Did her aunt give her any advice? Only this. "Remember," she would say. "There is only one life."

Hannah returned to London where she finally landed a job at the BBC ("I'd always wanted to work there, but I could have papered a room with rejection letters," she says). She and her aunt met again just twice; in 1988, Nica died suddenly, following a heart-bypass operation. But by then, Hannah was hooked. She wanted to know all about her aunt's mysterious life, to sort fact from fiction. In the face of some opposition from her family, one of whose primary traits is, she says, was "obsessive secrecy", she decided to research her story. It wasn't easy. Rothschild women refused to take her phone calls; she also received two threatening letters. Nica's children, having initially been enthusiastic about the idea of a book, would not co-operate. "I can't speak for them, but perhaps they felt protective, that this is their story, not mine. Do they think I'm trespassing? I don't know."

But she got there in the end. In the years since, Hannah has made a radio documentary about her aunt and a film – and now, finally, she has written her biography. Is she done? "Yes," she laughs. "I've called time, I promise." And does she think she has solved the mystery of Nica and Monk? Were they lovers? Or merely loving friends? (Monk had a devoted wife, Nellie, from whom he never officially separated.) "I don't think it was a steamy, hot love affair," she says. "There's a tendency to sexualise every relationship, especially one that crosses class and race. When you look at Nica with him [in photographs and on film], she's in love with him: the way she gazes at him, the way she's laid her life out before him like a golden cloak of devotion. But I don't believe that sex was at the heart of it, because I don't believe it would have lasted so long if it had been. This was 30 years of being tested [by Monk] to the extreme. It's difficult, though. I went through a lot of soul-searching. I didn't want to portray her as a sad groupie with lots of money. Is she using him? Is he using her? And why did Nellie put up with it? Because it didn't have to be sexual for Nellie to be fed up about it." Monk's last years, when his mental health became so bad that he disappeared from the jazz scene altogether, were spent at Nica's cat-infested New Jersey home. But at his funeral – Monk died in 1982, from a stroke – Nellie and Nica sat side by side in the front row of the church and mourners paid their respects to each of them, as if both were widows.

Hannah Rothschild grew up, not in some vast palace, but in Maida Vale, west London, in a house she now owns (a single divorcee, she lives there with her three teenage daughters). When did she realise, then, that her family was not quite like other people's? "I don't think I ever did. Your family is just your family." Her great aunt Miriam, a famous entomologist, lived in another huge country house, Ashton Wold, which was so overrun with ivy and buddleia and honeysuckle – to encourage insects – that in summer time, it was almost impossible to see out. "She had a pet owl and a pet fox. My grandfather Victor [also a distinguished scientist], had a pet owl, too, that used to fly around and hoot. So I realised they were quite eccentric, but that was all I knew. One thing I was certain of, though, was that they were frightening. They were fiercely intellectual and they didn't suffer fools. If you bored them, they would tell you. They'd leave the room mid-conversation.

"It [Rothschild] is a big name to have. It means banking, and Jewish, and money. But then, on my side, there was this extra thing of being intellectual and academically high-achieving. I didn't feel like a banker's daughter [her father, Jacob, is an extremely successful businessman, as is her brother, Nat]. But nor did I live up to this other side. Of course, there are huge advantages – huge – to growing up in that world. Financial advantages, yes, but also people remember your name, you get access to things. I cannot sit here and say: poor me. I was incredibly lucky."

Until the second world war, after which its fortunes were depleted – some 3,500 works of art were stolen by the Nazis and many of its companies nationalised – the family's rise had looked unstoppable. "Seven generations ago, we were living in a ghetto in Frankfurt in a house that was 14 feet wide," she says. "There seemed to be no chance that the family would ever escape that ghetto until, rather ironically, the French shelled it [in the 1790s], breaking down the walls, so the Jews were finally able to get out."

It was at this point that Mayer Amschel, the Rothschild patriarch, famously sent his five sons to five European capitals where, between them, they eventually built the biggest bank in the world. And of these five, it was Nathan Mayer, Nica's great-great-grandfather, who arrived in England in 1798 with no formal education, and speaking no English, who was most determined to succeed. By the late 19th century, the British branch of the family had a title, a collection of priceless art, many country estates (a painting Hannah shows me depicts some 40 Rothschild houses) and the ear of the prime minister.

Nica's father was Charles Rothschild. Like his zoologist brother, Walter, who turned the grounds of Tring into an extraordinary wildlife park with kangaroos, giant tortoises, emus, rheas and cassowaries, and who drove a carriage pulled by zebras, Charles had a passion for natural history. He was a keen amateur entomologist and named his youngest daughter Pannonica for a species of moth. However, this was not an interest he was free to pursue. Instead, every day, he was required to put on a suit and go to work in the family bank. This did not suit him one bit.

Charles also suffered from a mood disorder that may or may not have been schizophrenia (so, later, did his another of his daughters, Liberty). At times, he would not speak for days. On other occasions, he was manic, unable to sleep or stop talking. As he grew older, the gaps between these episodes grew ever shorter. Finally, in 1923, he went into a bathroom, locked the door and slit his throat with a knife.

Hannah believes that Charles's suicide lies at the heart of Nica's unlikely bond with Monk, who suffered from a similar illness, with similarly debilitating symptoms. "Her father's death was incredibly violent, but afterwards it was never mentioned: suicide was still illegal. When she met Thelonious, there must have been huge resonances. He behaved in a very similar way to Charles, and Charles had been made to live a certain kind of life, going to the office every day, when what he wanted to be doing was collecting butterflies. Her passionate attempts to dignify Thelonious's life, to protect him, to say it's all right to spend the day sleeping if that's what you want or need to do… I'm sure this was her way of addressing an earlier injustice. It was a kind of reparation. In return, he gave her an incredible sense of purpose and belonging. If you think of her as a woman who'd been evicted from a close family, that's quite a frightening thing. But Thelonious and a whole group of musicians said to her: come and be part of this. We'll hang with you." Monk wasn't the only one who wrote a song for Pannonica. So did Art Blakey, Sonny Clark, Kenny Drew and at least a dozen others.

But life in New York had a dark side, too. In 1955, Nica was evicted from her suite at the Stanhope when Charlie Parker, having turned up at her door one night with nowhere else to go, choked and died there (she claimed to have heard a clap of thunder as the life left him – a sound that has since passed into jazz folklore). In 1958, Nica decided to drive the impoverished Monk to a gig in Maryland. In a town called New Castle, Delaware, she stopped the car outside a motel so he could use the bathroom. As she waited, the police approached; in this part of America, the sight of a white woman and black man together was unusual enough to attract attention. There followed an altercation. Monk was beaten up. The police searched the car. When they found marijuana, Nica knew exactly what to do. Monk was too fragile to go to prison. She told them that the dope was hers.

The consequences of this moment of bravery were potentially dire. Nica faced a prison sentence of up to 10 years, followed by immediate deportation. Her life with Monk would be over, but the prospect of a return to England was just as painful. How would her family react? Would her husband ever let her see her children again? "It must have been terrifying," says Hannah. "I finally found out how terrifying in a letter she wrote to her friend Mary Lou Williams the night before her trial in January 1962. She is in Delaware. She writes that she is going to go into a church and light a candle. She writes, 'This is the day upon which my entire future may well depend.' She says she can't talk to Thelonious or Nellie about it because they have their own worries. She was so completely alone. I felt quite distressed on her behalf. Where is everyone? I thought." In the end, though, there came a miracle: the case was dismissed on a technicality, her lawyer arguing that the troopers had searched her car without her permission.

In The Baroness, Hannah tells this story with care, balancing narrative tension with a desire to lay out all the facts so readers can make up their own minds. Like the rest of the book, it is wholly gripping. So will she write another or will she return to her first love, films? "I am working on a novel. It's a strange thing. Having been a documentary film-maker for my entire adult life, suddenly I'm not sure if I am one anymore." It is getting harder to find homes for the kind of films – detailed, beady, slow-burning – that she makes. Her marvellous fly-on-the-wall Peter Mandelson documentary, filmed in the build-up to the 2010 election (the one in which he ate a yoghurt in a way that made him look like a stoat devouring a field mouse), was not commissioned and she only sold it to BBC4 after it was finished.

"It was quite embarrassing. He would say, 'Where is our film going?' And I would have to tell him that I didn't know." Has she heard from him since? "He saw the film, and he liked it, and that was it." Mandelson is a friend of her brother, Nat (Mandelson famously holidayed at his home in Corfu). But she can't talk about Nat. "He really, really hates it. I love my brother. He's a fantastic guy. But I have to respect that."

To outsiders, the wonder is that she works at all. A lot of people from her background would have gone to Oxford, as she did, and got on with the business of allowing their trust fund to screw them up. "Yes," she says. "But the truth is that work is fantastically interesting. It's the one thing you can rely on, actually, and it's so exciting – my father taught me that. But while I was writing the book, I did think about the work ethic, the way it persists in our family. Unlike aristocratic British families, perhaps we haven't forgotten where we came from." As for money, she has "made my peace with it… if it were swilling round without any purpose I would feel very differently, but the inheritance enables us to do good things at our foundation. I think it's completely right that people who have more should give it away and make life better for those who don't. My father is planning to give his away. Actually, he's already started, and good for him."

We move downstairs, to tour the house, where I enjoy the silly thrill of stepping behind velvet ropes and walking through doors marked Private. The house is very beautiful but, as she points out, it is all of a piece. The Rothschilds had no furniture to inherit and this one's interiors came courtesy of a couple of French hotels that were being demolished following Haussmann's remodelling of Paris. Does she wish it was in the family still? "Good God, no," she says, with a theatrical shudder. "But writing my book has made me understand these houses. They were a way for the family to anchor itself, to show the world that they mattered. When you really think about it, this house is just a three-dimensional calling card."