One of the most telling passages in Hard Choices, Hillary Clinton's new memoir-cum-manifesto, is her description of the transcendent moment in 2012 when her husband was bear-hugged on stage by Barack Obama. Bill had just nominated Obama for re-election at the Democratic National Convention in Charlotte, North Carolina, and their warm greeting was seen as a burying of the hatchet between the two presidents, whose relationship had been frosty and, at times, openly hostile. Hillary missed the historic event, stuck out, as she was, in Timor-Leste, on one of her epic global jaunts as secretary of state; but she managed to catch it on computer at the residence of the local US ambassador.

"As the two presidents embraced, the crowd went wild," she recalls in the book. "Watching from some 10,000 miles away, I was full of pride for the former president I married, the current president I served, and the country we all loved."

The passage is telling because it feels incomplete. The narrative arc of Hillary Clinton's life story remains unresolved – at least in the popular imagination – until the trinity is fulfilled: the president I married, the president I served, the president I became.

Will that fairytale ending ever be reached? That's the question on everybody's lips. It's why the TV networks have been clearing their primetime schedules for lengthy Hillary specials; why Simon & Schuster paid at least $8m for Hard Choices, recouping that (with interest) when the first million copies sold out in pre-orders; and why thousands of devoted supporters across the country have been queuing up for hours to get her to sign the book.

And even then – after all this "Vesuvian hoopla", as Joe Klein put it in Time magazine – she still leaves us dangling. "Will I run for president in 2016? The answer is, I haven't decided yet," she teases on Hard Choices' penultimate page.

There's no point inviting her to repeat that frustrating formula. So instead I begin, when we meet in the splendour of the Peninsula hotel on Fifth Avenue, by asking her: What's America's problem? Why is the country that prides itself as being the crucible of civilised modernity so incapable of doing something so simple as choosing a woman as leader? It's not as though there haven't been precedents: Margaret Thatcher, Angela Merkel, Golda Meir, Indira Gandhi, Corazon Aquino, Dilma Rousseff… the list goes on and on.

Clinton, to echo what Bill told that same Democratic National Convention in Charlotte in 2012, has done the arithmetic: "The last time I checked there were 49 countries where a woman had been a head of state or government," she says.

She points to a number of factors that might explain America's backwardness, including the direct election of its presidents compared with Britain's party-led parliamentary system, and the vast expense of presidential campaigns, which can dissuade women from running. "I've often thought that the gauntlet of American politics is more individualistic, more expensive, more unpredictable than in many other democracies. I'm hoping that we get it cracked, because it's past time, but it's going to be difficult."

There's also the personal toll of what she calls the "very combative, even brutal experience of running for president". She knows all about that too. Her 2008 bid to reclaim the keys of the White House for the Clintons left her, she writes in Hard Choices, "disappointed and exhausted". She was bruised by the overt sexism she faced on the campaign trail. When I ask her to pick out any memorable moments, she tips back her head and lets out an effervescent laugh: "There were so many, where would I start?"

Facebook Twitter Pinterest Hillary Clinton with Barack Obama in 2008 when both were Democratic presidential hopefuls. Photograph: Kevork Djansezian/AP

She might start with Barack Obama, and his notoriously disdainful remark in one of their televised debates: "You're likable enough, Hillary." In the book, she describes her first meeting with Obama after she conceded defeat to him in the Democratic primary. They sipped Californian Chardonnay and stared at each other like two teenagers on an awkward first date. "We did have to clear the air," she tells me, "on some things that bothered me greatly."

The attack on Clinton-as-woman has shifted in none-too-subtle ways over the years. In Living History, her previous blockbuster memoir of her time as first lady, published 11 years ago, she complains that she was constantly criticised for showing too much feeling in public; by 2008 the prevailing wisdom was that she lacked warmth and empathy, she was PC to Obama's Mac.

It must feel at times, I say, that she's damned if she does, damned if she doesn't. She laughs raucously again, mirth appearing to be, incongruously, her way of acknowledging pain. She faced "a kind of cultural zeitgeist, if you will; there were so many comments that were so out of line. There's a huge emphasis on all the externals: how you look, how you dress, and many women in politics who take on this role are acutely aware they can never satisfy everybody, no matter how hard they try.

"But at a certain point you say: 'Forget it!' You are who you are, and you are either going to be successful in getting people to vote for you or you're not. But it's a very hard learning process to get to that point."

Which brings us, inevitably, to the most portentous part of our conversation: scrunchies. She was tempted, she reveals in the book, to ditch the title Hard Choices and rename the memoir The Scrunchie Chronicles, in reference to the stir she caused as secretary of state when she cast aside (female) diplomatic niceties and began to clip her hair back. "If I want to pull my hair back, I'm pulling my hair back," she writes defiantly. The remarks have in turn caused a renewed debate over the past few days about hairstyles in public life, and I remark that sales of scrunchies must be going through the roof.

"I wish I had some stock in a scrunchie company," she concurs.

Appearances are pertinent in this regard – towards the end of her stint at the State Department, in which she scurried between 112 countries and spent more than a year of the four years on the road, there were times when she looked immensely tired and gaunt. She recalls in Hard Choices how she would grab sleep whenever she could – on planes, in cars, power naps in hotels – because she never knew when her next proper rest would come; she would even dig the fingernails of one hand into the palm of the other to stay awake in meetings. But the Hillary Clinton who presents herself in the Peninsula hotel looks transformed. She seems relaxed, recharged, energetic (and yes, her hair is down and she's wearing a dark trouser suit with a loose neck tie. There, I've said it).

Clearly, something about her time since she left Foggy Bottom and returned to "ordinary" life has suited her. Maybe it was the peace and quiet of the cosy, carpeted, sun-drenched study at home in Chappaqua, upstate New York, with views over the treetops, where she wrote much of Hard Choices in longhand. Maybe it was hanging out with Bill, now that they are both out of office for the first time since 1983 – walking their three dogs, binge-watching House of Cards, and generally "continuing a conversation that began more than 40 years ago at Yale law school and hasn't stopped yet". Maybe, as she told People magazine, it was the calm of organising her closets or doing water aerobics and yoga.

And then, of course, there's what she calls the "unabashedly giddy prospect" that she is about to become a grandmother, with all the excitement and anticipation that brings. I ask her what hopes she has for her granddaughter or grandson (Chelsea's child is due in the autumn). She says she hopes the child will grow up to feel "optimistic, positive, can-do. That it was really up to him…"

"Him or her," I interject.

"Or her," she agrees, hurriedly.

I ask her whether she's just inadvertently handed me Scoop of the Year. I can see the headline blazing around the world: "Clinton Lets Slip Chelsea's Baby Is A Boy".

"No I haven't, I really haven't," she insists. "No, I honestly haven't. I'm just using the usual pronoun."

Funny that. The woman who (undecided though she may be) yearns to become the first female in the most powerful job on Earth uses the conventional male pronoun when talking about her unborn grandchild.

That's about the only point in our encounter when she is in less than total control of the message she is transmitting. You quickly sense that Hillary Clinton, supreme operator that she has become, is all about control, all about guarding the presentation of her own image that is arguably the politician's most important challenge in the age of social media and 24-hour cable television. That sense of guardedness is heightened by the constant presence of secret service agents hovering around her, and by the figure of Huma Abedin, her long-time aide, who has been constantly at her side since the White House and who slinks into an adjoining room while we are talking.

Facebook Twitter Pinterest Hillary Clinton with daughter Chelsea and husband Bill at the Democratic National Convention in 2000. She hopes Chelsea's baby, due later this year, grows up to be 'optimistic, positive, can-do'. Photograph: Paul J Richards/AFP

Take the impression of "Hillary Clinton at rest" that she has put forward, with all that People talk of closet organisation and water aerobics. That's one way to frame her life since she stepped down as America's diplomat-in-chief. Another way, though, would be to see the past two years as extraordinarily hectic, almost on a par with her frantic schedule as secretary of state. Over the past two years she's earned $5m on the speaker circuit, worked alongside Bill and Chelsea to promote their foundation, the Clinton Global Initiative, set up several new charities, joined Twitter, and held numerous breakfasts and dinners with trusted friends and advisers to deliberate on whether to stand in 2016. Not forgetting writing 688 pages of Hard Choices and the gruelling book tour that will take her all over the US and Canada. The last time she did a tour like that, for Living History, she had to soak her wrists in ice to reduce the swelling from hours spent signing copies.

What's that about, I ask her. Why does she push herself to such extremes?

"It comes from the way I was raised," she says. "It comes from the example of my parents. My father was a small-businessman and if he didn't get up and go to work there would be no business. My mother had a very difficult childhood; she worked to overcome it and provide a great home for us."

Clinton writes movingly about her mother, Dorothy Rodham, who died in 2011. Dorothy's childhood in Chicago was marked by trauma and abandonment. She was rejected by her own parents and brought up by severe and unloving grandparents, who confined her to her room for an entire year just for having gone trick-or-treating without permission. Dorothy kept her hopes alive, avoiding bitterness, by taking comfort in other people's small acts of kindness.

"My parents instilled me with a sense of individual responsibility, a work ethic, and a belief in service. That's who I am. And I like it, it's not a burden to me. If you have resources and abilities, you should put them to work."

Which brings us seamlessly back to the question that everybody is asking, and that Clinton is studiously declining to answer. Will she, or won't she?

Rarely has there been a potential presidential candidate so battle-hardened and ready for combat. As former first lady, she knows how the White House operates; as former senator she is steeped in domestic policy and congressional politics; as secretary of state she is excessively familiar with the world stage. She has a further inbuilt advantage: she knows her enemies intimately.

And there are plenty of those. From the left, many American liberals remain wary of the Clintons in general, and Hillary in particular. Her contentious vote in 2002 in support of President George Bush's war resolution on Iraq played heavily against her in 2008, and remains an open wound. She tries to suture it in Hard Choices, admitting that she got Iraq "wrong. Plain and simple." (Though she can't restrain herself from blunting the confession by adding: "I wasn't alone in getting it wrong".)

Then there's the issue of entitlement, the suspicion, deeply held by some on the left, that the Clintons have an assumption of ownership over the Democratic party and over high office. The sensitivity of the subject has been exacerbated by hints from Chelsea that she, too, now has political aspirations, and by talk of Jeb Bush running on the Republican side in 2016, thus raising the possibility of an endless Clinton-Bush-Clinton-Bush cycle stretching far into the future.

Isn't the American tendency towards dynasty – the Kennedys, the Bushes, the Clintons – a weakness in its democracy, I ask her. After all, the country was formed upon the founding fathers' rejection of British monarchy and its hereditary principle.

"I don't think so," she replies. "It's a choice that people make, it's not ordained like aristocracy. In our system, people get to vote for you or against you. I don't think it's at all weakening of the system if people with the same last name put themselves forward to the electorate, when their experiences, their character and in my case gender may be different."

And money? What about money? Bill and Hillary have reportedly made more than $100m since they left the White House in 2001. Yet that didn't stop Hillary complaining to Diane Sawyer on ABC News that the couple had emerged from highest office "dead broke", a comment that ranks for its tone deafness alongside John McCain's admission in the 2008 presidential election that he couldn't remember how many houses he owned.

America's glaring income inequality is certain to be a central bone of contention in the 2016 presidential election. But with her huge personal wealth, how could Clinton possibly hope to be credible on this issue when people see her as part of the problem, not its solution?

"But they don't see me as part of the problem," she protests, "because we pay ordinary income tax, unlike a lot of people who are truly well off, not to name names; and we've done it through dint of hard work," she says, letting off another burst of laughter. If past form is any guide, she must be finding my question painful.

On the other side of America's great partisan divide, Clinton's enemies are all the more vocal. Like vultures, her detractors on the right are already starting to circle before she has even declared her intentions. "Yeah. They're waking up," she says with a smile.

All the old favourites are back. Karl Rove, George Bush's former attack dog, has been cattily suggesting that Clinton may have brain damage from her 2012 fall at home in Washington. Matt Drudge mischievously wondered in a tweet whether the People magazine cover shows her holding a walking frame (she is in fact leaning on a patio chair) – a play on the fact that she would be 69 at her inauguration, were she to run and win in 2016. The ribbing was all the more caustic given Drudge's status as the Clintons' historic bête noire – it was the Drudge Report that broke the Monica Lewinsky story in 1998. For good measure, Lewinsky, that "narcissistic loony toon", as Hillary is said to have dubbed her at the time of the scandal, has also made a comeback, writing an account of her humiliation in Vanity Fair.

Given that menacing line-up of spectres from Clinton's past, I put it to her that she must feel like she's back in the 90s, reliving those dark days when she famously spoke of a "vast right-wing conspiracy".

"I feel like they haven't changed," she says. "I've moved on, but they're still fighting a rearguard action. They would like to turn the clock back so my husband never won [the White House], I wasn't elected to the senate, Barack Obama never won, and on we go… I think that's a kind of sad commentary, to be honest."

Why does she think that she, like Bill before her, is such a lightning rod for conservative hostility?

"It started with the visceral negative reaction towards my husband's success. Because I am outspoken about inequality and economic opportunity, they transferred a lot of that feeling to me, and if you add to that I'm a woman saying these things, and that I'm prepared to stand my ground, well it's a recipe for lots of back and forth."

As she talks about her adversaries, lumping them under that non-specific but unmistakably bilious "they", she pulls her shoulders back, straightens her back, and looks me straight in the eye. She's like a bull stomping its hooves before a charge.

I'm given a two-minute warning at the end of our encounter, which is timed, fittingly, with military precision. I lob her one final question. What shape does she hope America will be in when her as yet unborn grandchild is 10 years old? (I leave it implicit, but that's the age the child would be when his — or her — grandmother completed two full terms in the White House.)

"I would like the social fabric that has begun to fray to have been repaired, for people to feel we were all in this together, that the American dream was real, not some distant vanishing image on the horizon, that fairness had been returned to the economy and politics, that our education system was doing a better job and more kids were healthy, and that we were once again respected for our values and how we presented ourselves to the world."

Is that all? And she says she hasn't decided yet whether to run for president.