“Sometimes it gets down to ‘You’re not black enough for them. You’re not R&B enough. You’re very pop. The white audience has taken you away from them.’” This was Whitney Houston, reflecting on the first significant setback of her career, when she was booed at the 1989 Soul Train Awards. By that stage, she had already won 11 American Music Awards, two Grammys, achieved the biggest-selling debut album by a female artist in history and a record-breaking seven consecutive US No 1 singles. But, despite all this success, some black radio stations refused to play her records, and opponents, including the Rev Al Sharpton, labelled her “Whitey” Houston. For some, she was simply Not Black Enough.

Kevin Macdonald’s new documentary, Whitney, doesn’t suggest insufficient blackness as a contributing factor in Houston’s tragic death in 2012. It points the finger more at her family, revealing her childhood sexual abuse (allegedly by her cousin Dee Dee Warwick) and scrolling through a gallery of money-grubbing, vicariously ambitious, gravy train-riding relatives. But Nick Broomfield’s earlier documentary, Whitney: Can I Be Me, does locate Houston’s demise specifically in her conflicted racial and sexual identity, starting with that hurtful night at the Soul Train Awards. “I don’t think she ever recovered from it,” says her saxophonist Kirk Whalum in Broomfield’s documentary. “It was one of those boxes that was checked, that when she ultimately perished it was because of those boxes.”

Looking at today’s landscape, where African-American music dominates the culture, it’s hard to believe Not Black Enough was ever a life-threatening condition. But where a white artist can “sell out” their fanbase at worst, it has always been different for black ones. As Ta-Nehisi Coates recently put it in The Atlantic, “the gift of black music, of black art, is unlike any other in America, because it is not simply a matter of singular talent, or even of tradition, or lineage, but something more grand and monstrous”. His point was that African-American musicians are expected to represent more than simply themselves; they must represent their culture and their history. “The gift can never wholly belong to a singular artist, free of expectation and scrutiny, because the gift is no more solely theirs than the suffering that produced it.”

Houston was pretty much engineered to be Not Black Enough from the start. She began with all the right credentials. She grew up in Newark, New Jersey, in the era of race riots, though she was hardly a child of the ghetto (she went to a private Catholic girls’ school). Her musical ancestry was very much rooted in the soul-gospel tradition, via mother Cissy Houston, cousin Dionne Warwick and honorary aunt Aretha Franklin. But when she signed to Arista Records, aged 19, CEO Clive Davis consciously moulded her into a white-friendly pop princess. “Anything that was too ‘black-sounding’ was sent back to the studio,” Arista’s head of promotion says in Whitney: Can I Be Me. “We didn’t want a female James Brown”.

Instead, it was syrupy ballads and perky dance-pop, complemented by innocuous music videos featuring as many white dancers as black ones, the better to appeal to MTV. In interviews, Houston’s persona could be described as colourless in every sense. Commercially it worked a treat, but “to the black ear, these records did not have a natural feel,” Arista promoter Doug Daniel tells Broomfield. “For the black audience, the perspective in the community was that Whitney had sold out.”

Houston in The Bodyguard, released in 1992. Photograph: Alamy

Houston began trying to put that right the very night of the Soul Train Awards. It was there she met future husband Bobby Brown – then the epitome of a virile, street-credible black man. She also broke out of her mould and began to record music more to her own tastes. For her third album, 1990’s I’m Your Baby Tonight, she worked with producers LA and Babyface and Stevie Wonder, and sounded funkier, sexier and altogether blacker. But throughout her career, Houston danced awkwardly around issues of race. She was firmly anti-apartheid South Africa and was the first major artist to perform there after Nelson Mandela’s election. She went stratospheric with The Bodyguard, but the movie barely acknowledged the interracial aspect of her romance with Kevin Costner, and scored her a huge hit with, of all things, a Dolly Parton cover (I Will Always Love You).

There was also Houston’s celebrated rendition of The Star-Spangled Banner at the 1991 Super Bowl, 10 days after the start of the first Gulf war. African Americans have traditionally viewed the national anthem with ambivalence. There was that problematic suppressed third verse for starters (“No refuge could save the hireling and slave / From the terror of flight or the gloom of the grave”). The song has prompted black protests from the Black Power salutes of the 1968 Olympics to Colin Kaepernick in 2016, but Houston gave it her unironic all, backed by brass, flags and fighter jets. She captured the patriotic moment so well, her rendition was released as a charity single for the US military.

One observer in Macdonald’s Whitney film frames Houston’s race issues in terms of “double consciousness”. The notion, first coined by WEB Du Bois in 1897, refers to the dual mindset held by marginalised peoples such as African Americans in the US, who must both retain their own black identity and also see themselves through the eyes of their oppressors. “Two souls, two thoughts, two unreconciled strivings; two warring ideals in one dark body, whose dogged strength alone keeps it from being torn asunder,” Du Bois wrote. Some would go even further and say Houston’s status as a woman counts as a triple consciousness. This “insider-outsider” mindset was regarded as an internal conflict but also a source of creative energy. Identify too much with one side, however, and you are Not Black Enough.

In retrospect, you have to wonder why Whitney was singled out. Many African-American artists have crossed over into more mainstream, traditionally “white” success without similar consequences – often because of where they’ve come from. In Houston’s era, crossover artists such as Diana Ross, Stevie Wonder and Tina Turner had paid their dues, starting at the bottom. Prince could stray into rock ballads such as Purple Rain on the back of deep funk credentials. By contrast, apart from her famous relatives, Houston was discouraged from mentioning her past by Arista. Even Michael Jackson – Houston’s only real equivalent – had evolved from a solidly black Motown tradition before the public’s eyes, although his physical transformations spoke of his own internal battles with his identity, racial and otherwise.

Houston with Dionne Warwick and Bobbi Kristina Brown. Photograph: Larry Busacca/Getty

The ground was already starting shift back in 1989, albeit too late for Whitney. While she was being hounded at the Soul Train Awards, writer Trey Ellis wrote an influential essay on what he called the New Black Aesthetic. Ellis identified himself as a “cultural mulatto”: educated, liberated by the civil rights gains of his parents’ generation, and operating freely in a multiracial mix of cultures.

“We no longer need to deny or suppress any part of our complicated and sometimes contradictory cultural baggage to please either white people or black,” he wrote. It was OK to like Jim Morrison as well as Toni Morrison. Ellis pointed out black culture’s growing cultural dominance, from Spike Lee and Eddie Murphy in the movies, to Prince and hip-hop in music – all of whom were universally popular but true to themselves, “true to the black”. By contrast, he singled out Whitney Houston and Lionel Richie as “cultural-mulatto, assimilationist nightmares; neutered mutations instead of thriving hybrids. Trying to please both worlds instead of themselves they end up truly pleasing neither.”

Today’s black artists operate with the freedom Houston dreamed of, but they are never immune from scrutiny. Witness Beyoncé, whose stance in the early 2000s was seen as carefully apolitical by critics, but who suddenly rediscovered her blackness in the Black Lives Matter era and came out celebrating hot sauce and “Jackson 5 nostrils” in 2016’s Formation. Beyoncé had sung her own, Houston-influenced Star-Spangled Banner to the Super Bowl audience in 2004; when she returned for the 2016 half-time show, she practically gave them a Black Power military drill.

The extent of the jolt was registered by Saturday Night Live’s brilliant sketch The Day Beyoncé Turned Black, in which white America’s realisation of the singer’s race triggers panic in the streets. As Jeff Guo put it in the Washington Post, “Beyoncé waited until black politics was so undeniably commercial that she could make a market out of it.” But Beyoncé and Jay-Z have continued to push the line. Their recent video for their joint track APES**T, filmed in the Louvre, turns it around and accuses European cultural history of being Not Black Enough.

And look how far Kanye West has pushed that line recently: all the way to claiming that slavery was “a choice” and brandishing his signed MAGA hat with pride. If that’s how far you have to go – to the edges of white nationalism - to earn a Not Black Enough charge these days, perhaps that’s a sign of progress. As Ta Nehisi Coates scathingly observed, “West calls his struggle the right to be a ‘free thinker’, and he is, indeed, championing a kind of freedom – a white freedom, freedom without consequence, freedom without criticism, freedom to be proud and ignorant; freedom to profit off a people in one moment and abandon them in the next.”

However that freedom is used, Whitney Houston is undoubtedly one of those who helped earn it, and she undoubtedly paid the price for it.

Whitney is out now