Post Malone’s rockstar, the No 1 song in the US and UK this week, is a tale of drugs and loveless sex – and joins similarly empty-sounding hits from 21 Savage, Future, Lil Uzi Vert and more. Why has mainstream rap become so bleak?

After releasing his single rockstar in mid-September, the US rapper Post Malone, accompanied by fellow MC 21 Savage, broke Apple Music’s streaming record with 25m plays in a single week. Since then, the pair have reached No 1 in the US charts, and now the UK charts, too, interrupting the month-long reign of Sam Smith’s ballad Too Good at Goodbyes and a year dominated by shiny, relatively toothless pop. With its dreary, droning loops, and dark lyrics about drugs and women, Rockstar is part of a menacing, nihilistic sound that is taking over the top 40.

A burgeoning generation of rappers are spitting bleak, hedonistic rhymes about prescription pills, loveless sex and premature death – and their presence on the charts swells week after week. Lil Uzi Vert’s XO Tour Llif3 spent 28 weeks on the Hot 100 charts with a catchy wail about a couple’s suicidal thoughts; Future extols an unnerving concoction of ecstasy and prescription drugs on Mask Off’s softly rapped chorus, “Percocet / Molly, Percocet”, creating one of the biggest rap hits of the year in the process; and 21 Savage reached No 2 in the US with his LP Issa Album, with tracks that advise “numb the pain with the money”, retelling Savage’s rise from an impoverished Atlanta childhood that saw him start carrying a gun at 14.

Other stylistically similar MCs, such as Kodak Black and XXXtentacion, create ostentatiously amoral rap that is matched by their own real-world criminality. Kodak Black was recently charged with first-degree sexual assault; XXXtentacion, already on probation for armed robbery, is accused of battering and strangling his pregnant girlfriend. Yet, after the revelation of gruesome details of these accusations, both artists reached No 2 in the US album chart in September, opening up questions about the ethics of enjoying music that revels in violent sex and amorality – especially when its creators have been accused of being sexually violent and amoral.

Comparing these MCs with other rap chart-toppers such as Chance the Rapper and Kendrick Lamar is, on the face of it, reminiscent of the 1990s distinction between the “conscious”, socially aware rap of KRS-One and Public Enemy – and its “gangsta” half-brotha, characterised by acts such as Ice T, 2Pac and NWA, with their grim yet vivid narratives of gunplay and crime. At the time, academics and outsiders often suggested that gangsta rap – with its misogynistic, homophobic overtures, portraits of dilapidated black ’hoods and tone of hopelessness – was not just inherently harmful but also encouraged more criminal acts in black and poor communities. In the decades since, many have come to grips with the idea that rappers can walk and chew gum simultaneously; that is, artists (including Lamar, Common and J Cole) can speak to the realities of an impoverished black existence while encouraging the street corners to critique the western world’s obsession with black crime.

So why are MCs such as Post Malone and 21 Savage diving back into a hopeless funk? Nihilism suggests that one finds pleasure, resistance and art meaningless, so as not to grow attached to the trappings of an illusory world. Take Vince Staples, who one writer called a “happy nihilist”, and who wittily addresses this emptiness. Yeah Right, from his 2017 album Big Fish Theory, finds the Long Beach rapper sardonically ticking off images of what other rappers might call the good life – “Is ya house big? Is your car nice? / Is your girl fine?” – before singer Kucka declares: “You pretend to get a better idea about the lifestyle you’re chasin’ / Keep pretendin’, make it real until every cell replaced, erased.”

Critiques like this are typical of black art, which is often defined by despairing and cathartic gasps into an anti-black ether. In this context, even the pursuit of wealth can become loaded with meaning. Jay-Z and Kanye West’s Niggas in Paris, with its gloss and gravitas, is about revelling in success in spite of anti-blackness: “If you escape what I escaped, you’d be in Paris getting fucked up too!” Nihilism and negativity show up in celebratory and critical black art – think Jean-Michel Basquiat, Toni Morrison and Ralph Ellison – so to immediately dismiss aloof, degenerate and, yes, gangstafied rap music as merely nihilist isn’t a deep enough reading.

Whether it’s the blues musicians of the past responding to the broken promises of America’s post-emancipation period; the bitter questioning of Marvin Gaye and Syl Johnson; the flipping off of the LAPD by Ice Cube; or Queen Latifah attempting to quell tensions bred over years of misogyny from men in the industry, black music is an endless loop of observing, feeling and talking back. In the 90s, various factors – the increased access to images of police brutality, a crack epidemic in black neighbourhoods and a growing wealth gap – infuriated some to the point of militancy. But alongside those protests were “gangsta” artists including Bone Thugs-n-Harmony and Nate Dogg who, with their portraits of black people just getting by, wailed just as wearily as their more abrasive counterparts, and rooted themselves just as staunchly in rebellion.

Today is no different. Beside political discussion from artists such as Lamar, Run the Jewels and Jay-Z – on everything from economics in black communities to the white supremacist in the Oval Office – there are young, disenchanted rappers reflecting, however indirectly, the emotional toll of these fights, and the ennui of newfound wealth. They have built on the musical aesthetics of Bone Thugs and Nate Dogg to create an introspective and downright tortured pining, somewhere between a rap and a melody.

Meanwhile, the audience expectation for a trap rapper – one with a criminal past or present, who reflects on that life in their lyrics – is that they live on the legal periphery. Perhaps that is why when those artists fulfil expectations, they gain even more popularity, as with XXXtentacion and Kodak Black. But when an artist doesn’t match the image their music suggests – when an artist lives at a distance from the nihilist trappings of the trap – the reactions show how malignant and ingrained assumptions about black musicians have become.

21 Savage recently supported his famous feminist girlfriend, Amber Rose, at her third annual sexuality-celebrating SlutWalk, carrying a sign declaring “I’m a hoe too”. To many, his presence belied the nihilistic persona of his music – or as Amber Rose would later say on Instagram: “How is my man not a Savage any more? Like, do y’all want him to pistol whip me? Beat me up? Shoot up the SlutWalk?”

The instance ties back to another this year, when Savage mentioned Amber Rose helping him with his diet: “She makes me do shit I don’t normally do, like take vitamins ... and drink water.” There was some mocking of this on Twitter, of course, but no one seemed to question how difficult it might be to access vitamins or even clean drinking water in the kind of impoverished black community Savage grew up in, or the role environmental racism plays on harmful consumption habits.

Savage, of course, has a self-aware sense of humour, be it with that blunt placard, or the naming of his album Issa Album after his deadpan explanation of a face tattoo – “issa knife” – became a meme. But even with caveats, not only do these instances speak to the particular misogyny inextricable from trap rap – Savage was mocked because Rose was taking care of him – but also to the idea that Savage and rappers like him shouldn’t believe in certain social or political causes. It’s the “stick to sports” retort faced by numerous American sports stars and journalists. Hip-hop, and most other industries thriving under a racist, patriarchal order, is full of men who have experienced stunted growth and difficult childhoods, and we shouldn’t denigrate those who seek maturity and try to transcend the bleakness they rap about. For Savage, the jury is still out on whether that maturity will extend into his future work, but the new video for his track Nothin’ New suggests he is thinking about larger systems of oppression and forms of resistance. In it, clips of Colin Kaepernick, Lebron James and Barack Obama open a film in which a young black man is shot by the police.

Anyway, the darkness that feels baked into current rap – the frightening disillusionment, the drug use, the sex devoid of real pleasure and the dystopian sonic gloom rumbling underneath – is not unique to hip-hop, nor black art, nor this particular time period. According to a recent study of more than a million songs, while references to drugs have jumped since the 90s, it’s country musicians who mention recreational drug use the most.

Stereotypically black criminality, rambunctiousness and rebelliousness has always excited white audiences, but never to the point of them wanting to solve it. It’s just entertainment, until they realise the same vicious system affecting black people resides in their midst as well; that drugs, misogyny and rape culture are as insidious in white spaces as they are in black ones, and sometimes even more so. It’s not black nihilism, but something more universal, that is sitting at the top of the charts.