To the outsider, jazz can seem less like a living form of music than a set of signifiers: music played to an older audience, aware of its own history, redolent of the cool of a half-remembered, semi-fictionalised past. Dig a little further, and a different set of signifiers emerge: “difficult” music, often improvised, played in small clubs, to audiences of chin-stroking beardies. What the word “jazz” rarely summons is the idea of music that isn’t just perfectly of the moment, but helping to shape and influence groundbreaking artists in other genres. But that’s what it has become, thanks to a new generation of jazz musicians for whom working across disciplines is as natural as improvising in front of hardcore fans.

As the Los Angeles saxophonist Kamasi Washington puts it: “We’ve now got a whole generation of jazz musicians who have been brought up with hip-hop. We’ve grown up alongside rappers and DJs, we’ve heard this music all our life. We are as fluent in J Dilla and Dr Dre as we are in Mingus and Coltrane.”

The influence cuts both ways – from jazz to hip-hop and back again. Jazz musicians have always improvised over different rhythms but, if you go to a jazz gig these days, you’re likely to hear a lot of musicians playing over the “slugging” beat popularised by the hip-hop producer J Dilla. It’s that wonky, slightly drunken-sounding funk beat that seems to have joined the arsenal of rhythms used by jazz musicians, alongside such mainstays as swing, bossa nova and the jazz waltz. “It’s basically the sound of someone sampling a funk beat on an Akai MPC sampler and editing it wrong,” says Rob Turner, drummer in the Mercury-nominated Manchester jazz trio Go Go Penguin. “Instead of starting the sample at the ‘transient’ – the start of the beat – it starts fractionally after that point. So the snare drums and hi-hats are all in slightly the wrong place. It sounds sluggish and disjointed and slightly screwed up, but it also sounds quite cool. And it’s something that young jazz drummers have worked out how to play. Go around music colleges and you’ll hear student drummers dividing up a bar into countless subdivisions and working how to ‘slug’ fluently – somewhere between ‘swung’ crotchets and ‘straight’ crotchets. Nowadays, so many young jazz drummers have learned to play like that we’ve started to call it the ‘college beat’. It shows you how jazz musicians have thoroughly internalised the hip-hop they’ve grown up with.”

Hip-hop’s love affair with jazz goes back more than 30 years, when the likes of Stetsasonic, Gang Starr, A Tribe Called Quest and Jungle Brothers were sampling heavily from jazz records and explicitly drawing connections between the two African-American art forms. Rappers such as the Notorious BIG (who was mentored by the New Orleans saxophonist Donald Harrison), Nas (the son of jazz musician Olu Dara) and Rakim (who studied the saxophone) have talked about how jazz influenced their phrasing and diction. However, when jazz musicians such as Herbie Hancock, Miles Davis, Branford Marsalis and Roy Ayers reciprocated, they always did so as curious outsiders: as “tourists” rather than native speakers. Younger jazz musicians, however, treat things differently.

Washington has become the poster boy of this bilingual generation. Not only is his own spacey hard bop outfit filling out festivals and concert halls around the world – including a recent Prom – but he has also worked closely with hip-hop musicians, working as the musical director for Kendrick Lamar’s album To Pimp a Butterfly, the landmark 2015 album whose stellar cast also included trumpeter Ambrose Akinmusire, guitarist Keith Askey, bassist Chris Smith, drummer Robert Searight and producer Terrace Martin – all of whom have worked in the worlds of jazz, R&B and hip-hop.

“A lot of us grew up playing with Snoop Dogg’s band, the Snoopadelics,” Washington says. “And we all played in big bands and jazz quartets and so on. I never had a problem moving between jazz and hip-hop. People like to compartmentalise music, especially African-American music, but it’s really one thing. One very wide thing. I mean, it’s like all those great records by Marvin Gaye and James Brown back in the day – there are tonnes of jazz musicians playing on them. When I first played some Coltrane-type stuff on the Pimp a Butterfly sessions, Kendrick got it immediately. ‘I want it to sound like it’s on fire,’ he’d say. That’s the kind of common ground that the best jazz and the best hip-hop have.”

“There’s a tendency for the old guard to sneer at hip-hop,” says Robert Glasper, the pianist who also featured on To Pimp a Butterfly. “But jazz can learn so much from hip-hop. I’m reminded of how Miles Davis, when recording Miles Smiles, told [pianist] Herbie Hancock to not use his left hand at all. It made Herbie approach playing in a different way – he couldn’t play chords, so it forced him to find other ways to express himself. And it’s the same when you’re working in the world of hip-hop. You have to use repetition, you have to play sounds that cut through. Sometimes that can be even more difficult than the most complicated improvisation.

“If you want to create a great breakbeat, for instance, there is a sonic language and a feel that you have to understand. For instance, you can’t do it with an 18-inch bass drum [the standard size for jazz drummers]. You need one with a 20- or 22-inch diameter. You have to tune your snare drum differently. You can’t crash your cymbals every eight bars. Instead of adding fills, you have to remove components of that beat, or add juddering bass beats. And so on. If you’re not a hip-hop head, you won’t know that something’s wrong. The jazz musicians who have grown up with hip-hop, however, know this implicitly.”

You’ll find dozens of examples of this from young jazz musicians in their 20s and 30s. Check out pianist Kris Bowers and his solo versions of Kendrick Lamar songs on prepared piano; or drummer Dan Weiss’s solo versions of tracks by the likes of Busta Rhymes or Big Punisher; or Chris “Daddy” Dave recreating hip-hop breakbeats.

The bassist Thundercat is another example. As well as playing a key role on To Pimp a Butterfly (along with his brother, the drummer Ronald Bruner Jr), he makes his own, spacey soul-jazz albums and also collaborates with the LA producer Flying Lotus (himself a nephew of John Coltrane).

“Whether I’m working with Herbie Hancock, Kendrick Lamar or Flying Lotus, I try to get on that person’s soundwave and keep my mind open to what he or she is hearing,” he says. “You might be a gifted jazz improviser, for instance, but hip-hop will teach you how to edit yourself, how to connect to an audience, how to service a song. That is always crucial.”

Shabaka Hutchings. Photograph: Alamy Stock Photo

Speaking nearly 20 years ago, the British saxophonist Courtney Pine once remarked, in frustration, that jazz musicians would often enthuse about their love of funk, reggae or punk in private but never mention it professionally. “Put them in front of a camera or a journalist,” he said, “and they’d talk about Art Blakey and Duke Ellington, but they’d clam up if you asked them about Burning Spear or Diana Ross. They were ashamed of that side of their musical formation! And I think it’s why some jazz music stagnated.”

That seems to have changed . “The generation that emerged in the 1980s had to be a bit like that,” says London-born saxophonist Shabaka Hutchings. “Guys like Wynton Marsalis were young players who were keen to be taken seriously by an older generation of jazz musicians, and keen to assert that jazz was a high art form. Mentioning reggae or hip-hop might seem to dilute that commitment. So you ended up with a somewhat dogmatic genre, one that was quite self-contained.

“The difference now is that my generation don’t feel the need to solely identify ourselves with one genre. We’ve grown up with rave and electronica and hip-hop, and have no qualms about integrating that into our music. And those artists who have collaborated with Kendrick Lamar are perfect examples of that.”

To Pimp a Butterfly was certainly on the radar of David Bowie and the jazz musicians with whom he recorded his final album.

“It was an album that David talked about a lot, both with me and with producer Tony Visconti,” says saxophonist Donny McCaslin, the musical director of Blackstar. “It is a staggering piece of art. And oddly, I found Kendrick’s rapping – his intonation, his rhythms, his syncopation, the spirit of his phrasing – heavily influenced my own saxophone playing, both on Blackstar and on my new album. David was very much intrigued by a musician like Kendrick who was creating a personal vision: something that was a mix of many genres, but not really sounding like any of them. I guess it’s something that he has always done. I mean, what is Blackstar? It’s not a jazz record or a rock record, it was David’s personal vision. And it’s the same with To Pimp a Butterfly.”

Blackstar producer Tony Visconti has suggested that only jazz musicians could keep up with Bowie during his most creative moments. “That’s very kind of Tony to say that,” says McCaslin, with a laugh. “But, with David, the collaboration was very organic. He has been a big fan of jazz since he was a teenager – he’d talk about how he was obsessed with Stan Kenton and Gil Evans and Charlie Parker and Maria Schneider and whatnot. And he was comfortable in a jazz setting. It was like we were a four-man basketball team and he was the fifth member who fitted in seamlessly. Because that’s what jazz musicians do – one guy plays something, another hears that and moves in a different direction, a third covers for them both, and so on. And David was always in the heart of the action, a participant among these jazz musicians.

“I was doing a joint interview with Tony around the release of Blackstar, and he observed that jazz had always been present in David’s music – it’s just that it had always been a deeply embedded component. My own band have been exploring his catalogue and have started playing a few Bowie songs in our live sets, as a tribute – Warszawa, Look Back in Anger, Hello Spaceboy, A Small Plot Of Land, as well as Lazarus. There is a lot of interesting harmonic information, a lot of interesting intervals. As a jazz musician, they are very satisfying to play.”

Again, McCaslin’s own music draws as much from hip-hop and contemporary music as it does from jazz. His new album, Beyond Now, features versions of songs by Aphex Twin, MuteMath and deadmau5, while there’s a rhythmic sensibility that comes from being immersed in music by the likes of Squarepusher and Skrillex.

What is astonishing about all of these jazz musicians is the sheer number of radically different projects in which they are involved. Since finishing Blackstar, McCaslin has recorded his own new album, toured with Mike Manieri’s fusion band Steps Ahead and found time to play with Maria Schneider’s big band. Glasper has, in the past year alone, soundtracked Don Cheadle’s recent Miles Ahead movie and a Nina Simone biography, put together a Miles Davis tribute with assorted singers and released a new album of soul-jazz covers, ArtScience. He is also co-producing Herbie Hancock’s upcoming album.

Hutchings seems to be taking more work than even McCaslin and Glasper. As well as the Comet Is Coming – his Mercury-nominated collaboration with ravey synth-and-drums duo Soccer96 – he fronts his futuristic marching band, Sons of Kemet, and leads a new collaboration with a group of South African jazz musicians called the Ancestors. And that’s not to mention his work with the jazz-punk outfit Melt Yourself Down, the DJ/producer Floating Points, the hip-hop-influenced trio Thousand Kings, the Sun Ra Orchestra or the Sun Ra-influenced Heliocentrics.

Flying Lotus. Photograph: Peter Van Breukelen/Redferns

“It’s just a function of growing up liking lots of different music,” he says. “Be it bashment or hip-hop, or Björk or Antony and the Johnsons, or Fela Kuti or Thomas Mapfumo. And all that influences how you approach jazz. With the Comet Is Coming, for instance, we play in much clubbier venues than we might with Sons of Kemet. You see how stand-up audiences respond to certain rhythms, and that takes you in a certain direction.

“What I always find frustrating is that so many jazz musicians have forgotten how to write decent melodies. They are often so keen to show off their chops and their harmonic knowledge that they forget how to connect to an audience. That’s something that pisses me off. In fact, it was working with the Ethiopian musician Mulatu Astatke that really gave me the confidence to play simple, strong melodies – drawing from folkloric music. That’s something that all jazz musicians can benefit from.”

What seems to unite many of this new generation is the idea that jazz is less a genre and more a sensibility. “The thing that keeps jazz from being old is the fact that you can apply it to where you’re at,” says Thundercat. “Being a jazz musician is a mentality as well as an ability. It’s like being able to speak different languages – jazz is a tool that helps you, allows you to understand more, and can take you to different places.”