Life was in a rut and New York City was calling. The Park Slope neighbourhood of Brooklyn – its bookshops, cafes and general civility – was an antidote to drab, unhappy Norwood Junction in depressing south-east London where, ten years ago, I existed day-to-day. Everything in my life had become geared towards securing my next fix, and the music that kept drawing me back. Nothing at home could match the improvisational majesty and ecstatic grooves of saxophonist David S Ware, bassist William Parker and pianist Matthew Shipp.

And nothing could have prepared me for my first in-the-flesh encounter in a New York jazz club with the David S Ware Quartet - drummer Guillermo E. Brown, Ware, Parker and Shipp. What was this music called? I couldn't find a convincing label - a fact I liked. Was this free jazz? Ware, who died in 2012, embodied the exploratory musical instincts and spirit of 1960s free jazz pioneers John Coltrane and Albert Ayler. The inspiration of his other great saxophone hero Sonny Rollins resounded inside every melodic turn and overarching phrase, and yet his music understood something very fundamental: merely retracing the dislocating rhythmic ricochet and melodic cry points characteristic of the 1960s mother music was no freedom at all.

The title of Ware's 1997 album, Wisdom of Uncertainty, was as much a statement of intent. Chewy, inchoate slabs of composed beginnings were a call to improvised action, or a Ware Quartet performance might equally generate itself as the musicians zoned inside each other's emerging patterns: a toe in the water triggering a torrent of sound. A Ware gig might leave you punchdrunk from the brutalist beauty of his music's relentlessly scattering lines and forms; or with your emotions buoyed by floating on a slipstream of euphoric grooves for longer than is usually considered decent.

One thing was for sure: this music felt thrillingly at odds with idea of jazz that was then being peddled by the major record labels desperately trying to make jazz economics add up. The once-treasured Blue Note label was reduced to marketing their star saxophonist Joe Lovano's theme-park jazz – Celebrating Sinatra and Viva Caruso, while Wynton Marsalis' mantra that jazz was "America's classical music", and therefore a smiley repertory music safely frozen in time, dimmed the ambitions of anyone gullible enough to listen. Ware, Parker, Shipp and their colleagues begged to differ: jazz for them was very much now, a music that roamed free in the realm of speculation.

Into this ideological minefield I stepped, an innocent abroad. I'd heard a few CDs, and about Shipp's infamous punch-up with jazz writer Stanley Crouch, the Peter Mandelson scowl behind Marsalis' Tony Blair grin. I circled the scene surrounding Ware trying to find a point of entry both as a jazz-lover and as a music writer, and a few things quickly became obvious. David Ware didn't do small talk, but Matthew Shipp was happy to shoot the breeze and spoke with devastating candour about his own vision of music. I was genuinely taken aback when a few minutes into our first interview he said: "I love Coltrane, I love Ornette, I love Monk – but fuck them all, fuck the tradition and certainly fuck Chick Corea and Keith Jarrett. I'm weary of being thought of as a free jazz pianist. I want to find new ways of messing with people's minds." And as I disappeared into the New York night – every night – to sometimes obscure little jazz clubs around Brooklyn, Queens and downtown Manhattan for a nightly dose of jazz, those words stayed with me.

For a while, Shipp junked the whole jazz format and such albums as Equilbrium and Antipop Consortium vs Matthew Shipp unpicked the processes of DJ Culture and funk from the vantage point of his steely improviser's gaze. Since then Shipp has reintegrated the lessons of those records into his work as a solo and group improviser, his elastic rhythms and coat of many melodic colours re-energised by thinking beyond the acoustic box. Typically, when Shipp begins his three-day residency at Café Oto tonight, he'll be playing with British free improviser John Butcher and German synth improviser Thomas Lehn – musicians for whom jazz represents a marginal creative interest.

Ware has gone, handing responsibility to the likes of William Parker and Shipp to continue renewing the quest for freedom by keeping basic jazz beliefs under scrutiny. I was fortunate to have wound up in New York when it felt jazz was in serious flux, and witnessing these musicians confront the music as they were playing it was ear-opening and intellectually liberating. The formats of my own life came under question, and after three years of shuttling backwards and forwards between London and New York, I relocated not to New York, but at least to a different part of London, away from depressing Norwood Junction, happy never to return there.

• Matthew Shipp is at Cafe Oto, Dalston, London, 19-21 February, 2014.