I met Joe Casely-Hayford in the late 1990s through his brother Gus, who produced a documentary about me for Channel 4. I’d heard of Joe, this brilliant fashion designer, but I’d never met him until Gus made the introduction. I feel extremely privileged that he did.

Joe was a very quiet person, retiring, mysterious and dreamy, distant but very focused. He was like a deep pool. That mystery was what was so wonderful about him. He was very comfortable not being on the surface. It was like he was some kind of magician. Maybe it’s because he was a bit older than me, but it felt like he’d already peeked behind the curtain and understood life’s workings. The things that you come to realise much later on, Joe seemed to have already figured out.

I remember visiting him at his first-floor studio on Shoreditch High Street, east London, with its big, chapel-like vaulted ceiling. This was in the late 90s, in the middle of an edgy, creative London, but the studio felt slightly removed from all that. It was very quiet, very contemplative. I’d arrive in the late afternoon and look at the clothes he’d just made, and he’d play some music.

We’d talk about the 1980s, when he was starting out, as being quite a bleak period. After training at Saint Martin’s and the ICA, he launched his first label, KIT, in 1983 and started designing under his own name the following year. These were hard times for a black man, both creatively and in business, but he didn’t bear a grudge. Somehow he found a way to turn the difficulties into opportunities to reinvent himself and be innovative.

We worked together in 2002. I asked him if he would design a T-shirt to go with my Freedom One Day exhibition at the Victoria Miro gallery. He combined his logo and a monkey motif from one of my paintings, incorporating the red, black and green colours of Marcus Garvey’s African American liberation flag. Collaborating with Joe was seamless and I remember being amazed by the result.

My daughter once saw me wearing one of Joe’s tops and she said: “I like what you’re wearing, but is it a shirt or a jacket?” And I said, “Actually, I think it’s both.” Joe’s work always had that fusion, it could be two things at once.

He was very much a gentleman but also could be incredibly street. He fused the cut-and-paste of a hip-hop track with the cut of a Savile Row suit. He came from an incredibly accomplished family. Gus is director of the Smithsonian National Museum of African Art in Washington DC [soon to become director of V&A East]. His sister, Margaret, is chair of the board at the Globe theatre. Seeing them all having a conversation together was like watching the downtime of a very high-powered board meeting.

In fashion, Joe was quietly influential, always doing things differently, never really following trends. His practice was pretty discreet. Obviously, he’s been a massive influence on Charlie, his son, a menswear designer. Maria, Joe’s wife, still works with Charlie today.

I didn’t see Joe a lot after I moved to Trinidad in 2005, though I’d bump into him from time to time. The last time we met was in east London a few years ago. It was like we’d seen each other just the day before. I remember Joe as somebody who was extremely calm but very determined. He had a constant desire to learn and was always very interested in what was new, what young people were doing, how he could be close to the creative centre of things. It’s no coincidence that he looked and behaved a lot younger than he actually was.

Joe was the kind of guy who left no footprints in the sand. At the same time he was very, very funny and had a really infectious laugh. I feel extremely fortunate that I spent private time with him.