Vince Hilaire is sitting in the conservatory of a trendy seafront restaurant in Old Portsmouth, a pristine copy of his autobiography close to hand and evocative accounts of a bygone era on tap. He talks up Terry Venables and Crystal Palace’s Team of the 80s and creases into fits of giggles with tales of Alan Ball and his pack of “mongrels” who restored Pompey to the top flight.

There are grimmer memories of the racism to which he was subjected as one of a number of homegrown black forwards who illuminated the English game from the late 1970s, and the depression which has hounded him from shifts on a building site to life as a roofer since his playing days. But the sparkle which made Hilaire stand out remains undimmed.

“All I ever wanted was adulation,” he says. “I remember running out at Palace and that first song they sung about me: ‘We’ve got a brown boy on the wing, tra-la-la, la-la’ to the Boney M tune … Now it would be seen as derogatory but I thought it was brilliant. ‘They’re singing about me.’ If my own fans were happy with me, I wasn’t bothered by what was thrown at me by the opposition.”

Hilaire remains an icon at Palace and Pompey, two of the six English clubs for whom he played over a 16-year professional career. Some would argue he failed to fulfil the potential first made public in a Thames Television documentary as a teenager. “Vince” was Malcolm Allison’s brain child, tracking a youngster’s journey into professional football but also capturing Hilaire the cricketer scoring a wristy 37 for Essex. A need for footage from a school sports day would necessitate him competing in 100m, 200m and 400m races and left him vomiting on the sidelines.

He would establish himself at the heart of a group who went from winning successive FA Youth Cups to briefly sitting atop the First Division. His talent earned him nine England under-21 caps, with his skill and vision compelling whether employed behind split strikers at Palace or on the flank with Pompey. Images of him in full flight, all splendid afro and blue and red sash, will adorn the big screens when the 58-year-old revisits Selhurst Park next month for the derby against Brighton. Fratton Park throws open its doors to him more regularly as an ambassador and member of Portsmouth’s hall of fame.

Now a resident of Southsea, he has found a home from home in the city to the extent that, when he spent a night in prison for unpaid parking fines in the early 1990s, he “felt safe because it was in Portsmouth”. This is a player who agreed an improved contract with Venables at the urinals at Stansted airport, has been bitten by Dennis Wise and, having incurred a six‑game ban for pushing a referee, argued that he had mistaken the official, clad in black, for the player who had fouled him, Garth Crooks. Plenty of his stories reflect an innocence, from selling his Cup final tickets to a tout to the drinking culture in Ball’s team, but the accounts of racist abuse are harrowing and he admits in the book he “should have been more sensitive of the social situation”.

“But I never saw myself as a figurehead,” he says. “The reason we didn’t say anything was because we wanted to get on. I’d hear: ‘Kick the black lad because they don’t have too much bottle,’ or ‘It might be a bit cold for him today.’ I’ve had coaches say it to me. The reason I didn’t come back with: ‘What the fuck do you think you’re talking about?’ is because I wanted to play. Big C [Cyrille Regis], Laurie [Cunningham] were the same. None of us liked it but there was no support like there is now. The banana throwing and the monkey noises ... back then, it was almost normal.”

Hilaire on the wing for Crystal Palace in 1979. Photograph: Colorsport/Rex/Shutterstock

An opposing centre-half shouted “Who’s marking Midnight?” on Hilaire’s senior debut at Lincoln. As an opposing player at Elland Road he had fans singing “Nigger, nigger, pull the trigger” as he prepared to take a corner. He once ran out to warm up at Anfield only for the pre-match murmur to be broken by a lone voice chanting “Day-o, day-o”, with the Kop then chorusing: “Daylight come and he wan’ go home.” “I remember going on a mazy against Nottingham Forest and being flattened. The player, to stop the ref booking him, apologised: ‘The little man was just way too quick for me.’ Then, once the ref’s turned around, he’s picked me up by my afro and said: ‘Get up, nigger. Do that again and I’ll have you.’ I chased after him, the ref’s blown up and booked me for reacting. But, even then, I didn’t speak to anyone about it afterwards. It really sounds wrong saying it but you were just used to it.”

The account meanders off into more conventional, if gruesome, brushes with the hard men of the era. “At Pompey, Bally had this idea that Pat van den Hauwe was a weak link at Everton and I’d take him to the cleaners: ‘He can’t play. Cannot play.’ The ball’s come to me from the kick-off, I’ve miscontrolled it, but it looked like I’ve done a trick and taken it round Van den Hauwe, so Bally’s screaming: ‘Told you! Get the ball to the little man. He’s got him on toast.’ The next time I had a chance to run at him was about 20 minutes in and Bally’s up again: ‘Go on, take him on ...’ Anyway, as they’re carrying me off on a stretcher, I remember Kevin Dillon peering over and muttering: ‘You really did have him on toast, didn’t you.’”

That Pompey team was awash with characters, from Mick Channon to Noel Blake, with Hilaire accepted from the moment he scored a diving header on his debut against Blackburn. At Palace the east-end boy and his peers had risen to earn the moniker Team of the 80s. “Football came too easily and, because of that, you never think it’ll go south. We’d just beaten Forest, the European Cup winners, and we all thought this was just a dead easy game. ‘Who’ve we got Saturday, boss? Liverpool? Bring ’em on.’

“Terry said to me before that game to keep an eye on Terry McDermott because ‘he makes these forward runs from nowhere’. I was just like: ‘Don’t worry about that, gaffer.’ Anyway, it was 0-0 approaching half-time and I’ve stopped to scratch my nose, looked up and McDermott was about 15 yards in front of me and gone … next thing the ball’s in the back of the net and I’m running to the other side of the pitch pretending I’d had nothing to do with it. They’d been playing in third gear and won 3-0 but it could have been 15. We’d never really known what it was like losing before. It snowballed from there.”

Hilaire’s form warranted a full England cap in those early days in the top flight but the break-up of that side would see his career stagnate. “It turned into a bit of a Fred Karno’s circus at Palace. I remember losing at Coventry where Clive Allen scored a goal that should have counted [the free-kick hit the framework at the back of the goal with the referee insistent it had struck a post] and Venners saying to us: ‘Look, we’ve just got to keep going and stick together. Things will come right.’ We came in on the Monday to find out he’d gone to QPR. Man overboard.”

Life after football has been tough at times on Hilaire. Depression left him listless and lonely – “it never leaves you” – and ventures into other lines of work were fleeting. His fear of heights limited his days as a roofer to jobs on bungalows, and the building site was a wake-up call. He thanks Portsmouth, the community, for embracing and saving him and, now a grandfather, spends his time as a part-time DJ with ambassadorial duties at a football club reborn. Kenny Jackett’s team are climbing. Roy Hodgson’s are attempting to stay up. Palace and Pompey still consider Hilaire one of their own.

Vince: the autobiography of Vince Hilaire, with Tom Maslona, is published on 22 March by Biteback Publishing (£12.99).