I was shocked to hear that England’s players are considering walking off if they hear racial abuse from the crowd when they play Bulgaria on Monday night. There is something poignant about the resilience black players have shown over the past 40 years. They have always reminded themselves of who they were representing and the decision has always been not to leave the pitch, no matter what is happening in the stands.

I think it will set the game back if England walk off.

The ramifications extend a lot further than the result. Many black people I have spoken to in and out of football think that racism will win if England head down this path.

Marcus Rashford says football ‘going backwards’ in fight against racism Read more

Where does it go from there? For many years I have said that racism in football provides fascist movements with a vehicle to promote their message and taking this initiative will play into their hands.

Players have to stay on the pitch. If they walk off, the people chanting abuse have achieved their aim. Are England going to walk off if they are 3-0 up with 10 minutes left? Will they make that statement? Think about it. The team bus is on the way to the game and fans are surrounding it, banging on it, shouting racial abuse. Are the players going to tell the driver to turn around and go to the airport? Of course not.

Another potential ramification is whether the racists will react badly if the teams go off. Will they feel wronged? Will they want some kind of payback outside the stadium? Will they aim their anger at England fans? What if there are black England fans at the game in Sofia?

Uefa and Fifa think that fans will self-govern, that other fans will stop the racist chanting when they hear it, but this just leads to anarchy on the terraces. The people doing this have an agenda – they want violence. The risk is going back to the dark days of hooliganism.

Facebook Twitter Pinterest Howard Gayle celebrates scoring for Birmingham in 1983. Photograph: Mirrorpix/Getty Images

Uefa and Fifa have to do more. They have to deal with it ruthlessly after the game and the punishment should be known before the game. This is a popular sport, a spectator sport.

I know that Gareth Southgate has said that England have faith in Uefa’s three-step protocol, but imagine people watching at home waiting for 10-15 minutes for something to happen if the referee takes the players off in the event of racist chanting.

We also have to look at it from a playing perspective. The players will need to warm up when they return to the pitch to guard against injury.

The problem is that Fifa and Uefa are reactive, not proactive; they act after the event. The punishments they hand out for racism have been menial. They should have a ruthless mindset if there is racist chanting. Be radical. If your fans racially abuse players or spectators your next three or four games should be away from home. If it happens, the ban should be increased.

I never thought about walking off when I was playing. It was not an option and black players back then probably would have been on their own if they had protested by walking off. Your manager probably would have just brought on a substitute.

England ready to walk off if Bulgaria fans racially abuse us, says Abraham Read more

What comes into question is your mental state. Modern players are paid vast amounts of money to be in this arena. If someone abuses you on the streets, away from that arena, how will you behave? Are you going to give up on life?

Players can call on sports psychologists these days. There are so many resources available to them. Players of my generation did not. We had to get on with it. That is how people judge your character. This is a social problem, not a football problem. The only way to silence the racist voices is to shut the stadiums down, to play behind closed doors.

The best advice I can give to Tammy Abraham or anyone is to play well. Score a goal, score a hat‑trick. Then go and stand in front of them again to say they cannot beat you, they cannot affect you. Walking off is tantamount to submission. They have won if you do that.

Show Racism the Red Card hall of famer Howard Gayle was Liverpool’s first black player, whose career included spells at Birmingham, Sunderland and Blackburn.