Sepp Blatter has issued a public apology for his comments on racism in football but insisted he will not resign as Fifa president. Blatter said sorry on Friday for making remarks which suggested that any racist abuse on the football field could be settled by a handshake at the end of the match.

"I am sorry and I regret that my statements earlier this week have resulted in an unfortunate situation," he said. "I am committed to the fight against racism, have no doubt about that. And I want to make it very clear: I will not stop until we have stamped racism out of football."

The apology was read out by Tokyo Sexwale, a black Fifa official and South African government minister seen with Blatter in a photo that was circulated on Wednesday in what some saw a cynical ploy to burnish Blatter's anti-racism credentials.

The contrite football chief, who has been condemned by the prime minister David Cameron and others, underlined his apology in an interview with the BBC in Zurich, saying he "deeply regretted" using "unfortunate words".

He added: "When you have done something which was not totally correct, I can only say I am sorry for all those people affected by my declarations. It hurts and I am still hurting because I couldn't envisage such a reaction."

Blatter, who was re-elected to a four-year term in June, has faced calls in Britain for his resignation but made clear he intends to stay put. "I cannot resign," he told the BBC. "Why should I? To leave would be totally unfair and not compatible with my fighting spirit, my character, my energy."

Speaking in Johannesburg, Sexwale said: "It takes a big man to say 'I'm sorry'. In Fifa we kick the football and not the man … I think for him to bend his knee like this shows the way. I hope it will help us send the message of anti-racism and discrimination worldwide."

The minister said he is not willing to "crucify a man or kick him when he's down" and would "find it hard to go after a man who has apologised" but stopped short of ruling out Blatter's resignation. "Such statements are quite damaging but the magnitude and intensity of the damage cannot be known," he said, adding that when this became clear, the Fifa executive committee would have to decide Blatter's fate.

Sexwale said he called Blatter on Thursday night and "could feel this was a man full of contrition". Sexwale neither sought nor received a personal apology for the use of the photo. "I'm informed that in their response Fifa used the picture of Mr Blatter and myself," he continued. "In some circles this may have inadvertently created the impression Mr Blatter was trying to clean up his image."

Sexwale, who was a member of the armed wing of the African National Congress during the struggle against racial apartheid, now sits on a Fifa committee that campaigns against racism and said Fifa uses his image "from time to time worldwide".

"I don't want to suspect any motive behind the authorisation of that picture," he continued. "But a perception, impressions will be created in certain circles worldwide that this was an attempt at damage control or clean up one's image.

"But I would like to believe that perhaps it was used to say, 'We are so committed, here's a man we all know is a freedom fighter, and he has fought against racism, how I can be involved in racism when I work with people like him?' That's my interpretation."

Sexwale acknowledged, however, that Blatter's recent comments were "flippant" and ill-judged. "I'm a freedom fighter. This statement disturbed me … What was problematic with his initial statement as reported was when he said racism does not exist on the pitch. This was rather unfortunate. We have had countless cases of racial slurs being spewed on the playing field by players against others.

"We should distinguish racial remarks from things that are said when players are fighting for a ball. If you say a player is not worth his salt, or you are rubbish, you are not fit to be here, you are not worth the money … at the end of the day handshakes as well as the exchanging of jerseys is about these things, even though we don't encourage them.

"If you use a racial slur it won't go away, you cannot exchange a jersey, you can't mitigate it with a handshake, that's why we call for action to be taken against players. We should differentiate between racial remarks and remarks when players are fighting for the ball."

Sexwale added: "I think what Mr Blatter was saying, he mixed up those emotions and was trying to say 'Look, let's get on with the game'. It was an unfortunate mistake. But he has inadvertently raised the bar in the fight against racism on the pitch."

He also insisted that the Fifa president is no racist. "Mr Blatter is well known worldwide as a champion against racism and discrimination. It is well documented that he fought tooth and nail to have the first Fifa World Cup in Africa, which was hosted in South Africa in 2010.

"His engagement with the Nobel Prize winners Nelson Mandela, Desmond Tutu, FW de Klerk and presidents including Thabo Mbeki and Jacob Zuma are well documented to assure them of his commitment to the success of the Fifa 2010 World Cup."

It was also Blatter who influenced Fifa's decision to recognise a football association on Robben Island, where prisoners had used the sport to further their cause. "It is therefore completely unthinkable to refer to such a man as a racist."

Sexwale also called on the English FA to deal harshly with any players found guilty of racist abuse and said "a slap on the wrist will not suffice". He also suggested that John Terry should not have captained England in Tuesday's friendly against Sweden while the investigations into allegations of racist abuse towards Anton Ferdinand remain ongoing. "It is not helpful for any side to honour him with the captains armband while the case is fresh," Sexwale said.