Daniel Ailey didn't give up playing football when a badly broken leg left him on the sidelines for a year. He didn't quit when a trial with Doncaster Rovers as a teenager ended in rejection. But Ailey, who is deaf and plays as a semi-professional for Potters Bar Town, in Hertfordshire, nearly walked away from the game following a home match in October against Grays Athletic, when the sound of the calls he uses to alert team-mates to his position was mocked by the away team's fans.

As the ground reverberated to the sound of fans imitating his vocal sounds, the game was briefly stopped by the referee, and the police were called. When the incident was picked up by the local paper, it was further inflamed when local resident John Griffin, boss of minicab firm Addison Lee and a Tory party donor, suggested that rather than taking action against the fans of Grays Athletic, the police should have "demanded that he [Ailey] discontinue making noises that could be misinterpreted by members of the crowd". Griffin later apologised for his comments.

Yet even as Ailey, who uses British Sign Language (BSL) as his primary means of communication, found himself at the centre of a heated debate, he was never himself interviewed. "Nobody asked me how I felt about it," he says. "The reporters should have spoken to me with an interpreter."

Ailey, 30, bears a resemblance to the Manchester United player, Danny Welbeck. Like Welbeck, he is a striker, strong and athletic, who has represented his country – Ailey has played for the Great Britain deaf football team. But he has a very different lifestyle from that of a Premier League player. He plays in the eighth tier of the English game, in the Ryman League, and is looking for welding work to supplement his wages.

Of the abuse he suffered that night, Ailey says: "I think it's the same as racism." He mentions the widely publicised racial abuse that England's under-21 side suffered during a match against Serbia earlier this season. "How I feel is the same. When I found out about [Griffin's] comments, it felt horrible."

Having come on to the pitch as a late substitute, the first Ailey knew of the abuse – because he couldn't hear it – was when the referee stopped the match. Later, when the ball went out for a throw in, he says: "I saw the fans copying my gestures, such as waving for the ball, and I saw them pointing at their ears. Then I saw the players [from both sides] shouting at the fans, asking them to stop." When the match ended, he was interviewed by the police. Upset, he asked them via his manager: "Would they do that to a Paralympian?"

After nearly 15 years in the game, Ailey was on the verge of quitting. "But I spoke to my mother, and she told me 'Please be strong – forget it and focus on football'. I realised that if I quit, the fans win. So I stayed. I tried to carry on for other deaf footballers."

Only a few deaf footballers have reached the top of the game. The most famous was Cliff Bastin, who played in the 1930s for Arsenal and England, holding the north London team's goal-scoring record for nearly 60 years. In the modern game there are only a handful of deaf players in the UK, all of whom are playing, as Ailey does, at semi-professional level.

Ailey has been abused on the pitch before. "A defender I played against once took the piss out of me. But I let my feet do the talking. I have a strong mind. I keep going," he says.

However, he found it harder to forget being mocked by a whole crowd of fans. The next game he played after the abuse by the Grays fans, he was inhibited, making fewer calls and gestures, as he says he didn't want it to happen again.

But his team-mates told him to carry on using his voice as before. "I still felt self-conscious because of the fans," he admits.

Potters Bar Town FC has strongly supported him, allowing him to take a break from playing. Club chairman Peter Waller says: "Daniel apparently was subdued afterwards, but still turned out for training and to play. I admire that and we cannot give in to ignorance."

So what sanctions have been taken against the fans responsible? The secretary of Grays Athletic apologised to Waller on behalf of the Essex club at the time of the incident, but he has since been informed by the police that Potters Bar cannot bring charges against any individual fans because Grays has not identified any of the culprits.

The Football Association is also powerless to impose any punishment on the club or its fans. An FA spokesman says that it carried out a full investigation and treated the incident "in exactly the same way as any other case involving discrimination by supporters". However, he added that the FA cannot take disciplinary action because it "has no legal relationship with individual football supporters". He added that, under FA rules, on match day the responsibility for managing the behaviour of supporters lies with the home football club.

The only action that the FA has taken, according to the spokesman, is a request for Grays Athletic "to do all it can to educate supporters that mimicking a person's disability is not considered acceptable" before the next game between the two teams, on 16 March. However, when Grays Athletic was asked, for this article, for its response to the incident and how it plans to ensure that its fans do not abuse Ailey for a second time, it replied: "The club is not prepared to comment."

This is in stark contrast to the sanctions clubs at a higher level have faced for their fans' transgressions, such as fines, individual fans being banned, ground closures or even the deduction of league points. It is clearly a concern for Ailey. "I want Grays Athletic to say that they're sorting it out with their fans. I'm worried about their fans at the next game, but I have to be strong and go and play."

On signing Ailey, Waller says: "I thought that his deafness was irrelevant, as long as the manager thought he could play football. The 'difference' is that his team-mates cannot just shout to him to pass the ball. Everyone is learning how to adapt and must talk to his face. Daniel, I notice, is also looking at his team-mates more."

The abuse Ailey suffered hasn't dented his love of the game he discovered on the playing fields of a boarding school for deaf children at the age of 16, but it has cemented a sense of difference, of being singled out for no fault of his own. However, he is proud of who he is and says: "When I am in the hearing world, I feel my deaf identity stronger."