After I first moved to Brazil in 2012, I was struck – like many football-loving newcomers – by a mystery: why are the stadiums so disappointingly empty? It was the last thing I expected, having been brought up on a TV diet of rapturous Brazilian crowds, tales of the "beautiful game" and its enviably brilliant exponents like Zico, Pelé and Romário.

No matter how hard the cameramen tried to focus on the small pockets of fans, every wide shot of the pitch revealed the cavernous empty space that condemned game after game to near silence.

I've since heard countless explanations, including TV broadcasters' demands for late-night match schedules, high ticket prices and the exodus of big-name players to Europe. The reason that makes most sense on a visceral level was the one told to me when I first suggested to Brazilian friends that we go and watch a match. "It's too dangerous," they replied.

While I have been to plenty of matches without incident, recent news reports and statistics show this is not an baseless fear. In the latest horrific case, a 34-year-old Santos supporter was killed on Sunday as he waited for the bus home from the derby match with São Paulo. Two car loads of rival fans set on the victim with iron bars, kicking and beating him. Marcio Barreto de Toledo, who was wearing a shirt that identified him as a member of a Santos fan group, died in hospital from multiple head injuries. "It was a cowardly act to attack a 34-year-old father who was just trying to get home after a match," Cosme Damião Freitas, a director at the Santos fan group Torcida Jovem, said.

The lethal assault barely warranted a mention in the domestic media because such incidents have become all too commonplace. In each of the past three years, football-related killings have hit a new record. According to the Globo newspaper, 23 people died in 2012 and 30 in 2013.

All of the cases have been tragic, some macabre, such as the beheading of a referee who stabbed a player to death in an amateur match in Maranhão last June.

Elsewhere, supporters' groups, known as "organizados", operate like criminal mobs both in the way they fight for territory against rival gangs and in their use of threats and intimidation to influence their own clubs.

Games have been interrupted and players so frightened by physical assaults and intrusions on to a training pitch that they have threatened to strike. It is probably the frequency of the violence that is most alarming. Scenes of fighting are becoming staples on news programmes. Security guards are often deployed to escort players and the referees on and off the pitch. As I reported in December, riot police also had to be called in to break up fights during a match between Atlético Paranaense and Vasco da Gama. That game was interrupted for an hour as police quelled the unrest with rubber bullets, and a helicopter airlifted a badly injured fan from the pitch to hospital.

Such scenes help to explain why there is so little focus this year on the threat posed by English hooligans. That is unusual. Before almost every World Cup in my lifetime, notorious fans and firms have reared as large in the media buildup as damaged metatarsals, dodgy squad selections and shopping opportunities for Wags.

There will once again be travel bans for 2,500 convicted hooligans in the UK and undercover spotters are likely to co-ordinate with local police but these measures have generated less attention than in the past. This is perhaps because – compared to what is happening in Brazil – even the nastiest English pitbull fan seems relatively tame.

This is no cause for celebration for England. Rather, it is a clear sign that Brazil must do more to get its house in order or risk a far greater tragedy. Given the combination of rising violence and shoddy stadium construction, a Heysel-type calamity is far from unthinkable.

The World Cup ought to be the start of an improvement. The tournament is supposed to lift standards of play, public interest, stadium infrastructure and income for the sport. As in the UK from the late 1980s on, higher ticket prices are also meant to drive out the most violent fans.

Once the tournament is over, Brazil's authorities will still have their work cut out persuading the public that football matches are no longer disasters waiting to happen. If they cannot do that, the big new stadiums will be just as empty as the old ones.

• This article was corrected on 25 February. The original stated that 40 people died at Brazilian football matches last year, and had "seton" instead of "set on".