The Fifa president, Sepp Blatter, has admitted world football cannot "turn a blind eye" to the deaths of hundreds of construction workers in Qatar as the country prepares to host the 2022 World Cup.

Under mounting pressure to act over the Guardian's revelations that dozens of Nepalese workers had died this summer in conditions described as "modern-day slavery", Blatter said he would meet the new emir of Qatar to discuss the issue.

But the Fifa president's claim that it had no direct influence over the situation and that there was plenty of time to resolve the issue angered those campaigning for change on the ground.

The International Trade Union Confederation (ITUC) said: "Fifa's offer of only a 'courtesy visit' to the emir of Qatar is totally inadequate and fails to put in place any plan to stop more workers dying."

The ITUC has warned that 4,000 construction workers could die before a ball is kicked if conditions do not improve for the labourers working on infrastructure projects worth an estimated £137bn before the 2022 tournament.

In a letter to Fifa released to the media, the 2022 organising committee chief executive, Hassan al-Thawadi, said it considered the "findings presented by the Guardian newspaper to be of the utmost seriousness".

It said: "Our prime minister has personally stated to us his firm and resolute commitment towards ensuring that genuine progress is made in the sphere of workers' welfare.

"The health, safety, wellbeing and dignity of every worker that contributes to staging the 2022 World Cup is of the utmost importance to our committee – and the state of Qatar – and we are steadfastly committed to ensuring that the event acts as a true catalyst towards creating sustainable improvement for every worker in our country."

Blatter refused to threaten the Qataris with the loss of the tournament if the death toll was not reduced in the runup to the World Cup. Instead, he reassured organisers that the tournament would take place come what may.

"You are looking to the future and I can't look to the future. I am going to Qatar now and we will put the situation of Qatar with the situation we are facing with their responsibility," he said. "We have plenty of time concerning Qatar but it is 2022, it is in nine years."

Nicholas McGeehan, Gulf researcher at Human Rights Watch, which has reported on labour abuses in Doha, said Fifa appeared to be saying the rights of workers building Qatar's 2022 World Cup were somebody else's problem.

"Blatter's assertion that there is 'plenty of time' to resolve the issues suggest he is either ignorant of or indifferent to the appalling abuses happening right now in Qatar," he said.

But a senior international diplomatic source working on the crisis said Fifa's decision to place the onus on Qatar's leadership was correct, as in practical terms only Doha could solve the problem.

They said Fifa's decision not to threaten Qatar helped provide "a safe space" in which it could respond to the widely exposed problems and engage with the UN's International Labour Organisation and other groups over the next six months.Over the last week international scrutiny of the problem through diplomatic channels has swelled, pressurising the Qatari leadership into action, the source said.

On Wednesday Qatar's ministry of labour said it had hired the international law firm DLA Piper to undertake an independent review of allegations it had failed to enforce international conventions on forced labour.

Blatter said Fifa could not control safety on building sites but admitted that growing international pressure had forced world football's governing body to acknowledge the problem. "What has happened now, we are not indifferent to that. We can't turn a blind eye and say this does not concern us," said the 77-year-old.

"That is why a trip to Qatar is planned. You can plainly see what Qatar has already done. We have 209 associations, in 209 associations you have different social and cultural associations. we can only do something when we see, when we hear and when we know ourselves – it has been confirmed by the Qataris themselves that something is amiss." Blatter, who also announced the creation of a working party to consult on the controversial decision to move the 2022 tournament from summer to winter, said European construction companies must also take their share of responsibility.

"The workers' rights will be the responsibility for Qatar and the companies – many of them European companies – who work there. It is not Fifa's primary responsibility but we cannot turn a blind eye. Yet it is not a direct intervention from Fifa that can change things," he said.

"The executive committee requested the president of Fifa to go to Qatar. I will meet with the new emir for a courtesy visit to confirm the World Cup 2022. We will also touch on this concern, the working conditions, but we are not the ones that can actually change it."

Ramesh Badel, a lawyer in Kathmandu who represents Nepalese workers in Qatar, including those who have lost hands and legs in construction accidents, said Blatter was dodging Fifa's duties. At least 70 Nepalese workers have died on Qatar's building sites since the start of 2012.

"Fifa should take responsibility to restore the rights of the workers themselves," he said. "All this construction is happening because of their World Cup. If there is slavery, how can they just keep quiet?"

Mahendra Pandey, chairman of the Parvasi Nepali Co-ordination Committee in Kathmandu, which this week received reports of 13 more injured workers in Qatar, said Blatter's failure to set demands for basic labour standards it expected from World Cup hosts showed Fifa was not treating the problem seriously.

"Blatter would be better off visiting a migrant labourer camp when he goes to Qatar than going to see the emir," he said.