It was a cause for great optimism after years of scandal and corruption at Fifa, when in 2015 the US authorities initiated criminal prosecutions against some of the worst alleged offenders, and a new top leadership took office with sweeping promises of reform.

We, the chairman and two independent members of the new governance committee were part of that reform effort. We took seriously the task entrusted to us, of implementing reforms, enforcing rules on candidates’ eligibility for Fifa positions, supervising elections and the furthering of social responsibility and human rights in football. Today, none of us is any longer in office. Our sin? We appear to have taken our task too seriously.

We knew, of course, that changing an institutional culture would be a long process; two steps forward, one step back. We were dealing with a game, and its organisations, including national football associations, generally extremely resistant to independent scrutiny, lacking real democracy and integrity, dominated by a small group of people resistant to public accountability, in a context of huge economic stakes and endemic political interference.

There is a huge structural conflict of interest at the heart of Fifa: its leaders depend for their survival on those whom they ought to reform; power in Fifa is a political cartel. This is why the leadership of football survived for so long despite the many scandals surrounding it. With a little help from the FBI, and the independent Fifa ethics committee, whose two chairmen also had their tenures ended at the Bahrain congress in May, it might be that some bad apples have finally been removed. But the system remains largely unchanged. The leadership of football does not answer to the court of public opinion; it responds to its own constituency that would replace leadership which seriously tried to reform football.

All of us, united in our love for the game but with no ties to it, and with considerable experience in law and governance, accepted our appointment having received solemn promises regarding our independence.

Understanding that reform would not happen overnight, we showed flexibility when we thought it was proper. But we drew some red lines. We tried to effectively and impartially enforce Fifa’s stated principle that members of the Fifa council must be politically neutral. Our decision, in that context, to ban the Russian deputy prime minister Vitaly Mutko from elections led to a backlash from the new Fifa leadership against our independence. We also tried to enforce gender equality, human rights, and regulate the integrity of Fifa elections, and faced resistance on all of them. Ultimately it became clear to us that the leadership, in order to secure its own survival, could no longer support our independence.

We have concluded that Fifa cannot reform from within. Those responsible for leading such reform are politically dependent on the associations and officials they need to reform, and may remove members of the judicial and supervisory independent committees at a whim. Sport is one of the most important areas of social and economic life, representing around 2% of the world’s GDP, yet it is governed by transnational private organisations. They regulate access to sports markets, and define and enforce their rules. Yet this regulation is not open to public participation, nor to any effective form of public scrutiny and accountability.

What can be done? We advocate decisive external action. Parliamentary inquiries are good starting points, but it is necessary for them to produce concrete results. No country on its own – including Switzerland which hosts many of the world governing bodies – has the effective power to regulate such transnational organisations. The European Union is, though, in a privileged position: it brings together 28 member states – while the UK is still a member – which, collectively, these governing bodies cannot ignore. We believe the EU should take the lead.

The European Commission is entitled to act under competition or internal market powers, and the European Parliament could seize the issue by demanding effective action from the commission and the council. One initiative, which we strongly support, is now in a draft resolution of the Parliamentary Assembly of the Council of Europe, following a report by the former Luxembourg sports minister, Anne Brasseur. It calls on the EU to create an independent agency, which would not govern sports itself, but have the authority to review and supervise ethical issues and structures, to guarantee that transnational sports organisations conform with good governance. A second initiative would be to make the criminal investigation and prosecution of transnational sports-related criminal activity a priority for the forthcoming European Public Prosecutor.

Billions of people are passionate about sports, yet they have no effective scrutiny of the organisations that control them. Someone must act on the people’s behalf to protect the integrity of our sports. We cannot let the beautiful game remain in the hands of an ugly organisation.

• Miguel Poiares Maduro: the former chair of Fifa’s governance committee, is the director of the School of Transnational Governance in Florence.

• Navi Pillay is a former UN human rights high commissioner and judge at the International Criminal Court.

• Joseph Weiler is professor at New York University School of Law, and former president of the European University Institute.