The Senate has passed new legislation that challenges the ingrained secrecy of the US government and exposes federal agencies to greater public scrutiny, after the Democratic senator Jay Rockefeller dropped his opposition at the last minute.

In a day of high drama in the usually more sedate world of freedom of information campaigning, Rockefeller agreed to withdraw the block that he had effectively placed on the new legislation just hours before a final deadline. Open government advocates had warned that if he sustained his objections to the bill beyond Monday night there would be no hope of it passing in this Congress.

Shortly after 5pm, the co-sponsor of the Foia Improvement Act of 2014, Senator Patrick Leahy, announced on Twitter that it had passed the Senate. The bill now moves back to the House of Representatives which is expected to speed it through before the end of session on Thursday.

The bill, which has been two years in the making, is backed by more than 70 good governance organisations and is seen as a critical step towards a more open and accountable flow of public information. Amy Bennett of OpenTheGovernment.org, a coalition of groups that have led the push for the changes, said that the Senate vote amounted to a “huge moment for making sure that the US is more open and accountable. It’s really critical for ensuring that we will never see a return to the kind of secrecy we saw under the Bush administration.”

Ryan Shapiro, an open government expert at the Massachusetts Institute of Technology (MIT), said he was “thrilled” that Rockefeller had finally “decided to do the right thing and let the Foia reform bill pass the Senate. The records of government are the property of the people and we should have access to them.”

As the name of the legislation suggests, the bill seeks to improve the way that individuals, journalists and academics can request information under the Freedom of Information Act, which was first introduced in 1966. It would remove several of the most egregious impediments to government accountability that exist under the current law.

For a start, the bill would codify and make permanent the presumption of disclosure that Barack Obama introduced in his first term – that is, the idea that federal agencies must divulge information unless they are specifically exempted from doing so. The bill would enshrine that principle into law so that it could not be overturned by any future president.

The bill also tackles an exemption in the current Foia law, known as b(5), that is so broadly cast that it allows federal bodies to wriggle out of disclosing virtually anything. The invocation of b(5) exemptions has mushroomed under the Obama administration, despite the president’s promise to be a champion of open government.

The CIA, for instance, is fiercely resisting the disclosure of its official internal history of the Bay of Pigs Invasion in 1961 on the grounds that the report was merely a “draft” and therefore exempt under b(5). The CIA claims that disclosure of the report “could confuse the public”, even though it was written decades ago – an excuse that could no longer be used were the new legislation passed.

A third major reform would be to prevent the government from discouraging use of Foia requests by imposing prohibitive fees. Shapiro gave the example of one Foia bid he is currently pursuing in which he has asked the FBI to make public all 30,040 pages of documents that it has amassed from its inquiries into Daniel Ellsberg, the source of the Pentagon Papers leak on the Vietnam war.

The FBI has agreed to supply the MIT researcher with its entire file of papers on Ellsberg, but at a price. The bureau says it will burn the documents digitally on to CDs, imposing an arbitrary maximum of 500 pages per CD. At $15 per CD, the FBI has told Shapiro it will charge him a total of $890.

The new legislation, which is now expected to pass into law by the end of the year, would prevent government agencies from erecting “toll gates” in which they dissuade people from exercising their rights by hefty fees. “This bill doesn’t go anywhere near as far as I’d like, but it would prohibit several of the dirty tricks that federal agencies use to side-step their duties under freedom of information,” Shapiro said.

For much of the past six months, the Foia Improvement Act had enjoyed the smooth passage through Congress that is rarely seen in these gridlocked days. It swooped through the House and the Senate judiciary committee with unanimous bipartisan backing.

But then Rockefeller unexpectedly put his spoke in the wheel. His eleventh-hour opposition opposition to the bill puzzled many observers, given his previous general support for open government. The senator from West Virginia gave little explanation for his actions other than a statement released in the dead zone of late Friday night.

In that statement, Rockefeller said that he was worried that elements of the bill would have the “unintended consequence of harming our ability to enforce the many important federal laws that protect American consumers from financial fraud and other abuses … These provisions would make it harder for federal agency attorneys to prepare their cases, and they would potentially give defendants new ways to obstruct and delay investigations into their conduct.”