George Osborne has claimed members of Gordon Brown's circle were "clearly involved" in pressurising banks to manipulate the London inter-bank offered rate (Libor) at the height of the financial crisis of 2008.

In an interview with the Spectator magazine to be published on Thursday, Osborne raised the political temperature by pointing the finger of blame on members of the former Labour government, mentioning Ed Balls by name as one of those who had "questions to answer".

His comments follow a note by Diamond that claimed "senior figures within Whitehall" had been involved in applying pressure to Barclays to keep the key inter-bank lending rate down during the financial crisis.

In a hearing with MPs on Wednesday at which the former Barclays chief executive made a staunch defence of his tenure at the bank, Diamond made it clear that he did not think Paul Tucker, the Bank of England's deputy governor, had given the bank free rein to cut its interest rates during the October 2008 banking crisis.

He mentioned two political figures, former City minister Lord Myners and Baroness Vadera, as he was questioned about remarks he attributed to Tucker that Whitehall figures had been concerned about the high bids Barclays had been submitting for Libor and its European equivalent Euribor.

But he was careful not to suggest that they were the figures that Tucker might have been referring to.

Osborne said in his interview that both the Financial Services Authority (FSA) and the US department of justice were "very clear" that the Bank of England did not issue instructions to Barclays to cut its Libor rate. "As for the role of the Labour government and the people around Gordon Brown, well, I think there are questions to be asked of them."

Singling out Balls, once a city minister but who served as children's secretary under Brown's premiership, he went on: "My opposite number, who was the City minister for part of this period and Gordon Brown's right-hand man for all of it … he has questions to answer as well. That's Ed Balls, by the way." Speaking of Brown's government, Osborne said: "They were clearly involved and we just haven't heard the full facts, I don't think, of who knew what when."

Former Labour ministers denied any involvement in the rate-rigging scandal.

Vadera, the economist who advised Gordon Brown and was made a minister, told the BBC Radio 4 show World at One that she had not spoken to Paul Tucker, the deputy governor of the Bank of England, "or anyone at the Bank of England" about the setting of the Libor rate.

"I didn't speak to him; I've been through my files. I don't remember anything that would suggest that anyone in government understood some of the allegations that are in this report by rogue bank traders on Libor."

She denied writing a paper about reducing the Libor rate, but insisted it was a "completely legitimate concern" for the government.

Vadera refused to be drawn on whether one of her colleagues might have spoken to the Bank of England about the rate.

"I can only speak for myself – I can't obviously speak for everybody in government. You asked me if Libor was a concern, and of course it was a concern. There's nothing wrong with concerning yourself with access to credit – it's the job."

Balls hit back at what he said were false allegations being levelled against him.

He told BBC News: "What I can say to you categorically – because there's been some false allegations made about this overnight for political reasons as always – at no point did I have any conversation with Mr Tucker at all at any time when I was Treasury minister, a Treasury adviser, or subsequently to that when I was a cabinet minister.

"And I had no conversation with anybody about the Libor market during any of those periods and at no point in any of the time when I was a minister or an adviser were concerns raised about the Libor market to me from the FSA, the Bank, or the Treasury."

He added: "The thing which I can't understand is why government ministers are so worried about having a full, independent, and judicially led inquiry into these matters."

David Cameron reiterated his preference for a parliamentary inquiry as the best option for a "swift and decisive" investigation into the banking scandal, as Ed Miliband pressed him to consider a two-part judge-led inquiry into the culture of the City.

Balls said a judge-led inquiry would help reveal who the "Whitehall figures" were.

"We don't know that that's what happened – it's in the Bob Diamond note. I think one of the reasons why we want an independent public inquiry, rather than simply a parliamentary inquiry, is to get to the heart of this."

Myners also insisted "categorically" that he did not speak to Tucker about Libor. He said he was not aware of anyone in Whitehall raising the issue with the Bank of England, and insisted the Treasury at the time was not interested in the Libor-setting process.

"I am not aware of any 'Whitehall figures' putting any pressure on the Bank of England or raising this issue with the Bank of England," he told the Today programme.

"I am not falling into the class of people who have no recollection; I can say quite categorically that I did not speak to Paul Tucker or anybody at the Bank of England about the Libor rate-setting process."

Alistair Darling, the former Labour chancellor, said on Tuesday he had made no calls to the Bank asking them to put pressure on anyone to lower Libor rates and said he would find it extremely surprising if the Bank of England had ordered Barclays to lower its rates.

"I would find it absolutely astonishing that the Bank would ever make such a suggestion, and equally I can think of no circumstances that anyone in departments for which I was responsible – the Treasury – would ever suggest wrongdoing like this."

Tucker was preparing to fight back against any accusation made by Barclays that he encouraged the bank to lower its bid to the Libor rate setting committee, which led to last week's record fine for the high street bank.

In a statement issued before Diamond appeared before the Treasury select committee, Tucker requested an audience with the committee to put his side of the story.