The chairman of Royal Bank of Scotland described on Monday how he felt like he was having an "out of body experience" when a banker protested to him that his £4m pay deal was unfair.

In a rare insight into the pressure top employees try to put on boards over their pay levels, Sir Philip Hampton said he had been contacted "quite a lot" by bankers wanting bigger pay deals via email and in face to face meetings.

Since becoming chairman of RBS after its £45bn bailout five years ago – it is now 81% owned by the taxpayer – Hampton has had to fend off repeated criticism of the bonuses paid to investment bankers and to former chief executive Stephen Hester. But the 60-year-old admitted to being surprised by the demands made by some staff.

"I can tell you I've had some completely out of body experiences in recent years where I was talking to somebody about potentially getting a £4m pay package. And outrage coming across the table from the other side because they know that somebody doing a comparable job at another bank is getting £6m. This is absolutely outrageous to them, that somebody is getting 50% more," he said.

Speaking at a debate in London organised by the High Pay Centre, the Freidrich Ebert Stiftung foundation and the Guardian, Hampton did not name the banker who had been disappointed by the £4m deal. On an intellectual basis he could understand the argument, he said, if not the feeling about the size of the payment.

It illustrated "part of the different world that can be inhabited", said Hampton, who attends meetings of the remuneration committee but does not have a formal seat on the board committee that sets pay. He receives £750,000 a year.

Hampton, who started his career as an investment banker at Lazard and became finance director of British Steel in the mid 1990s, said that in banking, expectations of bonuses had become "deeply embedded".

He said it was part of banking culture to work for your bonus, but considered that the power of bonuses to motivate staff was "overrated". However, he did not oppose bonuses as a way to pay staff because they were effective in aligning them with shareholders – especially as payouts could be clawed back if performance turned out not to be as good as it first seemed.

In 2012, he said, bonuses in the investment bank were 80% lower and the top 10,000 staff did not get pay rises.

The culture of bonuses being paid regardless of performance led to the EU capping them at a single multiple of salary – or twice with the approval of shareholders. The government is challenging the bonus cap, which regulators fear could push up fixed salaries. Barclays has been discussing the possibility of handing its bankers an allowance to make up for lost bonuses and Hampton would not disclose RBS's proposals for pay once the cap comes into force on 1 January.

"There will probably be some inflation of base pay," Hampton said. But, he warned, the impact could be to increase fixed costs for the bank. Some aspects of bonuses had encouraged unwanted behaviour, he said, citing incentives to sell payment protection insurance that have left the industry with a £20bn bill to compensate customers. "It is absolutely possible to have incentives that are disastrous. That was the case with PPI."