David Cameron has given his strongest hint yet that he may be preparing a giveaway for higher-income earners when he said he would "love" to raise the level at which people started paying 40% tax.

Speaking to workers at a water company on Wednesday, the prime minister said many people "who don't see themselves as fundamentally wealthy" were paying the rate, which applies when people earn more than £41,865 a year. Stressing that he could not make any promises, Cameron said he understood the problem with the rate applying when people "are not earning a lot of money".

Any rise in the threshold would help people at the higher end of the earnings spectrum. The last official annual figures showed the average salary for men in full time employment was £29,300, while it was £23,600 for women, according to the Office for National Statistics.

More than 2 million extra people will start paying the higher rate of tax over the course of the parliament, according to the Institute of Fiscal Studies – a phenomenon known as fiscal drag. That is because the rate does not rise, but salaries rise with wage inflation.

According to some estimates, the number of people in the higher-rate tax band has gone up to one in six of the population from the one in 20 who were affected when it was introduced by the former chancellor Nigel Lawson.

At the event, in Warrington, Cheshire, Cameron said: "I know that a lot of people believe that the 40p rate now kicks in quite early and quite a lot of people who don't see themselves as fundamentally very wealthy are paying that 40p tax rate. Now I would love to be able to stand here and say we are going to sort all this out, we will raise the thresholds of all these tax rates. I can't make that promise today."

Cameron said his priority was to lower taxes for the lowest earners, but added: "I understand the problem with the 40p rate kicking in when people are not earning a lot of money, and I have to look very carefully at the books before I can make any promises about it."

Cameron and George Osborne, the chancellor, have come under strong pressure from senior Conservatives to raise the threshold.The proposals were raised directly with the chancellor, George Osborne, at the backbench 1922 Committee in March, urging him to increase the trigger to £44,000 to stop workers such as teachers and police officers being hit. But in the budget, Osborne announced that the income threshold at which the 40p rate applies will increase by just 1% for each of the next two years, less than the rate of inflation.

Labour said it did not oppose the idea, instead criticising Cameron for "vague talk of jam tomorrow" instead of action. The party also accused Cameron of "flirting" with the idea of tax cuts for low and middle earners when he wants their vote, saying he has disappointed people before on this front.

"Under this government, people on middle and low incomes are paying more because of changes like the VAT rise, while millionaires have been given a tax cut," said Chris Leslie, a shadow treasury minister.