David Cameron has personally intervened in the row with Spain over "unacceptable" delays at the border with Gibraltar after days of increasing diplomatic tensions between the two countries.

Downing Street said Spain's prime minister had agreed to reduce delays at the border after Cameron rang his Spanish counterpart, Mariano Rajoy, to make a formal complaint. Cameron tweeted afterwards: "Constructive call with Spain's PM Rajoy. I made clear my concerns re Gibraltar and that our position on sovereignty won't change."

A statement issued by the Spanish government made no reference to any such concession on reducing delays and insisted procedures at the frontier were proportionate.

Despite Cameron's intervention, drivers trying to enter the British overseas territory faced the worst queues for several days with a four-hour wait in high temperatures.

Spanish border guards insisted on checking most vehicles and questioning drivers. They are said to be acting on direct instructions from the central government in Madrid and have been told to maintain the blockages for most of the day. Families with young children, many running low on water, were among those infuriated by the continuation of the tactics, despite Downing Street announcing a breakthrough.

"The Spanish guards are getting instructions direct from Madrid," said one Gibraltarian frontier official. "That's what they have told us. They are checking vehicles coming into Gibraltar which they don't need to do and even asking people if they have large sums of money they are going to spend in Gibraltar. They have told us they have instructions they will keep doing this until this evening. If they think this is going to encourage us to become Spanish, they are mistaken."

A No 10 spokeswoman said Cameron had also raised concerns with Rajoy about suggestions from the Spanish foreign minister, José García-Margallo, that Spain may introduce further measures including a €50 (£43) border fee.

"The PM made clear that our position on the sovereignty of Gibraltar and its surrounding waters will not change. He also reiterated, as the PM and Mr Rajoy had previously agreed, that the issue should not damage our bilateral relations. However, there was a real risk of this happening unless the situation at the border improved.

"Mr Rajoy agreed that he did not want the issue to become an obstacle in the bilateral relations and that we needed to find a way to de-escalate the issue. As a next step, the foreign secretary should speak to Mr García-Margallo to discuss a way forward.

"In the meantime, Prime Minister Rajoy committed to reducing measures at the border. Both leaders agreed that there should be a solution to the fishing dispute."

Cameron said: "I had a clear and constructive conversation with the Spanish prime minister. I think it is important that we recognise it is not acceptable, what has been happening with the people of Gibraltar, in terms of delays and other things that they have faced. I made that point clear.

"Of course, there is a fishing dispute between Gibraltar and Spain and that needs to be settled and we agreed that should be settled but it is not right to escalate things in the way that has been done and I made that very clear to the Spanish prime minister.

"We agreed that our foreign ministers would speak and try to resolve these issues but I am very clear that Britain will always stand up for the people of Gibraltar."

Speaking on a regional tour in Devon, Cameron said: "I am satisfied that we agreed to make some progress but I will only really be satisfied when this is properly de-escalated and the interests of people in Gibraltar are properly looked after."

But a statement issued in Madrid painted a different picture of the conversation, defending the border controls, claiming they were necessary to prevent smuggling.

"The head of the Spanish government reiterated his will to find as soon as is possible a solution to the current situation created by the Gibraltar authorities, which has produced a deep unease and a great concern because it hurts the environment and fishing activity," it said.

"In this respect, Prime Minister Rajoy argued to Prime Minister Cameron that the unilateral act of installing the blocks of cement in the bay of Algeciras was unacceptable."

It continued: "The head of the government explained to his British counterpart that the development of controls at the border are due to the fulfillment of the duty to control illegal trafficking.

"Prime Minister Rajoy reiterated that the measures were perfectly within the Schengen borders code, guided exclusively by the principles of randomness, proportionality and non-discrimination."

In Gibraltar on Wednesday Stephen Bangham, his wife Michelle and daughter Hannah, six, had been crawling towards the border since 10.20am when they finally crossed at 2.10pm.

"We've been coming here for 12 years and we've had border delays but nothing as bad as this," he said. "It is all a bit childish, but I think it is six of one and half a dozen of the other. It is like stoking a wasp's nest with a stick. You are going to get stung and that is what Gibraltar has done with Spain. I am visiting family, but if I was a tourist I would turn around and go home."

The prime minister returned to Britain from holiday on Monday and immediately dispatched Giles Paxman, the British ambassador to Madrid, to complain formally about the mounting threats to the territory, including the possibility of closing air space.

But Cameron decided to intervene after UK ambassador failed to receive assurances that the Spanish would not impose a €50 border crossing charge, a move that would seriously threaten the livelihoods of hundreds of Gibraltarians who travel to Spain each day for work. Cars travelling to Spain have been subject to extensive delays.

There had been hopes that the threats were limited to statements by the Spanish foreign minister, but it is understood work is now under way across the Spanish government about what punishments could be inflicted on Gibraltar. Some of the row has been stoked by the Spanish rightwing government's need to distract from corruption charges that have been levelled against its most senior figures.

Downing Street believes the Gibraltar dispute is not going to dissipate this month, and a direct complaint at prime ministerial level was appropriate.

The dispute, at root a centuries-old argument about sovereignty of the rock off Spain, has been sparked by the Gibraltar government dumping 70 concrete blocks in the bay, something that has angered Spanish fishermen.

Britain believes the potential move on border fees breaches bilateral agreements and may also fall outside EU rules. But proportionate fees may be possible since the UK is not part of the Schengen agreement.