"In recent weeks, we've seen 10 million barrels a day of gasoline production. It continues to run very high. You're not talking a lot of breathing room. If there's a big refinery that goes down in the heat of the summer driving season, you can still expect a pretty big reaction," said Patrick DeHaan, senior energy analyst at GasBuddy.com.

Nineteen states are already above the national average, with nine averaging above $3 a gallon, and California heading toward $4 per gallon. Drivers in 25 cities are already paying 70 cents a gallon more than this time last year, according to GasBuddy.com.

Gasoline prices are expected to peak at a national average of around $3 per gallon by the Fourth of July holiday. But with summer driving season officially kicking off on Memorial Day about two weeks from now, the average price of unleaded gas is already at $2.85 a gallon, according to AAA.

Already expected at four-year highs, gasoline prices could be be especially vulnerable to spikes this summer, with demand at record highs and refineries running full throttle.

In 2014, when gasoline prices were last at $3 per gallon, the average family paid about $1,600 for the summer season. Drivers this year, however, should see the national average peak at about $3 to $3.10 per gallon, but not stay at or above that level as they did in 2014, Kloza said.

This summer, the average family could pay about $1,318 for gasoline from May through September, compared to $1,070 in the same period a year ago, according to Tom Kloza, head of global energy analysis at Oil Price Information Service.

With oil prices at a four-year high, gasoline prices are climbing too, and analysts are carefully watching developments with Iran in the Middle East and the continued decline of Venezuela's oil production.

But it's not just the potential hiccups at domestic refineries and pipelines, and even summer hurricanes that analysts say could threaten to send U.S. prices even higher. It's the fact that for the first time in a while, geopolitics has been driving oil prices higher.

Source: AAA

Gasoline demand, meanwhile, was at a high 9.8 million barrels last week, just shy of the record of 9.86 million barrels, and it is expected to rise during the summer. Analysts say low unemployment and tax breaks could be pushing up demand as more people drive to jobs and vacations.

"Certainly, we could see several weeks this summer where gasoline demand is in excess of 10 million barrels a day…We could see new weekly records set this summer," said Andrew Lipow, president of Lipow Oil Associates. Refineries are running at a high level, and utilization has been running over 90 percent.

Gasoline prices have been rising steadily, with the national average up about five cents per gallon in the last week and 20 cents in the past month. Last year, at this time the average was $2.33 per gallon, according to AAA.

The highest prices ever for unleaded gasoline were in the summer of 2008, with national prices for unleaded averaging $4.11 per gallon. Oil that year spiked above $145 per barrel, double current levels.

Fuel prices have been climbing this year, along with crude, which is up about 18 percent year-to-date. Oil prices have jumped recently as President Donald Trump moved to withdraw the U.S. from the Iranian nuclear deal. Trump dropped out of the agreement this week, and shortly after, there was the first direct military confrontation between Israel and Iran, with Iran aiming rockets at Israeli forces. Analysts said if that type of activity continues or intensifies, it could drive crude —and gasoline —prices even higher.

West Texas Intermediate crude futures Friday were at about $71, up more than 3.5 percent in the last two weeks. Gasoline futures were up about 6 percent in the same period.

"It's possible I would say we could have a spike if the tensions in the Middle East heat up to the point where it begins to impact crude oil supplies. If that's the case, the oil market would respond accordingly, and you could see $3.50 a gallon for gasoline," Lipow said. "I would say the probability is less than 25 percent, but that could easily change."

Venezuela's dwindling production, now at about 1.5 million barrels a day, is about 1 million barrels a day less than last year, and more declines are expected.

The future output of state-run Petroleos de Venezuela, called PDVSA, is even more unclear now that ConocoPhillips is seeking the company's Caribbean assets in return for a $2 billion arbitration award related to Venezuelan's seizure of Conoco assets in 2007.

A number of ships headed for PDVSA Caribbean operations recently have been diverted to avoid seizure of their cargoes. Other companies are expected to lay claim on PDVSA assets, including Canadian miner Rusoro Mining which is seeking Citgo assets.

"We're looking at a possible implosion, not only of their crude oil exports, which are pretty low but of their refineries. Maybe they're operating at 30 percent of capacity," said Kloza.

He notes that PDVSA was a chief supplier of fuel in Central and South America but its problems might mean the U.S. steps up to be "a larger supplier if things continue to get run down in their refining operations."

The U.S. exported 1.9 million barrels per day of crude last week, and another 4.8 million barrels of refined products, including 581,000 barrels a day of gasoline. The U.S. also imports some gasoline into the East Coast from Europe and Canada, but it has become a net exporter of refined products with a substantial amount of diesel exports.

"The U.S. is now exporting more crude oil than Venezuela," GasBuddy's DeHaan said. "That's a stark turnaround. Even a year ago, it just seemed impossible…That's a situation that's not going to be reversed."

Kloza said if oil prices do spike, Trump can release oil from the Strategic Petroleum Reserve or press Saudi Arabia to return some oil to the market.

U.S. oil drillers continue to increase their output, but Kloza said inadequate pipeline capacity is one factor that could limit the increase of U.S. crude. U.S. production is continuing to grow, hitting a record 10.7 million barrels a day last week and is expected to top 11 million by year end.

But Kloza said the U.S. also produces light sweet crude, while the Gulf Coast refineries process heavier crudes, like that from Venezuela.

"We have just too much light and sweet. That's why we're going to be exporting it to places like China and India, and in some cases here to refineries that can run more light crude. This is why Venezuela is such a big deal," he said.