Hurricane Irma destroyed a quarter of the homes in the Florida Keys and badly damaged many more, federal officials said Tuesday, as millions of people in the nation’s Southeast remained without power in the storm’s wake.

“Basically every house in the Keys was impacted in some way or another,” Federal Emergency Management Agency Administrator Brock Long said at a news conference. “This is why we ask people to leave.”

Meantime, Jacksonville, Florida’s largest city, was in recovery mode after the St. Johns River that flows through downtown experienced record levels Monday.

Mayor Lenny Curry said Irma hit the city with a storm surge like a Category 3 hurricane, even though its winds were at tropical-storm level when it reached northeast Florida. More than 350 people had to be rescued, he said. On Tuesday, he lifted an evacuation order but warned returning residents would have to navigate serious flooding and road blockages.

August and September mark peak hurricane season in the Atlantic basin. Here's why the conditions in these months make them more likely to form there. Photos: NASA/NOAA

The state of 20.6 million people has begun the difficult task of assessing damage and rebuilding from Irma, which first made landfall in the Keys Sunday morning as a Category 4 hurricane. The storm’s winds, heavy rain and surging seas have crippled the state after 6.5 million people were ordered to evacuate to other areas.

Many places including Miami Beach were just beginning to reopen, highways leading into the state from Georgia had bumper-to-bumper traffic, and long lines at gasoline stations remained a major frustration in cities like Fort Myers.

Residents and business owners in the Upper Keys were allowed back into the area Tuesday morning as far south as Islamorada. But Monroe County officials urged people to stay away. “Fuel, water, power & medical super limited,” the county said on Twitter.

The county has about 53,000 housing units, census figures show. Nearly all are on the Keys, a 110-mile ribbon of low-lying islands linked by bridges. Monroe County is home to 79,000 people, the vast majority of whom live on the archipelago.

Florida Gov. Rick Scott on Tuesday said transportation authorities were inspecting those bridges to make sure they can bear significant weight.

Local residents walked along a destroyed trailer park in Plantation Key in the Florida Keys on Tuesday. Photo: carlos barria/Reuters

For a third day, friends and relatives of Keys residents who rode out the hurricane on the islands scrambled for news on the well-being of holdouts, while evacuees searched for information on their homes and boats.

Alex Chennault, who lives in Los Angeles, was relieved Tuesday to finally get a 30-second voice mail from her mother in Key West, who was fine. Her mother’s ex-husband on Summerland Key “lost everything,” Ms. Chennault said, but was at least unharmed.

“Stuff is stuff,” Ms. Chennault said. “As long as he’s OK and well, there’s always more stuff available.”

Tammy Berard, the daughter of Key West Mayor Craig Cates, used Facebook to relay an update from her isolated father. The city’s sewer plant is down and needs a generator, she wrote. With numerous leaks in local water pipes, residents have had access to water only from 10 a.m. to noon and must boil it.

But she also reported that Naval Air Station Key West had received a food shipment, and a mobile hospital had been flown in for first responders.

There were some signs of hope. Florida Keys Electric Cooperative, which serves the Upper and Middle Keys, said Tuesday “the backbone of our power system withstood Irma’s impact well.”

While mobile home parks in the area are “pretty much destroyed,” Key West sustained far less structural damage than many other parts of the Keys closer to the mainland, Ms. Berard said.

Throughout Florida, power outages remained a major problem Tuesday. A state tally late Tuesday showed more than 4.7 million outages about 45% of the electrical accounts, were still offline.

Roughly three-quarters-of-a-million customers were still without power in Georgia and the Carolinas by late Tuesday, according to local utilities.

Power outages remained widespread in Jacksonville. In the city’s downtown Tuesday afternoon, downed tree limbs and debris littered some areas of a city still drying out from flooding a day earlier.

In the San Marco neighborhood, Worth Turner and colleagues were ripping out Sheetrock, discarding light fixtures and trying to mop, squeegee and vacuum up brown water that the St. Johns River poured into his year-old plumbing and lighting showroom.

Photos: Florida Begins Long Road to Recovery Hurricane Irma rolled over Florida, leaving floods, downed power lines and collapsed buildings in its wake. Jose Orosz walks his dog, Karen, on Wednesday by a home in Vilano Beach, Fla., that was destroyed by Hurricane Irma. Sean Rayford/Getty Images 1 of 18 • • • • • 1 of 18 Show Caption Jose Orosz walks his dog, Karen, on Wednesday by a home in Vilano Beach, Fla., that was destroyed by Hurricane Irma. Sean Rayford/Getty Images

“It’s devastating,” Mr. Turner said. “This was my pride and joy.”

When he heard Monday morning that the river was rising fast, he came to the tree-lined streets to find them rapidly filling with floodwater.

Swimming to his 8,000-square-foot showroom, he said he found 4 feet of water. He estimates he lost $500,000 in merchandise, which he doesn’t expect to be covered by flood insurance, as the building was.

The recovery could take longer in the Keys, where the pleasures of island life in the subtropics have long come mixed with danger. One of the worst hurricanes ever to hit the U.S. slammed into the Keys on Labor Day in 1935 at Category 5 strength, killing 408 people, according to the National Hurricane Center.

A monument honoring the dead stands at mile marker 81.5 in Islamorada, near the farthest point on U.S. 1 that residents and business owners were allowed to travel Tuesday.

Cheryl Meads, a member of the Islamorada village council who took refuge in Orlando from Irma, said she was relieved to learn her property survived.

The risk of hurricanes is “part of our lives,” the 58-year-old said, but “it’s worth it to us.”

Write to Valerie Bauerlein at valerie.bauerlein@wsj.com, Scott Calvert at scott.calvert@wsj.com and Jon Kamp at jon.kamp@wsj.com