Officials in San Francisco had planted their dreams in the empty concrete box beneath the Transbay Transit Center, calling it an end point for Caltrain commuters from the Peninsula and bullet trains zipping in from Southern California.

Now, that may not happen for decades. It might not happen at all.

In his first State of the State address Tuesday, Gov. Gavin Newsom said he would shrink down the big legacy project of his predecessor Jerry Brown, focusing on the Central Valley segment of the planned San Francisco to Los Angeles high-speed rail. For the foreseeable future, it would run from Bakersfield to Merced, stopping more than 100 miles from its long-touted destination in the South of Market neighborhood.

Boosters say they will find a way. But the potential for bullet trains would bring dollars to help extend the track underground from Pennsylvania Avenue near the 22nd Street Caltrain Station to the freshly built center on Mission Street. So far, the city has only identified $1 billion in local sales-tax and bridge-toll revenue for what probably would be a $6 billion project; the rest would come from a mix of federal and state sources, including funds earmarked for high-speed rail.

Without that track, no trains would roll into the structure’s cavernous basement.

Supervisor Aaron Peskin, who chairs the county Transportation Authority board, was unfazed. At this point, he said, San Francisco and the South Bay should go it alone.

“Our responsibility has always been connecting Santa Clara County to downtown San Francisco by bringing Caltrain to the Transbay Terminal, and we will continue to do that with zeal,” he said. “Otherwise, we will have built the most expensive bus terminal in the history of humankind.”

The shorter but more realistic link from Bakersfield to Merced would be a mere shadow of what Newsom described at the 2010 groundbreaking ceremony for the transit center, when he was mayor of San Francisco.

“We’re going to be building ... something that is arguably a generation overdue,” he said. “My gosh, I am sick and tired of hearing about how wonderful the transportation system is in France and Japan and ‘Have you been to Shanghai?’ Or, ‘Do you know what they’re doing in Beijing?’

“Well,” the mayor continued, “finally California is going to get it right with high-speed rail, and that northern terminus will happen here.”

As governor nearly a decade later, Newsom traded that expansive vision for something practical. And it might end far away from the transit center in SoMa, which is already mired in problems. When it opened, the $2.2 billion structure was hailed as a “Grand Central Station of the West” — just six weeks passed before workers found cracks in two of its steel girders, forcing the center to shut down.

The mishap-plagued building still glistens on Fremont and Mission streets, its metallic shroud draped over steel trusses, its rooftop garden silent and empty.

In a statement Tuesday, Mark Zabaneh, executive director of the Transbay Joint Powers Authority — the independent government agency in charge of the transit center and rail extension — signaled that, like Newsom, he’s concentrating on what’s possible in the short term.

“We share the governor’s goal of connecting California and investing in efficient, smart transportation that supports economic and workforce development,” Zabaneh said.

“In the meantime,” he added, “when reopened, the transit center will immediately serve tens of thousands of commuters every day. Additionally, it is providing open space, shopping and dining, public art, and is an economic driver for the community and the Bay Area.”

To some people in the transportation world, Newsom’s comments offered a sobering glimpse of reality.

During his campaign, Newsom had touted a “Valley to Valley” reduction of Brown’s plan, stretching from the farms in Bakersfield to the tech campuses of San Jose. But even that distance was going to be hard, said Randy Rentschler, legislative director of the Metropolitan Transportation Commission, which plans road and transit infrastructure throughout the nine-county Bay Area.

“Look, we all knew San Jose was difficult,” Rentschler said, “and the Transbay Terminal — that was a long way off, even on a good day.”

So Newsom “is just telling us in plain-speak what those of us in the business kind of already knew,” Rentschler added. “We’re just surprised that he said it.”

Others insisted the full bullet-train project is still alive.

“High-speed rail is going to happen, and it’s going to connect San Francisco to Los Angeles,” said state Sen. Scott Wiener, D-San Francisco. “Nothing the governor said today is going to change that.”

To Wiener and others, the Central Valley snippet seems like a prudent beginning. It’s already under construction and easy to isolate from the eternal fundraising debates about a railroad-to-who-knows-where.

If people like the rail line once it opens, they could help rally enthusiasm for the rest of the project, said Carl Guardino, president and CEO of the Silicon Valley Leadership Group, a business and public policy association that supports high-speed rail.

“There’s the theory of something, and then there’s the reality of using it,” he said, noting that reality generates much more excitement.

Officials at Caltrain were also optimistic. California’s High-Speed Rail Authority is required to invest $713 million into an ongoing effort to modernize the Peninsula rail system, swapping its old diesel locomotives for a new electric fleet. That commitment still stands, and Caltrain still aims to run its tracks from Pennsylvania Avenue to the transit center, said agency spokesman Dan Lieberman.

That plan predates voters’ approval of high-speed rail in 2008. Lieberman and other local officials are determined to keep it chugging along.

Rachel Swan is a San Francisco Chronicle staff writer. Email: rswan@sfchronicle.com Twitter: @rachelswan