President Donald Trump late Wednesday demanded California return the $3.5 billion it received from the federal government for a "disaster" high-speed rail project.

"California has been forced to cancel the massive bullet train project after having spent and wasted many billions of dollars," Trump tweeted. "They owe the Federal Government three and a half billion dollars. We want that money back now. Whole project is a 'green' disaster!"

The $77 billion bullet train project, a planned system linking San Francisco to Los Angeles, has faced cost overruns and delays. California Gov. Gavin Newsom at his first state of the state address Tuesday announced a scale back of the high-speed rail project, saying the current plan "would cost too much and take too long."

However, Newsom responded to Trump's tweet Wednesday by calling it "fake news. We're building high-speed rail, connecting the Central Valley and beyond."

https://twitter.com/GavinNewsom/status/1095867826202505216

The Democratic governor also suggested in the tweet that Trump was "desperately searching" for some border wall money.

Newsom said Tuesday during his address at the State Capitol in Sacramento that he approved of going ahead with part of the high-speed rail link — a section between Merced and Bakersfield. He said the state can work toward connecting the Central Valley to other parts of California while also seeking more federal funding and private money.

A decade ago, California voters approved Proposition 1A, authorizing nearly $10 billion in bond money for the construction of the high-speed rail system expected to travel at speeds of up to 220 miles per hour. The state also was awarded about $3.5 billion in federal funds in 2010 to help with the project.

Some observers say the federal government would have a tough time forcing California to cough up the funds.

"As long as they've met all of the guidelines that were required, I'm certain they're not going to have to return the money," said Ray LaHood, who was Transportation secretary during the Obama administration when California was awarded the rail funds. "If the Trump administration wants to pursue it, it will probably have to go to court to do it."

LaHood also believes the federal funds were spent correctly by California, so the Trump administration may have a tough case if it pursues litigation. "The money they received from the federal government was used for the purpose it was intended," he said.

The former Transportation secretary said he still supported high-speed rail projects and was critical of Newsom's plan to scale back the train, calling it "very short-sighted. Obviously, he has no vision for rail — and I think he's made a very bad decision."

Yet the former Transportation official also said it wasn't unusual that massive projects such as the California bullet train experience setbacks.

"All big projects have problems," LaHood said. "That's the way it works. If anyone know this better than anybody it's the president, who has built a lot of projects."

Construction on the 119-mile Central Valley segment of the fast-speed rail line began in 2013. The 520-mile rail system from the San Francisco Bay Area to LA wasn't expected to begin operating until the 2030s.

Newsom acknowledged Tuesday that if the state abandoned the project, it could be forced to give back billions of dollars to the federal government.

"I am not interested in sending $3.5 billion in federal funding that was allocated to this project back to Donald Trump," Newsom said. "Nor am I interested in repeating the same old mistakes."

An audit last year faulted California's High-Speed Rail Authority, or CHSRA, for "flawed decision making and poor contract management." In addition, a business plan released in early 2018 showed its projected baseline cost soared by up to 20 percent from two years earlier and indicated the cost could rise to nearly $100 billion.

The governor Tuesday ordered "new transparency measures" on the rail project and said the state will "hold contractors and consultants accountable."

The U.S. Department of Transportation's inspector general initiated an audit last April of the Federal Railroad Administration's (FRA) oversight of the California rail project, including looking at how federal funds were spent on the Central Valley segment. The results of the federal audit are expected to be issued in spring 2019.

A DOT spokesperson told CNBC in a statement Thursday the agency remains in contact with California regarding the high-speed rail project as they learn more about the state's plans going forward.

"The [Transportation] Department's Inspector General is continuing to conduct an audit of FRA's funding for CHSRA," the spokesperson said. "As the IG noted, continued management attention and strong oversight will remain critical to ensure that federal funds are not subject to an unacceptable level of risk."

Newsom, who took office Jan. 7, has been outspoken against Trump for everything from immigration to climate change. He has also blasted the Republican president for threatening to withhold federal funds to the state for wildfire relief.