Sen. Rand Paul was blocking the Senate's move to quickly pass its massive budget deal Thursday with fewer than five hours until government funding lapses.

The Kentucky Republican held out as he sought a vote on an amendment to maintain budget caps, arguing that the proposal spends recklessly. For the Senate to hold a vote on the spending package Thursday, all senators must agree.

As the impasse dragged on past 7:30 p.m., ET, it boosted the chances of Congress failing to pass a spending bill before midnight. Even if the Senate approves the legislation Thursday night, a skeptical House still needs to pass it and get it to President Donald Trump's desk.

Paul, a fiscal conservative, opposes the boost to military and domestic spending proposed by bipartisan Senate leaders. The bill before the Senate would set up a roughly $300 billion increase in the budget caps over two years.

He also argued that the potential for larger budget deficits has rattled stock and bond markets.

Shortly before 6, Senate Majority Leader Mitch McConnell took to the Senate floor and urged Paul to stop his objections and instead raise a budget point of order. The senior Kentucky Republican said, "I would argue that it's time to vote."

Senate Minority Leader Chuck Schumer then said: "We're in risky territory here." He argued that the Senate did not have the time to vote on Paul's amendment because other lawmakers would bring up amendments, as well.

McConnell requested to start a vote series on the plan, arguing that Paul could make his point about budget caps with a budget point of order. Paul objected, continuing the stalemate.

"I can't in all good honesty, in all good faith, just look the other way because my party is now complicit in the deficits," the senator said on the Senate floor.

"What you're seeing is recklessness trying to be passed off as bipartisanship," he later added.

Spotted leaving the Capitol shortly before 7:30 p.m, ET, McConnell said "it's up to Rand" whether the Senate worked all night, according to NBC News. The chamber will vote at 1 a.m. or earlier, "whenever he decides for us to move ahead," McConnell added.

Paul signaled that he could keep the Senate debating until early Friday morning to prove his point. That would take the chamber past the shutdown deadline. The Senate can vote on the bill at 1 a.m. even if Paul continues to object.

On the Senate floor, Paul criticized the U.S. military's broad presence around the world and highlighted projects abroad and at home that he deemed wasteful. He also objected to other motions to start up the vote series.

As Paul railed against lifting spending caps, the nonpartisan Congressional Budget Office estimated that the bill would cost about $320 billion. Most of that would come in the first year.

Paul argued that he was elected to fight what he deemed reckless spending and would not stop doing so because of the funding deadline.

He also said he thinks financial markets are "jittery" and showing an "undercurrent of unease," because investors are worried about long-term government debt and inflation. News of Paul's resistance came as the market was consumed by another dramatic selloff Thursday. The Dow Jones industrial average closed down more than 1,000 points for the second time this week.

When the Senate gets to a vote, the measure appears to have enough support to pass.

If the Senate approves the proposal, the House would then have to pass it before midnight Thursday and send it to Trump for his signature.

In the House, both fiscal conservatives and liberals who sought a deal to protect young immigrants from deportation threatened the plan's passage. On Thursday morning, House Speaker Paul Ryan said he believed his chamber had enough support to approve it.

GOP Sens. Jeff Flake of Arizona, Bob Corker of Tennessee and Steve Daines of Montana said they would oppose the plan over spending concerns.

Flake said in a statement that "fiscal responsibility is more than a talking point to trot out when the other guys are in charge."

In his statement announcing opposition, Corker said, "to say I am discouraged by the outcome of these negotiations would be an understatement."

The vast majority of the Republican lawmakers who are opposing the budget agreement, including Paul, voted for the GOP tax law. The massive tax cuts are estimated to add more than $1 trillion to budget deficits over 10 years, even after economic growth is taken into account, according to the CBO.

— CNBC's Ylan Mui and Mike Calia contributed to this report.