The world's oldest operating bank is racing to avoid a government bailout.

Italy's Monte dei Paschi di Siena (BMDPF) is trying to salvage a rescue plan that would see investors inject €5 billion ($5.4 billion) into the bank. Political and economic uncertainty following the resignation of Prime Minister Matteo Renzi is keeping investors from signing on the dotted line.

Sources familiar with the talks between the bank and investors say a decision is likely by the end of this week. If the recapitalization fails, the Italian government is likely to have to step in with a bailout.

The bank declined to comment.

Renzi resigned after his constitutional reforms were rejected by voters in a referendum Sunday. He is staying on temporarily, probably until the weekend, to oversee the passage of Italy's budget through parliament. But what happens next is far from clear, and the uncertainty could hit the country's already weak economy.

Renzi had been trying to avoid using taxpayer money to bail out Monte dei Paschi because, under European Union rules, that would mean inflicting losses on thousands of small investors in the bank.

Monte dei Paschi, which was founded in 1472, is one of the weakest banks in Europe. It failed an important stress test this summer by Europe's top banking regulator.

The bank has roughly €20 billion ($21.4 billion) in bad debts that it has been trying to spin off into a separate entity as part of its turnaround plan.

Italian banks are saddled with about €360 billion ($386 billion) in non-performing loans, roughly a third of the eurozone total, and urgently need to raise new funds.

Shares in Monte dei Paschi were volatile Tuesday. They have fallen by about 85% since the start of the year, slashing the bank's market value to just €530 million ($570 million).

Related: Understanding 'Pizzagate' in the U.S.

How would a bailout work?

If the government has to rescue the bank, the bailout is likely to take the form of a precautionary recapitalization. This means the government and investors would cough up funds to keep the bank upright, resulting in heavy losses for shareholders and some bondholders. But deposits would be 100% safe.

The situation for mom and pop investors who hold the bank's bonds is more complicated -- and potentially politically explosive. Small savers reportedly own $250 billion worth of Italian bank bonds.

EU rules may offer some protection for these investors if Monte dei Paschi was found to have sold them bonds without properly explaining the risks.

Related: Big tech firms too slow to remove hate speech

Shares in other major European banks were volatile on Monday following the Italian referendum. Investors are concerned that further weakness in the Italian economy could ripple across the eurozone.

Other big European banks are watching nervously.

"The result of the constitutional referendum in Italy is a harbinger of renewed turbulence that could spill over from the political arena to the economy -- with Europe particularly endangered," Deutsche Bank (DB) CEO John Cryan said in a letter to employees.

Italy's banks have been struggling with high costs and low returns for years. Billions of euros in loans have soured due to economic stagnation. The country's economy has barely grown for a decade.

--CNN's Chris Liakos contributed to this report.