ROME—Italy entered a period of political instability on Monday after national elections boosted populists but failed to produce a winner with enough support to patch together a parliamentary majority.

Sunday’s two big winners—the 5 Star Movement and a center-right coalition including former Premier Silvio Berlusconi and the anti-immigrant League—each claimed Monday to have won enough support to earn the right to try to form a government. But with neither group having won an outright majority, Italy is likely to face weeks...