BRUSSELS—Sunday’s referendum vote in Italy reinforced a widening split between the economics needed to sustain Europe’s common currency and the continent’s rising tide of populism.

Italians resoundingly rejected constitutional changes aimed at streamlining lawmaking and boosting competitiveness, marking a sobering start to what could be a defining year ahead for the European Union.

National elections are set for 2017 in three of the bloc’s founding members. Sunday’s vote makes it more likely that Italy, too, will have parliamentary polls next year.

In all of these countries, mainstream parties have been losing ground to populist movements, many of them on the far right. To be sure, in Austria, however, voters on Sunday turned back the presidential bid of a right-wing populist Norbert Hofer, who campaigned on curbing immigration and reinstating national borders. Green party leader Alexander Van der Bellen prevailed with more than 53% of the vote for the largely ceremonial position.

The biggest winner from Sunday’s ballot in Italy was Beppe Grillo, a comedian-turned-politician, and his populist 5 Star Movement, which wants a nonbinding referendum on Italy’s membership in the euro, an end to EU-mandated government-spending limits and income guarantees for all citizens.