CARPENTRAS, France—Europe’s populist politicians hoped this would be the year they rocked the Continent’s establishment. Instead, their assault on the European Union has brought election defeats, recriminations and self-doubt.

Hervé de Lépinau, a parliamentary candidate for France’s far-right National Front, ended a rally before parliamentary elections in June with an attack on his own party’s leadership. “Making a French exit from the euro the essence of our platform was an absurdity,” he said. His party subsequently lost...