ISTANBUL — Prospects for a period of instability in Turkey increased on Tuesday after attempts by the dominant party to form a new coalition government officially ended in failure.

The development helped create the basis for President Recep Tayyip Erdogan to call for a new election, which would mean the installation of a temporary government just as Turkey is facing new threats from Islamic State militants in neighboring Syria and a re-energized Kurdish insurgency at home. An Islamic State video released on Monday called for Turkish Muslims to revolt against the president.

Mr. Erdogan’s Justice and Development Party, which has shaped Turkish politics for more than a decade, was stunned in a June 7 election, losing a parliamentary majority partly because of gains by the pro-Kurdish People’s Democratic Party.

Lengthy negotiations by Mr. Erdogan’s prime minister, Ahmet Davutoglu, ended without an agreement for a coalition, first with the largest opposition party, the Republican People’s Party, then with a smaller, far-right nationalist party.