Nationalists from rival ethnic groups claimed victory in Bosnian elections on Monday, but it seemed likely that their power would be curtailed by an electorate frustrated at economic stagnation and corruption.

Milorad Dodik, a secessionist with strong ties to Moscow, was leading by two percentage points with about 80% of the vote counted in the race for the presidency of the Serb half of the country, the Republika Srpska (RS), which he has long vowed to lead to independence. He said his policy would be for the RS to function “less and less an entity and more a state”.

Even if Dodik manages to hold on that lead, he could find it harder to run a government in the main Serb town of Banja Luka. Although his party emerged as the biggest single block in the RS assembly, it now controls less than half the seats, meaning opposition parties may be able to form a ruling coalition if they were able to overcome their differences.

Dodik’s ally, Zeljka Cvijanovic, was facing defeat by a more moderate candidate, Mladen Ivanic, in the vote for the Serb seat on the Bosnian state presidency, in which Serbs, Croats and Muslims, known as Bosniaks, share power. Ivanic is seen as potentially a more cooperative partner in the tripartite presidency, which could help strengthen Bosnia’s weak central institutions.

Among Bosniak voters the nationalist SDA party, which led the country’s Muslims through the war, returned to dominance at the expense of the mixed Social Democrats who suffered an electorate meltdown. Bakir Izetbegovic, the SDA son of the country’s wartime president, won the Bosniak seat on the presidency, and said he hoped the voters would choose candidates capable of “overcoming the current deadlock that blocks the country’s integration into EU and Nato”.

However, Dragan Covic, another nationalist, was leading the race for the Croat presidency, strengthening pressures not only to undermine the Bosnian state but also the Croat-Bosniak federation that makes up half the country.

“It is initially disappointing how well the nationalists did, but over the next few days, I think we are going to see a more complex picture emerge,” said Jasmin Mujanovic, a Balkans analyst at York University.

Reformists pointed to the vote for the country’s various legislative assemblies as a sign that the population as a whole wanted to change, but it was far from clear whether reformist parties would be able to form a working coalition.

“Ruling majorities got less votes than the opposition on all levels. Bosnia seems to have voted for change, the question is whether the post-election coalition negotiations will respect the will of the people for change,” said Reuf Bajrovic, the head of the Emerging Democracies Institute.

Turnout was estimated at about 54% of the country’s 3.3 million electorate.

Nearly 19 years after the end of the Bosnian conflict, which killed 100,000 people, the early results suggest that the fundamental questions over which the war was fought, national identity, ethnic distinctions and government structure, are still far from resolved. The 1995 Dayton peace agreement stopped the bloodshed but entrenched the results of “ethnic cleansing”, cementing the divide between the two halves of the country.

It has also led to an elaborate multi-tiered system of government with cabinets and parliaments on state, entity and cantonal levels, overburdening Bosnia with politicians and civil servants, who have generally paid themselves salaries out of proportion with Bosnia’s impoverished condition.

The official unemployment rate is 44%, with youth unemployment even higher, and an average monthly salary of €415. Corruption in the six layers of government is estimated to cost taxpayers €750m each year. The already stalled economy was further hit in May by devastating floods believed to have caused €2bn damage, equivalent to 15% of Bosnia’s gross domestic product.

The frustration and anger at the country’s stagnation among ordinary Bosnians boiled over into street protests in February. There have recently been more ominous developments. A few hundred Bosnian jihadists are believed to be fighting in Iraq and Syria, raising concerns of the threat of violence they would represent on their return. The country was also unnerved by the arrival over the past fortnight of about 150 Russian cossacks in the main Serb town of Banja Luka, at the invitation of Dodik’s government. They were ostensibly there to commemorate wartime Serb-Russian alliances but it emerged that their leader, Nikolai Dyakonov, had also led an armed Cossack unit involved in the invasion and annexation of Crimea.

Sunday’s vote was the seventh set of elections since the war. It showed no signs of extricating Bosnia from a vicious cycle of poverty and discontent fuelling nationalist parties which deadlocked state institutions, which in turn further deepened the economic malaise.

Progress towards EU accession stalled in 2009, when the European Court of Human Rights demanded a change in the Bosnian constitution so that members of ethnic minorities could run for senior posts currently reserved for Bosniaks, Croats and Serbs under the system inherited from Dayton. The demand went unheeded.

“No one can celebrate in this country. Those who won the largest number of votes will be put to the test,” the Sarajevo-based Dnevni Avaz daily commented. “If those parties don’t take in the seriousness of the situation and the message sent by the population during February’s protests, what will come next will be much more violent.”