The men on the frontline of Iraqi Kurdistan's fight for existence have been there before; many wearing the same uniforms, carrying the same ageing weapons, and championing the same cause.

"I have fought three enemies in my lifetime: Saddam's Ba'athists, [former prime minister] Nouri al-Maliki, and now Islamic State," said Rashid Tarjani, a veteran member of the Kurdish peshmerga forces, standing under a shelter 30 miles south-west of Irbil and two miles from the village where his latest foe lurked behind homes abandoned several days earlier. "Of all of them, Maliki was the worst."

A portly middle-aged fighter stood next to him, four rust-tinged rocket-propelled grenades protruding like arrows from a quiver on his back. The weight of the munitions had reduced his swagger to a stoop. "I don't know about that," he said, challenging his fellow fighter's claim. "If the Americans hadn't helped us, Da'ash would be in our houses by now," using the colloquial term for the group also known as Isis.

Both men had been members of the peshmerga for more than 30 years, and each had stories of struggle and sacrifice that were true to the Kurdish force's legend. But here and across the fractured series of battle lines of the latest war, the well-worn narrative of vaunted fighters uniting to defeat all-comers was starting to splinter. In its place rose a picture of fragility; there were limits to Kurdish authority that had been badly exposed by an unprecedented foe.

Earlier this month, the Isis militants who had caused carnage in Arab Iraq had turned their guns on the Kurds, a group who had until then kept their distance from the central government's problems, while steadily building a state and a fortune. But rather than turn the insurgents back, as it had in the past, the peshmerga withdrew from areas inside Iraq that it had held since the Iraqi army fled in June.

The retreat of government forces had left tens of thousands exposed to the savagery of Isis, especially those from the country's minorities, including Christians and members of the Yazidi sect. Communities that had coexisted for several thousand years fled their homes and were seeking refuge in Iraqi Kurdistan. Up to 50,000 desperate Yazidis climbed a nearby range, Mount Sinjar, to escape a genocide.

"The same way the Nazis attacked Leningrad as the birthplace of Lenin, Isis attacked Sinjar as the birthplace of the Yazidis," said Mahmoud Omar Saleh, a member of the peshmerga affairs committee in the Iraqi Kurdistan parliament.

Such was the scramble to rescue the Yazidis that a Syrian Kurdish militia, the YPG – an affiliate of the Turkish PKK, which is proscribed as a terror group by Ankara and Washington – crossed into Iraq to secure an exit north. Female Kurdish fighters were among the group, which operated for many days under US air cover.

Peshmerga units took up positions around where they are now stationed on the baking hot plains not far from Irbil. Isis fighters had been creeping forward until the US air force slowed their momentum one day earlier.

"They're not trying to advance now," said Tarjani, as he stood among a mix of Dad's Army warriors and eager new recruits in American-issue combat kit." Word circled around the men that Isis had withdrawn from three of four villages they had seized, and tension quickly eased.

A visiting dignitary and his entourage pressed their way through the throng to try to find a leader. "He's a Jalali," one fighter whispered – a pointed reference to the man's links to one of two political strongmen, Jalal Talabani and Massoud Barzani, who have led the Kurdish cause since the 1970s.

In years gone by, Talabani's PUK faction had battled Barzani's KDP for power and influence; the feud is yet to be reconciled. And on a KDP-run checkpoint, even with a common enemy lingering, the visitor's history drew suspicion.

Less than an hour's drive from the capital of what would be Kurdistan, the fighting men of the north looked and acted more like a community militia than a united military force. The party line split that polarises much of Kurdish society was on clear display in the institution that now matters most.

"The peshmerga force is not a nationalist army," said retired commander Ali Ahmad Saalim. "It is made up of forces belonging to the political parties. The PUK and the KDP have not united their forces. These … units are under the command of their parties."

Saalim said a disparity between salary and conditions for officers and soldiers had weakened a resolve to fight.

"I believe the fact the peshmerga were defeated [in early August] was not so much about weapons, it was about morale and conviction. I was injured in 1981, 83, 84 and 87 because I believed in my struggle.

"We fought previous regimes out of conviction but these days a peshmerga receives around 500,000 dinars (£260) per month. How can you expect a peshmerga with a 500,000-dinar salary to fight Isis while officials get far more benefits?

"There is a difference between Isis and others like Saddam we fought in the past. Saddam was bloodthirsty and carried out the Halabja massacre and Anfal campaign. But even he would have boundaries. Isis massacres even the civilians. It is much more bloodthirsty than Saddam."

Many peshmerga fighters are forced to take second jobs to supplement their income. Amid the pristine roads, soaring new buildings and ostentatious new wealth of Irbil, few who fight for the home front say they can get by on just an official wage.

A peshmerga intelligence official who identified himself as Sarbast said: "I have been married for 13 years and I still live in rented accommodation. I pay 500,000 dinars for rent and if it was not for my taxi I would not have been able to live.

"Is it not embarrassing for the peshmerga ministry that as an officer I am driving a taxi to make a living? Or for other peshmerga to drive taxis or become guards outside companies and supermarkets?"

Another peshmerga fighter said: "We all have a duty now to defend our land, but once this is over, there need to be big changes."

Even as US and French weapons start to flow to the peshmerga, Kurdish officials concede urgent changes are needed – starting with genuine reconciliation between the feuding factions.

"A vacuum existed because there was no real military system for the different groups of fighters," said Saleh. "There is no unified command. But I think they have realised that and started to put aside their differences. Our priority must be to have a competent and coordinated military force in Kurdistan.

"If we don't put this theory into practice now, then we will have lost a hugely important historical time."

Commander Saalim also urged the Kurdish groups to unite, or run the real risk of being similarly subsumed by the chaos ravaging Iraq and Syria, which threatens to topple the borders of both states. "This is a historic moment for the Kurds, given the broad international support from the US and the EU countries," he said.

Reluctant for more than a decade to arm the Kurds, lest it bolstered their ambitions for sovereignty, the speed of the US response to the threat from Isis has convinced many in Irbil that the long quest for a Kurdish nation could still emerge from the rubble of the Middle East.

Split into four geographic locations, in Iraq's north, eastern Syria, south-eastern Turkey and western Iran, the Kurds' quest for statehood has remained elusive ever since the Ottoman empire was carved up almost a century ago. Efforts to unite the disparate groups have until now been lost in a myriad of competing ambitions and decades of political turmoil.

"The 100-year-long struggle of the Kurds in the four parts of Kurdistan, especially the 40-year-long freedom and democracy struggle of the PKK, has heightened the awareness of the Kurds," a spokesman for the PKK, which has fought a separatist campaign against the Turkish government, wrote in an email to the Guardian. "After the fall of Mosul, we [declared to] all the parties in south Kurdistan that our guerrilla forces are ready to defend our people in south Kurdistan. We have declared that all the Kurdish parties and groups should work to form a national defence force.

"The conditions in each part of Kurdistan are different, and the settlement models may differ from each other. But all parts should have common strategies."

"Iraq will never be the same again," said Saleh. "We need competent visionary leaders to make this work and we need them now."