The new Iraqi "border" is marked by a two-metre-high wall of earth. The berm, as it is known, cuts through farmland and orchards, separating the shrinking lands of the Iraqi state as it has existed for 95 years from the expanding territory of the new Islamic caliphate.

On the northern side, the black flags of Islamic State (Isis) shimmer in the afternoon haze. But on the Iraqi side it is not a national flag that flutters but a black Shia banner.

"This land is what separates good from evil," says a Shia fighter, pointing at the no man's land between the two forces. "Here you see the flag of Imam Hussein and there you see the black flags of Isis. This is the same history repeating itself," he says, referring to ancient Sunni-Shia enmities that played out on these plains centuries ago.

When the Iraqi army capitulated in the face of the Isis onslaught earlier this summer, it was left to Shia militias to fill the void and check the Islamist progress towards Baghdad. Like the Kurds in the north, the Shias are emerging as a far more effective fighting unit to confront the Islamists, whose murderous recent activities have elevated them to global public enemy number one.

But relying on the Shias brings problems of its own. On Friday, Shia militiamen were blamed for killing 70 people at a Sunni mosque in Diyala. It is attacks like these that have persuaded large numbers of ordinary Sunnis who live in the vast spaces between Baghdad and Damascus to side with Isis. In the Middle East, as the British foreign secretary, Philip Hammond, said last week, my enemy's enemy is not always my friend.

On the ground, Shia militiamen are eager to stress their passionate dedication to the fight, but deny that they are just as bad as Isis.

"Our name brings terror and they fear us. They think we are like Isis but we are not like them," the militiaman adds. "We don't kill families and we don't attack women or children or elderly people."

The route to the frontline leaves the visitor in no doubt: this is a war. Military debris lies scattered along the two sides of the highway. An occasional military truck or a Humvee speeds in the opposite direction, ferrying the injured and dead, passing the wreckage of an artillery piece, a blown-up turret from a Humvee and a great multitude of mangled metal objects.

Fertile fields famed for their melons, wheat and barley are now parched wastelands after irrigation canals were destroyed by shelling. Hamlets lie deserted or destroyed and the remaining mud houses have been taken over by military and militia units after their Sunni inhabitants fled further north.

In these almost medieval settings, modern intrusions can seem absurd. A road sign on the nearby highway, the main artery connecting Baghdad to Kirkuk in the north, declares, with misplaced confidence, distances that cannot be measured in kilometres any more, but instead by how many men will die trying to traverse them.

"See the electricity towers in front of us? That's the town of Udhaim. It's under Isis control," said Mujtaba, another young Shia militiaman, as he sped towards the frontline. He nodded with his floppy hat to the left and added: "They are also parallel to us now – only the Tigris river separates us."

Shia fighters waiting to ambush Sunni militants in the countryside of Diyala province. Photograph: Ghaith Abdul-Ahad

Mujtaba is a good example of the new breed of Shia fighter, hellbent on confronting what they see as an existential threat against them, battle-hardened by more than a decade of conflict in Iraq and Syria and, in some cases, trained in Iran and Lebanon under the unrelenting attention of Hezbollah.

Mujtaba is in his mid-20s, has fought in Syria, and has little faith in the Iraqi army's capability to fight this war. "You can't depend on the army – even if they put 2,000 soldiers in this village I won't take them seriously or count on them," he said. "We are a resistance faction that have been fighting for 11 years. Each one of us has been sent to at least three outside training camps in Iran and Lebanon under supervision of Hezbollah. Each lasted for two months. Do you know what it means to go for 60 days under constant gruelling by Hezbollah? You come back as a new person. You can't compare us with those soldiers who joined the army for money."

He was so impressed by his experiences that he named his first son after Imad Mughniya, the legendary Hezbollah military commander.

At the frontline, it is clear who the poor relations are. At one corner of the berm, a group of Iraqi army soldiers in boxer shorts and T-shirts caked with dust and sweat stood dazed under a scorching sun. Instead of foxholes or shelters, they had spread coloured mattresses and blankets on the berm, giving it the look of a giant laundry line.

The soldiers are dependent on the militias to hold the line and on civilian volunteers and villagers to feed and water them. The government has given up trying to supply them.

"They say the Iraqi soldier is a coward but where is the government?" said one middle-aged soldier. The troops had only a few hours' worth of ammunition and of the two ancient Russian armoured vehicles positioned nearby, only one could fire. The other had broken down and was there for decoration only.

"Where are the parliamentarians who are bickering back in Baghdad? Why don't they visit the front, give us a box of machine gun ammunition?" the disgruntled soldier asked.

Further along, in a small concrete room, a group of middle-aged militiamen with salt and pepper beards were huddled half-naked in the sweltering heat trying to get some sleep. Brand new machine guns, rocket launchers and Kalashnikovs were lined neatly against the wall on one side.

The men had just returned from fighting in Syria to take charge of a sector of the front in their home province of Diyala. There were no complaints about weaponry. Instead, there was just impatience for the battle. "Why are we not attacking them?" asked a one-eyed policeman who doubled as a militia fighter. "Our enemy in Syria was much stronger and there we were foreigners fighting in a strange land. Now we are home, I know every village and pathway."

In front of him sat the commander of the unit, a quiet former school clerk who said the berms were bad for advancing the cause.

"Before, targeting them was easier. Now we have walls between the two communities and they have settled behind them."

Shia fighters waiting to ambush Sunni militants. Photograph: Ghaith Abdul-Ahad

For these men, the Sunnis as a whole are the enemy, regardless of whether they are Isis supporters or not. For them, western strategies of trying to defeat Isis by depriving it of mass Sunni support are nonsense.

"When I withdraw my forces now the Sunnis will come back and they will become an incubator for Isis again," said one fighter. "When I liberate an area from Isis why do I have to give it back to them? Either I erase it or settle Shia in it."

"If it's for me I will start cleansing Baghdad from today," added another fighter. "We have not started sectarian war, we are just trying to secure our areas, but if the sectarian days come back then I am sure it will be won by us."

The war resumes every night. Soldiers and militiamen open fire at will, shooting into the darkness until the early hours.

"If they don't see us firing they will presume we have abandoned the positions and they start moving against us," said a young soldier. "We fire at everything, anything beyond the ridge, even if it's a dog."

There are signs that the Shia-inspired fightback is having results. Here in Diyala they have managed to push 50 kilometres into Sunni territory, taking over a series of villages and solidifying their lines. Corpses of dead Isis fighters have been taken back by the commanders and displayed like trophies in the provincial capital.

A Sunni village near the frontline was deserted, doors and windows smashed and many houses burned, the walls scribbled with pro-army slogans. A lone mortar shell fell in a small garden and started a fire. Palm trees burned slowly, their fronds crackling and moaning while a heavy stench of dead bodies wrapped the village.

"This is a village of rich farmers," said Mujtaba. "They brought destruction on themselves just because they hated the Shia and supported Isis."

He looked at the small mosque, which stood intact. "Its a shame the mosque is still standing – we should have burned it."

In the provincial capital, Baquba, two corpses have been hung from a lamppost, one upside-down. The Shia militiaman said they were Isis fighters brought from the front. But for the Sunnis of Diyala, the corpses were locals kidnapped by the militias and killed in retaliation for militiamen killed at the front.

"If they lose men at the front, they come raiding our villages and snatching men in retaliation," said a terrified Sunni farmer who lived nearby. "Nine men have been kidnapped in the last month. We found the bodies of three. The rest are still missing."

Across the street from the corpses, men and women waited silently for a bus with their plastic shopping bags and children in hand, keeping their gaze away from the dead bodies.

By noon another group of militiamen arrived at the frontline. They wore identical black T-shirts and brand new combat trousers. They posed with the soldiers, filming themselves as they fired a volley of precious bullets.

Mujtaba walked away in disgust and said it was time to leave. "We have a tense relationship with them."

Back in his pick-up truck, he said the young men belonged to one of the new battalions formed recently which were competing with his militia over funding and ammunition. "We suffer from the problem of the new factions that are appearing now every week," he said. "They are disastrous. Every 20 guys are forming a battalion or a brigade. They receive support from the mobilisation office and from the state and they haven't delivered anything in return.

A Shia banner planted at the top of the berm. Photograph: Ghaith Abdul-Ahad

"We used to get a lot of Iranian support but now Diyala is under the control of Hadi al-Amiri [commander of the Badr brigade and a longterm ally of Iran]. All the support of the Iraqi state and the [Iranian] republic is channelled through him. Iran says you get your share from Hadi al-Amiri."

In Baghdad a senior Shia politician, whose own party has started arming and equipping a militia force of its own, said that he feared the Shia were becoming as radical as the enemy they were fighting. "We are in the process of creating Shia al-Qaida radical groups equal in their radicalisation to the Sunni Qaida.

"By arming the community and creating all these regiments of militias, I am scared that my sect and community will burn. Our Shia project was building a modern, just state but now it's all been taken by the radicals. Think of 20 years ahead – these are all schools graduating militias, creating a mutant that is killing people, that is amassing weapons. Where will they go when the fight is over here? They will take their wars and go to Saudi and Yemen. Just like the Sunni jihadis migrated, so will the Shia militias."