ERBIL -- Thousands of civilians streamed back into the northern Iraqi city of Kirkuk on Tuesday, a day after fleeing as Iraqi troops pushed Kurdish forces out of the disputed oil-rich city. The Kurdish peshmerga, meanwhile, lost even more ground to the Iraqi state forces.

With relative calm restored, the civilians headed back along a main highway to the city's east. The Kurdish peshmerga forces had built an earthen berm along the highway, reinforced by armored vehicles, but were allowing civilians to return to the city.

Many returnees were seen with their children and belongings packed tight in their cars.

A member of the Iraqi security forces holds a Kurdish flag in Kirkuk, Iraq, Oct. 16, 2017. Reuters

As Kirkuk's Arab and Turkmen residents on Monday evening celebrated the change of power, thousands of Kurdish residents, fearful of federal and Shiite militia rule, packed the roads north to Erbil, the capital of the northern autonomous Kurdish region.

Get Breaking News Delivered to Your Inbox

The Iraqi forces' retaking of Kirkuk came only two weeks after they had fought together with the Kurdish fighters to neutralize the Islamic State of Iraq and Syria (ISIS), their common enemy.

The move by the Iraqi military and its allied militias so soon after neutralizing ISIS in northern Iraq hinted at a country that could once again turn on itself after disposing of a common enemy.

President Trump expressed disappointment that the two sides were fighting, but insisted that the U.S. wasn't taking sides in the dispute.

"We don't like the fact that they're clashing. We're not taking sides," Mr. Trump told reporters at the White House Monday. "We've had for many years a very good relationship with the Kurds as you know and we've also been on the side of Iraq, even though we should have never there in the first place. We should never have been there. But we're not taking sides in that battle."

When the peshmerga battled and beat ISIS back in 2014, the territory they took over included Kirkuk. By that time, explains CBS News correspondent Charlie D'Agata, the Iraqi army had infamously fled. The "pesh" had been there ever since, much to the dismay of Baghdad. But when the Kurds voted for their independence in a non-binding referendum last month, the Iraqi government repeated its demand that Kirkuk be handed back.

After losing their grip on Kirkuk, the Kurdish peshmerga fighters lost even more territory in Iraq on Tuesday.

In the town of Sinjar, commander of the local Yazidi militia, Masloum Shingali, said Kurdish forces left before dawn on Tuesday, allowing Shiite-led militiamen who are fighting with Iraqi forces to move into the town.

Shingali says there was no fighting and that the Kurdish forces "left immediately, they didn't want to fight."

Town Mayor Mahma Khalil said the Popular Mobilization Forces, a predominantly Shiite militia coalition, was securing Sinjar.

Iraqi state and aligned forces retaking Kirkuk and surrounding areas completely changes Iraq's delicate balance of ethnic power.

The Kurds have long enjoyed semi-autonomy across a vast swath of northern Iraq, but in the matter of just a few days, their sphere of influence has shrunken dramatically.