Erbil (CNN) When Iraqi government forces seized control of the contested city of Kirkuk on Monday, hundreds of Kurdish families were sent scattering to nearby safe havens.

The swift military operation came just weeks after Iraqi Kurds voted overwhelmingly for independence in a controversial referendum that was condemned by the United States and Baghdad.

The loss of Kirkuk and its nearby oil fields is a setback for Kurds, who have held the city -- home to more than one million people -- for the last three years. The Kurds took control of the city after it was abandoned by Iraqi forces during ISIS' lightning offensive in 2014, but it lies outside the recognized borders of the autonomous Kurdistan Regional Government.

Now, driven out of Kirkuk, with their dreams of building a separate nation in northern Iraq suffering a major setback, displaced Kurds are still reeling.

Families flee Kirkuk on the road to Erbil and Sulaymaniyah on Monday.

From noisy cafés and bars, to the quiet of people's homes, everyone in Erbil -- the capital of the autonomous Kurdish region of northern Iraq -- is trying to make sense of what happened.

Many of the Kirkuk residents who fled to Erbil for fear of potential clashes expressed their displeasure, grief, and shock that Iraqi troops -- backed by Shia militias known as the Popular Mobilization Units -- were able to take the city in a single day.

The same forces were later cheered on by Arab and Turkmen residents on Monday as they removed a Kurdish flag that had flown over the Kirkuk governor's office.

"The shock is too great and I cannot imagine what happened overnight, especially after all the threat and intimidation of the Peshmerga forces against those approaching the province," Samal Omar, a 33-year-old government employee, told CNN.

"We fled to Erbil before noon on Monday when we heard that the Popular Mobilization [Units] would enter Kirkuk for fear of aggression."

Some residents celebrate after Iraqi forces took control of Kirkuk on Monday.

Some Kurdish civilians said they took up arms and deployed to the streets in an attempt to ward off the Iraqi army operation.

One of them, Mohamed Werya, 37, said he didn't sleep for two consecutive days before fleeing Erbil.

"I saw officials leave and I said to myself, 'why should I stay and danger my life and my family?'" Werya said, describing chaotic scenes as people scrambled to flee the city. "What I saw on the road I have not seen before, only during the [Kurdish] uprising of 1991."

"Who is responsible for what happened?" he asked.

A version of the same question was echoed by other Kurds.

Kurdish forces open fire on Iraqi troops in the streets in Kirkuk on Monday.

"I cannot express my sorrow and my displeasure. But the question is, why did the Peshmerga forces withdraw? Why they did not they tell us earlier and we lived in a strong feeling that there was someone defending us?" Abu Mahmoud, 55, asked.

"I did not expect the effort of many years lost in 40 minutes," he added.

There was still much confusion over what transpired during the clashes between Iraqi and Kurdish forces, with reports of a split between Kurdish factions. The Peshmerga General Command accused members of the Patriotic Union of Kurdistan, a political party within the Kurdistan region, of abandoning their posts as Iraqi forces entered, in what it described as a betrayal.

Kirkuk residents cross a Kurdish checkpoint in Altun Kupri on Monday.

"No one understands what exactly happened," said Fouad Aziz, 40, who fled Kirkuk for Sulaymaniyah, while his brother went to Erbil. "There are accusations against the Kurdish parties in the region, some accused of deception and leaving the fighting sites."

"We were welcomed by the people of Erbil and Sulaymaniyah in a way we did not expect," he added.

Dozens of young people lined the road to Erbil on Monday, handing out food and water to those fleeing, while other Erbil residents opened their doors to those displaced by the fighting.

Abu Nebez, 55, said he hosted dozens in his home.

"I welcomed in my house seven families consisting of 37 people, 14 of whom we do not know," Nebez said. "They were on the main road in a deplorable condition when they fled from Kirkuk to Erbil."

Locals wave to Iraqi forces as they arrive in southern Kirkuk on Monday.

On Tuesday, some Kurdish residents began to trickle back into Kirkuk, wary of what might come next.

Qais Book, a Kurdish blogger and social media consultant who lives in Kirkuk, stayed behind as others fled on Monday.

He watched the celebration in the governor's square, as Arab and Turkmen residents celebrated.

"There are many different feelings in the city now," Book said. "Some people feel disappointed about what happened, especially the Kurdish people, and some of the Arabs, because they were loyal to the Kurds here. And they feel sorry because many Kurdish families left their houses here and fled to Kurdistan."

"The city is calm now, but people are waiting to see what happens next."