The workers return long after the sun has set. Some walk across the concrete bridge which is the only link between the slum and the old city of Dhaka, then head towards the tenements on the far side. Others simply step off the high embankment beside the stinking river and, leaving the orange glow of three flickering street lights, disappear into the slum's narrow lanes.

In one alley, behind a mosque and a carpenter's workshop, is a row of tin shacks which are home to about 200 people. As elsewhere across Kamrangir Char, one of the biggest and poorest slums in the world, most of the men here work on construction sites or pedalling rickshaws. Women are employed as domestic staff for the city's growing middle class or, increasingly, in the booming garment industry which supplies tens of millions of cheap shirts, trousers, sweaters and socks to high street retailers in the west.

Sitting on a plastic chair outside his shack, Mohammed Jahangir is, like many of the 160 million inhabitants of Bangladesh this weekend, angered by the unstable south Asian nation's politicians. For most of last week Dhaka was paralysed by violent protests launched by the opposition party to mark its hostility to the current government's plan to hold an election in January without installing a neutral caretaker administration first. More than 20 people died as activists burned buses and threw makeshift bombs at police, who replied with teargas and live rounds. Most casualties were bystanders, caught in the crossfire.

"This is how our country is. This is how our leaders are. I am a registered voter, but I am not going to vote," Jahangir, a mason, says. "A poor man's vote never makes any difference."

His wife recently lost her 4,000 taka (£31) per month job as a timekeeper at a garments factory making trousers for a well-known western brand. Jahangir blames a downturn in orders from the west following the collapse in April of a huge complex in the north-west of Dhaka housing more than 3,000 garment workers. More than 1,100 were killed in the worst industrial accident for a decade. Many worry that the industry will now move elsewhere, worried in case more tragedies further tarnish carefully marketed brands.

"The rich get richer and the poor get poorer or they die. That's how it is here," Jahangir, 35, says.

More than four million people in Bangladesh work in the garment industry and economists estimate that at least as many again owe their jobs to the demand it creates. Four-fifths of exports from Bangladesh are garments. "If there is a pullout [of western buyers], it will be a catastrophe for Dhaka. It is not just a single-industry-based city. It is a single-industry-based country," said Professor Nazrul Islam, a local analyst.

In fact, the three months following the collapse saw Bangladesh garment exports increasing, not diminishing, and though a handful of brands have pulled out, more than 100 others have signed legally binding agreements committing them to the country for at least five more years. Primark, H&M and others all insist their commitment to Bangladesh is long-term. Srinivas Reddy, local director for the International Labour Organisation, believes the industry has reached "a turning point".

One reason brands are prepared to stay in Bangladesh and pay considerable sums to upgrade factories there is that the country plays a key role in their supply chains, with an almost unique capacity to deliver vast quantities of clothes quickly and cheaply. The Observer watched one woman in a Dhaka factory that produces clothes for a European retailer as she sewed buttons on to shirts at a rate of 600 an hour. One of 1,800 workers on the production line, she works six 10-hour days a week.

But western buyers are deeply concerned by the political turmoil. Though, by mutual agreement, the garment sector is spared the shutdowns that have brought Dhaka and other cities to a halt in recent months, the roads are not. Workers struggle to get to their jobs with transport paralysed and violent clashes on the streets. Trucks carrying garments from the big production zones around Dhaka are often targeted as they make their way down to the port at Chittagong for shipping to Europe. In the first nine months of 2013, 1,000 trucks were set alight along the route. "One truck and its load is a lot of money. So we are very scared about what is going to happen. The buyer doesn't want [their order] even a week late," said Rubana Huq, managing director of the Mohammadi Group.

There is little chance of the violence subsiding in the short term. Since it gained independence from Pakistan in a civil war in 1971, politics in Bangladesh has been turbulent. In recent decades it has been dominated by the battle between Sheikh Hasina, daughter of civilian independence leader and "father of the nation" Sheikh Mujibur Rahman, and Khaleda Zia, widow of the military independence leader Ziaur Rahman. Both men were assassinated.

Hasina's Awami League won power in polls held in 2008, following a period of military-backed government. After years of relative calm, chaos returned to the streets this year with prolonged bouts of demonstrations and counter-demonstrations. Many have been organised by the opposition Bangladesh Nationalist party (BNP) and its current allies, the Islamist organisation Jamaat-e-Islami. Others have involved a movement of religious conservatives called Hefezat-e-Islam.

Senior Awami League officials and some analysts speak of a "contest for the soul of Bangladesh". One said last week: "This has been a country of practising Muslims who are tolerant and pluralistic with a state that is fundamentally secular. Now that is threatened by these conservatives."

The BNP, however, rejects the charge. "If you let [the religious conservatives] loose on the streets, they will create mayhem. It is better to bring them inside. The real problem is corruption," said Shamsher Chowdhury, the party's vice-chairman.

The opposition may boycott the polls in January. The Awami League and Hasina have said they are determined to hold the election whatever the level of chaos. Further demonstrations are planned for today and later this week.

Ahsan Mansur, director of the Dhaka-based Policy Research Institute, said: "A compromise looks unlikely. I really can't see how or why. The Awami League thinks that if there is a fair election it will be out of power. The BNP think that, if there isn't a fair election, they will be out of power … The longer the hostilities continue, the more bloody and bitter the aftermath."

The violence on the streets echoes only distantly within Kamrangir Char's labyrinth of lanes. Only a mile or so from the site of bomb blasts, mechanics batter rusty metal scraps back into shape in a thousand dim workshops, porters push carts with vegetables, and children play badminton amid clouds of acrid smoke from the burning garbage by the riverside.

But for one community the politics have a direct impact. Negotiations are still continuing between garment factory owners, brands and unions over how much compensation will be paid to the survivors of the collapse in April and relatives of the dead. Campaigners have told the country's high court that they should receive 2,800,000 taka (£22,000) under international laws instead of the local legal minimum of 100,000 taka (£787) some factory owners have suggested.

A decision was unlikely until the instability subsided, said Jyotirmoy Barua, a lawyer representing the survivors and the bereaved. "We are waiting to get that [level of compensation] approved by the courts. But if anything goes on politically the high court doesn't sit, so we have to wait until everything in Bangladesh settles down."

Nobody believes that will be soon.