As many as 4,000 people are believed trapped by landslides in a valley near a Hindu shrine in the Indian Himalayas, days after floods killed more than 100 people.

Helicopters have ferried rescue workers and doctors along with equipment, food and medicine to Kedarnath in the state of Uttarakhand, the nearest town. Most of those stranded are Hindu pilgrims who were visiting four shrines.

Amit Chandola, a state spokesman, said authorities had so far been unable to reach eight villages feared washed away by the weekend floods in the worst-hit districts of Rudraprayag and Chamoli.

A damaged road near Govindghat in Uttarakhand, India, after heavy monsoon rain caused landslides. Photograph: AFP/Getty Images

He said the official death toll in Uttarakhand state was 104 but added: "We don't know yet what happened to hundreds of people living there." An additional 17 people died in collapsed homes in neighbouring Uttar Pradesh, said RL Vishwakarma, a state police officer.

Uttarakhand's chief minister, Vijay Bahuguna, told the New Delhi Television news channel that hundreds of people had lost their lives, but the exact number would be known only after a survey.

The flooding washed away roads and more than 20 bridges, demolished 365 houses and partially damaged a further 275 in Uttarakhand, the state government said.

An Indian army rescue operation in Uttarakhand. Photograph: Indian army/EPA

Hundreds people looking for relatives flocked to Dehradun, the state capital, where flood survivors were taken by plane and helicopter.

Those rescued spoke to their stranded relatives on the phone on Wednesday.

India's prime minister, Manmohan Singh, who made an aerial survey of the region on Wednesday, said the death toll exceeded 100. "It is feared that the loss of life could be much higher," he said.

Buses and trucks are submerged in the rising waters of the Yamuna river in Delhi. Photograph: Tsering Topgyal/AP

Bahuguna said the Kedarnath temple – one of the holiest Hindu temples dedicated to Lord Shiva, in the Garhwal Himalayan range – had escaped major damage, but up to four metres (10ft) of debris covered the area around it.

The latest rains have affected several states and the capital, Delhi, where nearly 2,000 people were evacuated to government-run camps on higher ground. Authorities there said the Yamuna river was expected to start receding on Thursday afternoon.

The large number of deaths appear to be the result of both man-made and natural causes. One of the worst-affected places was Kedarnath, which is 2.5 miles from the Chorabari glacier. According to reports the town was engulfed with what's being described as a two-storey high wave of ice, mud and water after a section of the glacier to collapse during torrential rail.

"At Kedarnath it was simply a natural disaster and not the result of human intervention," said Deb Mukherjee, a former Indian ambassador to Nepal. "Sustained rain far above the normal seems to have resulted in the collapse of natural structures high up in the mountain range."

But elsewhere the floods and landslides were being linked by environmentalists to the failure of provincial governments to devise a proper plan for the Himalayan region.

"Deforestation, sand mining, stone quarrying and unregulated and excessive construction of buildings have caused havoc to the environment in Uttarakhand," said Medha Patkar.

Scores of landslides and flood-swelled rivers swept away many homes and bridges. "Disturbingly, many of the houses had been built too close to rivers like the Mandakini are were swept away by the fury of the waters," said Mukherjee.