The bullet holes, smashed masonry and blood on the walls of the burned-out consulate on the southern outskirts of Benghazi are testament to the gun battle that raged through the building, claiming four lives including that of the US ambassador, Chris Stevens.

As the owners of the damaged building and the accommodation block a mile down the road took the Guardian on a tour of the sites on Thursday, reports that the attack was the work of an isolated group seemed to be at odds with the physical evidence and what their staff had told them.

"Better security would not have stopped this," said Adel Ibrahim, the owner of the accommodation building where blood is now spattered beneath a hole smashed in a wall by a heavy projectile. "A security unit is fine if you are facing 10 persons, but there were 400 attackers. [The Americans] would have needed an army to stop them."

It is clear the US staff and their Libyan guards were subjected to a terrifying, night-long ordeal, which began with protests outside the consulate in the al-Fawahat residential district. Demonstrators gathered in the narrow Venice Street outside the main gated entrance, voicing protests against the reported release in the US of a film that ridiculed Islam.

The protest quickly turned violent, the landlords said. Who fired first is a matter of dispute, with some claiming the Libyan security guards hired by the Americans shot in the air, panicking the crowd.

There is no doubt about what happened next. The compound is bound by a breeze-block wall topped by barbed wire, but it was not enough to stop the attackers. "They jumped in from everywhere," said Ahmed Busheri, owner of the consulate.

In the melee, a single rocket-propelled grenade appears to have been fired from within the compound, detonating on one of the heavy concrete barriers set up outside to deter car bombs.

Several of the consulate's Libyan guards were injured during the fighting and others melted away, the landlords said, leaving the diplomats to fend for themselves.

Embassy staff had prepared for such an eventuality, having built themselves two positions with sandbags on either side of a villa to the left of the main gate. Three of the Americans – including Stevens – were forced to retreat from these positions towards the small terrace in the centre of the villa. This, according to Busheri, is where two of Stevens' security guards died. Stevens himself ran inside the villa, the landlords said, where he suffocated in the smoke from the blaze that has left the interior of guest rooms and a kitchen charred, blackened and heavy with the stench of putrefying food.

Once the mob had seized the compound they looted it. The surrounding lawns are covered in stray US army ready meals, broken furniture and, incongruously, the dustjacket of Simon Sebag Montefiore's book on the history of Jerusalem.

Damage inside the burnt consulate building in Benghazi. Photograph: Gianluigi Guercia/AFP/Getty Images

There is evidence that at least some of the demonstrators were horrified by what they saw: a group of civilians found the ambassador, smoke-blackened and bleeding from a cut in his head, and rushed him to the city's main hospital, Benghazi Medical Centre.

Staff there had been expecting the ambassador at 11am on Wednesday as he had come to Benghazi to inaugurate a landmark medical exchange project between the centre and Harvard Medical School, the centre's director, Dr Fathi al-Jehani, said. Instead, Stevens' body arrived at the emergency ramp at 2am, together with a Libyan embassy translator who had been shot in the leg.

There was confusion as the civilians unloaded them from two private cars which had come tearing into the hospital grounds. "He had no form of identification on him," Jehani said. "One of the people with him said: 'This is the ambassador. The ambassador is suffocating from the smoke.'" Emergency staff had heard the sound of fighting hours before and sent an ambulance, but it became lost in the confusion. "We tried to revive him," Jehani said. He did not finish the sentence.

The night's fighting had not finished. A mile south of the consulate is a smaller walled compound used as accommodation for most of the dozen US staff posted to Benghazi. As the crowd surged into the consulate, one group of diplomats managed to bundle themselves into a white armoured jeep, forcing it out through the main gates, the landlords said. It tore down the highway towards the accommodation block, surviving an attack as it drove through the armed mob.

Busheri said the jeep arrived at the accommodation compound gates at 4am and, an hour later, the protesters that had torched the consulate arrived outside. They "came here to try and ambush, there was an exchange of fire," Busheri said.

Inside, he showed the result of the fighting. A projectile, possibly a rocket-propelled grenade, had clipped the outside wall and smashed into the main building, blowing out a chunk of masonry. Below it, flecks of brownish blood were spattered on the garden wall, where the fourth American was killed. Staff held off the mob for an hour until government security forces arrived on the scene as dawn broke.

Both compounds now lie empty. The consulate's buildings have been torched and three embassy cars are blackened wrecks. US officials arrived at the accommodation block on Wednesday to remove the belongings of the staff living there.

US consulate, Benghazi. Photograph: Gianluigi Guercia/AFP/Getty Images

Signs of the diplomats remain. The grey dumb-bells of a makeshift gym remain outside the main gate. Nearby, a whiteboard pinned to the wall has the instruction: "pick up your trash before leaving".

"If I could talk to the families of those who died I would say we are sad because, you know, the ambassador has been good for all Libyans," said Kolan Garmud, who co-owns the accommodation building with Ibrahim. "We will not forget how he helped us in the revolution."

It is a refrain heard across this shocked city. Stevens was one of the first US diplomats to arrive in Benghazi at the beginning of last year's revolution. He had a reputation here and in the capital, Tripoli, for his enthusiastic backing of medical and social initiatives, and for his informal style.

"He had good relations with my colleagues," said Jehani.

"So he was loved?" a journalist asked.

"He is loved," said Jehani, emphasising the word "is".

Almost overcome with emotion, he said two Boston doctors who had arrived for the inauguration this week had flown home. "The people who did this, they are a very small minority," Jehani said. "We did not get out of the [Muammar Gaddafi] dictatorship to be under somebody who is driven by hate."

Benghazi has suffered from activities blamed on jihadists for several months, and anger is directed equally at the protagonists and at a government seen as unwilling to stand up to them.

Last April, a UN convoy carrying the envoy Ian Martin was bombed in the city, followed by attacks on the Tunisian consulate and the office of the International Committee of the Red Cross. Jihadists also wrecked a Commonwealth war graves cemetery in the city, and, though it has been repaired, the vandalised stone cross that formed the cenotaph has not been replaced. In June, the British ambassador, Dominic Asquith, narrowly survived when a rocket missed him and hit the armoured car carrying his security detail, two of whom were wounded.

The Islamist Ansar al-Sharia brigade is blamed by many here for the violence, particularly with the appearance of a video that appears to show two US embassy vehicles plundered from the burning compound being driven into the brigade's Benghazi barracks. Brigade leaders have issued a denial of responsibility but were unwilling to talk to journalists on Thursday.

"They [Sharia] have pros and cons," said Saleh el-Fellah, who works in Benghazi's Friendship Cafe. "They were the ones who provided security [in the revolution]. The cons are all the bad things they have been doing. They take everything to extremes, they are ruining the reputation of Islam."

Sadness is everywhere in the city whose proud boast is that it was the cradle of last year's revolution. A Libyan photographer inside the ruined compound came over to offer an unsolicited apology. "We are not all like this," he insisted. "Libyans do not approve of this."

There is also unease about how America will react. It has not had an ambassador killed since 1979, and President Barack Obama's vow that "justice will be served" has fuelled rumours that US marines now deployed in Tripoli will head to Benghazi and take action.

But many also hope the government will move against the protagonists. "They have to show a fist of steel," said Mohammed el-Kish, a former press officer with the National Transitional Council which handed over power last month to Libya's newly elected congress.

"They have got to rid us of these people."