• Reports of more than 40 killed in and around UN shelter • 12 members of family killed in Gaza City air strike

The civilian death toll in Gaza increased dramatically today, with reports of more than 40 Palestinians killed after missiles exploded outside a UN school where hundreds of people were sheltering from the continuing Israeli offensive.

Two Israeli tank shells struck the school in Jabaliya refugee camp, spraying shrapnel on people inside and outside the building, according to news agency reports.

The medical director of the hospital in Jabaliya told the Guardian 41 bodies had been brought in so far and more could be on the way. Reuters journalists filmed bodies scattered on the ground amid pools of blood and torn shoes and clothes. In addition to the dead, several dozen people were wounded, hospital officials said. The Israeli military said it was looking into the reports.

A United Nations official in Gaza said the school was clearly marked with a UN flag and its location had been reported to Israeli authorities. John Ging, director of operations in Gaza for UNRWA, the United Nations Relief and Works Agency, said that three artillery shells landed at the perimeter of the school where 350 people were taking shelter. "Of course it was entirely inevitable if artillery shells landed in that area there would be a high number of casualties," he said.

Asked whether there were Hamas militants in the area at the time of the attack, Ging said it was the scene of clashes "so there's an intense military and militant activity in that area." He said UN staff vetted Palestinians seeking shelter at their facilities to make sure militants were not taking advantage of them. "So far we've not had violations by militants of our facilities," he said. Ging called for an independent investigation of the strikes near UN facilities.

"I saw a lot of women and children wheeled in," Fares Ghanem, a hospital official told the Associated Press. "A lot of the wounded were missing limbs and a lot of the dead were in pieces."

Majed Hamdan, an AP photographer, who rushed to the scene shortly after the attacks, said many children were among the dead. "I saw women and men parents slapping their faces in grief, screaming, some of them collapsed to the floor. They knew their children were dead," he said.

"In the morgue, most of the killed appeared to be children. In the hospital, there wasn't enough space for the wounded."

Elsewehere, at least 12 members of an extended family, including seven young children, were killed in an air strike on their house in Gaza City. The bodies of the Daya family were pulled from the rubble of a house in Gaza city's Zeitoun district after it was hit by two Israeli missiles. The dead included seven children aged from one to 12 years, three women and two men. Nine other people were believed to be trapped in the rubble.

Hours earlier, three young men – all cousins – died when the Israelis bombed another UN school, the Asma primary school in Gaza City. They were among about 400 people who sought shelter there after fleeing their homes in Beit Lahiya in northern Gaza.

The UN, which said the school in Jabaliya was clearly marked, said it was "strongly protesting these killings to the Israeli authorities and is calling for an immediate and impartial investigation".

"Where it is found that international humanitarian law has been violated, those responsible must be held to account. Under international law, installations such as schools, health centres and UN facilities should be protected from attack. Well before the current fighting, the UN had given to the Israeli authorities the GPS co-ordinates of all its installations in Gaza, including Asma elementary school."

The killings take the total toll in Palestinian lives since the Israelis launched their assault on the Gaza Strip 11 days ago to above 600. Doctors at Gaza hospitals say that at least one-fifth of the victims are children and a large number of women are among the dead.

Israel continues to insist that the bulk of those killed are Hamas and Islamic Jihad fighters, although its claim to be going to extraordinary lengths to target only "terrorists" has been undermined by one of its own tanks firing on a building being used by Israeli troops, killing four.

The sharp spike in the number of civilian casualties came as Israeli troops and tanks moved into Gaza's second largest city, Khan Younis, for the first time today, supported by intensive artillery strikes as the military pledged to press on with its attack.

In a separate attack earlier in the day, three Palestinians were killed in an air strike on another school run by Unwra, the UN relief agency.

Nine Israelis, including three civilians hit by rocket fire, have been killed in the conflict. At least five rockets fired from Gaza landed in Israel today, including one that hit the town of Gadera, 17 miles from Tel Aviv, police said. A three-year-old girl was wounded.

The heaviest fighting has been in northern Gaza, with witnesses reporting wave after wave of bombing strikes across the north of the territory accompanied by gunfire from helicopters and artillery from land and sea. Thousands of Palestinians have been ordered to leave their homes or forced to flee the fighting.

In Shajaiyeh, east of Gaza City, Israeli troops seized control of three apartment blocks and set up gun positions on the rooftops. Residents were locked in their homes and soldiers confiscated their mobile phones, neighbours said.

Three of the four Israeli soldiers killed by friendly fire died when a tank mistakenly fired on a building where the soldiers had taken up positions. There was heavy artillery fire to cover the evacuation of 24 soldiers who were injured, including the commander of the Golani infantry brigade, one of Israel's key fighting forces.

Israel's defence minister, Ehud Barak, said his country's troops would continue their operation despite mounting Palestinian casualties and growing international calls for a ceasefire.

"Hamas has so far sustained a very heavy blow from us, but we have yet to achieve our objective, and therefore the operation continues," Barak said.

The Israeli foreign minister, Tzipi Livni, said the offensive was intended to change permanently the shape of Israel's conflict with Hamas. "When Israel is targeted, Israel is going to retaliate," she said. Israel has rejected calls for a ceasefire.

The military said it had bombed more smuggling tunnels across the border with Egypt, in the south, and hit more than 40 other sites across Gaza including buildings storing weapons and rocket launching areas.

In Gaza, Mahmoud Zahar, the most senior leader of Hamas in the strip and a hardliner in the movement, appeared on the party's al-Aqsa television station and gave a defiant speech threatening attacks not only in Gaza but elsewhere.

"The Zionists have legitimised the killing of their children by killing our children. They have legitimised the killing of their people all over the world by killing our people," Zahar said. He urged Hamas fighters to "crush your enemy".

Another Hamas figure, a recognised military spokesman called Abu Ubaida, said thousands of Hamas fighters were waiting in Gaza to take on the Israeli military, and that rocket attacks would increase. More than 40 were fired into southern Israel yesterday, including one that landed in an empty kindergarten, which, like all schools near the Gaza border, has been closed since the conflict began.. Israeli police said a total of 520 rockets had been fired in the past 11 days of fighting.

Israeli troops are now deployed in and around the major urban areas of Gaza, particularly to the north, in Beit Hanoun, Beit Lahiya and Jabaliya. Using leaflets, telephone calls and radio announcements, they have ordered residents in many areas to leave their homes, forcing at least 15,000 Palestinians to flee to safety elsewhere. At least 5,000 are staying in 11 different UN schools and shelters.

The UN said more than 1 million Gazans were still without electricity or water and that it was increasingly difficult for staff to distribute aid or reach the injured. It said more industrial diesel was needed to reopen the strip's sole power plant, which has been shut for a week. Ten transformers have been damaged in the fighting.

More wheat grain is needed for food handouts, and the UN said Karni, the main commercial crossing, should be reopened to allow it in. Four ambulances and three mobile clinics were destroyed when bombs hit the headquarters of the Union of Health Care Committees in Gaza City.

John Holmes, the UN emergency relief coordinator, said Gaza represented an "increasingly alarming" humanitarian crisis, and that the territory was running low on clean water, power, food, medicine and other supplies since Israel began its offensive. Israeli leaders claim there is no humanitarian crisis.