The four children, aged between one and six, were found dead in the ruins of their home in the town of Beit Hanoun after their house had been hit by a tank shell as they were reportedly having breakfast. Their mother died later in hospital. Her two elder children were
critically wounded in the attack.
The incident looks set to lead to renewed claims that Israel is engaging in the 'collective punishment' of Palestinians living in Gaza, frequently used as a base for rocket attacks by militants on southern Israel.
A 17-year-old Palestinian passer by was also killed as a result of the military operation. In a separate incident in the area, Israeli soldiers killed an Islamic Jihad gunman.
The Israeli army said it had carried out a ground operation in Beit Hanoun, which has been used by Palestinian militants for cross-border rocket attacks. An Israeli soldier was slightly injured during the operation.
"We are looking into several versions of what happened and we cannot definitely say that we are not responsible [for the civilian casualties]," an army spokesperson said, adding that, "In any event, the attack was aimed at militants. They may, however, have taken shelter among peaceful civilians."
Earlier in the day, militants in Gaza launched at least seven Qassam rockets and nine mortar shells at the western Israeli Negev region, but no casualties or damage were reported.
Shortly after the attack, Israeli Defense Minister Ehud Barak said that the radical Islamic group, Hamas, which seized power of Gaza last June, was to blame for the deaths.
"We consider that Hamas is responsible for everything that goes on there... The army is acting, and will continue to act, against Hamas, including inside the Gaza Strip," The Haaretz newspaper quoted the official as saying.
Last week Hamas issued a statement saying it was ready to accept a six-month ceasefire with Israel, first in the Gaza Strip and then extending to the West Bank, but the Israeli government rejected the offer.
Palestinian President Mahmoud Abbas condemned Monday's attack, saying that it proved that Israel was not interested in a cessation of hostilities.
"This aggression does not aid efforts being exerted to achieve calm [in the region], and it hinders the peace process" Abbas told the WAFA news agency.