Story highlights Two brothers, four and two, died after being swept away by floodwaters during Sandy

Their mother said she had sought shelter at a nearby house, but was turned away

The homeowner has said he only saw a man outside, and could not help him

He said the mother, Glenda Moore, should not have been driving during the storm

As Superstorm Sandy ravaged New York, Glenda Moore drove frantically across Staten Island in an attempt to get her sons to safety.

Instead, Moore found herself and her boys -- Connor, four and Brandon, two -- caught in the full fury of the storm.

Buffeted by torrential rains and winds of up to 90 miles per hour, her Ford Explorer plunged into a hole. According to the account she would later give police, Moore carried her sons to a nearby tree, gripping branches along with her boys as she tried to shelter them from the storm surge.

She told police they clung together for hours, before Moore managed to make her way to a nearby property, and pleaded to be let inside. But according to her police account, rather than sheltering the desperate strangers, the occupant refused to let them enter.

In desperation, Moore told police she then went to the back of the house, and tried to break in using a flower pot, but was unable to do so. As the storm raged on, her sons were swept away by floodwaters.

The bodies of the boys were found near each other Thursday, about a quarter of a mile from where Moore last held them.

Relatives said Moore was too distraught to speak with CNN.

Meanwhile, public anger has been directed at the homeowner who allegedly failed to help Moore and her children. The man, who told CNN's Gary Tuchman that his name is Alan but did not want his full name used, disputed Moore's account, saying he saw only a man outside.

"He didn't come to the door... he must have been standing at the bottom of the stairs," said the man. "He took a concrete flower pot... and threw [it] through the door."

The man at the door didn't ask to enter the house, he said, but instead asked him to come outside in order to help.

"What could I do to help him?" he asked. "I had a pair of shorts on with flip-flops."

The man told CNN he sat up for the rest of the night, with his back against the door in the kitchen.

He said he did not know the fate of the children. Told that their bodies had been found, he said the deaths were a tragedy, but implied that the woman was at fault.

"It's unfortunate. She shouldn't have been out though. You know, it's one of those things," he said.

He said there was nothing he could have done. "I'm not a rescue worker ... If I would have been outside, I would have been dead."

The man said he had given his account to police.

Legal experts consulted by CNN said that no crime would have been committed by a failure to render assistance.