Seoul, South Korea (CNN) -- Rescue boats and helicopters scrambled to pluck passengers, most of them high school students, from a ferry as it listed and slowly sank off the southwest coast of South Korea on Wednesday.

It's unclear exactly how many of the 477 people on board were rescued. Many jumped from the listing ship to the freezing waters below.

At one point, South Korean authorities said 386 passengers had been rescued and that 104 remain unaccounted for. But later Wednesday, they said they couldn't provide exact numbers.

What is known is that at least two people -- a woman and a student -- were confirmed dead.

The rescue operation was still underway six hours after the ferry first sent out distress signal.

Authorities could not immediately say what caused the ship to sink. The weather at the time of the incident in the area was clear.

'I wanted to live'

Among the passengers the ferry, Sewol, was carrying were 325 students.

The group left from the port city of Incheon, just west of Seoul, for a four-day trip to the resort island of Jeju.

Around 9 a.m. local time, the ferry sent out its first distress call. It had begun to list.

A rescued student, Lim Hyung Min, told CNN affiliate YTN that he heard a loud bump. The ferry began to sink after that. Everyone was ordered to don life jackets and jump, he said.

Lim said he jumped into the sea before swimming to a rescue vessel.

"I had to swim a bit to get to the boat to be rescued," he said. "The water was so cold and I wanted to live."

As rescue crews dashed desperately to rescue passengers, the ferry slowly tilted on its side.

With the clock ticking, the 6,800-ton ferry sank. Only its white and blue hull remained above water.

Local media, including CNN affiliate YTN, reported that all students aboard the ship had been rescued. The South Korean Coast Guard hasn't confirmed those accounts.

Passenger Kim Seung Mok said that, despite his efforts and those of others, he couldn't get to several passengers on one of the decks.

"I stayed till the last to rescue people at the hall," Kim told YTN. "But the water was coming in so fast (that) some didn't make it out."

Survivors tell of panic on board as ferry tilts, then capsizes

More on other major ferry and ship sinkings

Cargo ships collide outside of Tokyo Bay

Philippines ferry disaster kills dozens

CNN's Euan McKirdy wrote and reported from Hong Kong. Paula Hancocks and journalist Stella Kim in Seoul contributed to this report.