Yeonpyeong Island, South Korea (CNN) -- Amid South Korean plans to hold live-fire military drills this week and North Korean threats of retaliation, many residents of Yeonpyeong Island are evacuating, afraid of being caught in the middle.

Villagers streamed onto what they believed was the last ferry to Incheon, South Korea, on Sunday, carrying what belongings they could. Some were holding their children as others helped the elderly.

"I'm leaving because they said the drills are tomorrow," said Kim Ok Jin, 66.

Kim said the island was once a good place to live and she does not want to leave.

"Of course I'm angry," she said. "But that's not going to change anything."

See more of CNN.com's special coverage of the Koreas

Many Yeonpyeong residents are evacuating for a second time. This time, however, they have warning. On November 23, they had none before North Korea began shelling the island. Two civilians and two South Korean Marines died in the attack.

The South Korean military said Thursday that its drills will take place in the seas southwest of Yeonpyeong Island between December 18 and 21, but bad weather forced a delay Saturday. South Korea's Yonhap news agency reported the drills will take place Monday or Tuesday in the Yellow Sea off the island.

As villagers departed, few signs of life remained on the island -- only military trucks patrolling and waiting for possible North Korean retaliation for the drills.

Not everyone on Yeonpyeong, however, was leaving. Song Young Ok said she has not been told to stop selling tickets for the ferry off the island, and doesn't know when the military drills will take place.

Song said she is planning to stay put, holed up along with others in a military bunker.

She said she doesn't know why it has to be this way -- if South Korea carries out the drills, North Korea has threatened to retaliate even more strongly. South Korea's insistence on conducting the drills is picking a fight, she said.

Others were more optimistic. "It's OK," said ferry passenger Lee Chun Nyeo, 83. "The soldiers need to do (the exercises), right?"

Yeonpyeong is located just south of the Northern Limit Line, the line drawn in 1953 by the United Nations just after the Korean War. The U.N. drew the line three nautical miles from the North Korean coast and put five islands close to the coast under South Korean control.

That was supposed to be a temporary arrangement. But in the absence of a full peace agreement, the Northern Limit Line remains in place.

North Korea has suggested an alternative line, but South Korea has resisted, as it would bring the North's maritime boundary close to Incheon, a main port.

Waters in the Yellow Sea are important for fishing and crab. For North Korean fishermen especially, the blue crab season between June and September is an important source of income. Crabs have a habit of migrating south during that time, so the water is sometimes crowded with boats from both countries, as well as vessels from China. However, the Yellow Sea has seen armed clashes in the past few years, the most serious of them in 1999 and 2002.

China and Russia have asked South Korea to reconsider holding the drills. Russia called for an emergency meeting of the U.N. Security Council on Sunday.

CNN's Tim Lister contributed to this report.