(CNN) -- Hundreds dead. Many more infected. Pervasive fear and denial are challenging authorities in west Africa who are trying to assess and address the Ebola crisis.

So far, it has killed more than 650 people, says Doctors Without Borders.

That's the highest death toll the World Health Organization has recorded in an Ebola outbreak. And it's getting worse.

The swelling numbers prompted heads of state to cancel travel plans on Thursday to direct their full attention toward fighting the outbreak of the virus that has crippled parts of Liberia, Sierra Leone and Guinea and stirred palpable concerns that it will spread around the region and the world.

Liberian President Ellen Johnson Sirleaf and Sierra Leone's President Ernest Koroma both canceled trips to the United States, and Koroma declared a state of emergency.

He announced a action plan that addressed many of the barriers international medical workers complain they face while fighting disease.

Some residents in affected villages have accused them of bringing the disease into the country and have barricaded their towns or otherwise blocked access to Ebola hemorrhagic fever victims.

A nurse with Doctors Without Borders, Monia Sayah, told CNN, "the most challenging" aspect of trying to help people is that "we go into communities where we are not necessarily welcome," because people don't want to believe they or their loved ones have Ebola -- in part because "they understand now that the survival rate is not very high."

Koroma said he will deploy police and military to accompany the aid workers.

They will search house to house for the infirm and enforce orders designed to curb the virus' spread.

What is the risk of catching Ebola on a plane?

Aid groups pull out

"The matter has reached a crisis point," Liberian Information Minister Lewis Brown told CNN's Richard Quest. "...The dire prognosis is that it will get worse before it gets better."

The dangers are so real that some humanitarian organizations are pulling out to protect their own.

Samaritan's Purse -- an international evangelical Christian humanitarian agency -- and the missionary group Serving in Mission have recalled all nonessential personnel from Liberia.

The Peace Corps announced Wednesday it is doing the same, removing its 340 volunteers from that country, as well as Sierra Leone and Guinea.

While there are no confirmed cases, a spokeswoman for the agency did say that two volunteers did come in contact with someone who ended up dying from the virus.

Those Americans haven't shown signs of Ebola but are being isolated just in case, with the spokeswoman saying they can't return home until they get medical clearance to do so.

American dies

One American, 40-year-old Patrick Sawyer, died in a Nigerian hospital earlier this month -- having come from Liberia, where he was a top Ministry of Finance official, and before he could go back home to Minnesota to celebrate his daughters' birthdays.

But the vast majority of those afflicted are Africans. They come from big cities and small villages, some of them falling ill without really knowing what hit them.

"This epidemic is without precedent," said Bart Janssens, director of operations for Doctors Without Borders, a group also known as Médecins Sans Frontières. "It's absolutely not under control, and the situation keeps worsening."

As of now, the outbreak has been confined to west Africa. But there are rising concerns that it could spread, especially since a person may not know they have Ebola or show symptoms for two to 21 days after being infected.

Sawyer, for example, collapsed getting off a plane in Lagos, Nigeria. He very well could have made it out of the region, perhaps to the United States, before showing symptoms of Ebola; it's only then that the virus spreads.

To further complicate matters, signs of Ebola include fever, headaches, weakness and vomiting -- symptoms that also define many other ailments, from malaria to the flu that Brown notes often pop up "at this time of year."

For all these reasons and more, Janssens says, "If the situation does not improve fairly quickly, there is a real risk for new countries to be affected."

Could outbreak spread to U.S.?

2 infected Americans

Ebola spreads through the transmission of bodily fluids. Those most at risk are loved ones of those infected, as well as health care workers tending to the ill.

Sawyer is believed to have been infected by his ailing sister, who he spent time with in Liberia -- even though neither likely knew she had Ebola -- according to Brown.

Then there are those like Dr. Sheik Humarr Khan, who fell ill early last week while overseeing Ebola treatment at a Sierra Leone hospital and died days later.

Two Americans affiliated with Samaritan's Purse who also were infected "have shown a slight improvement in the past 24 hours" though both are in serious condition, according to the Christian humanitarian agency.

One of them is Dr. Kent Brantly, a 33-year-old who last lived in Fort Worth, Texas. He has been the medical director for the Ebola Consolidated Case Management Center in Monrovia, Liberia, where he has been providing care for Ebola patients since October. After testing positive for the virus, Brantly went into treatment at ELWA Hospital.

The other is Charlotte, North Carolina, resident Nancy Writebol, a Serving in Mission member working with Samaritan's Purse to help fight the Ebola outbreak.

Ebola fears hit close to home

It is believed one of the local staff was infected with Ebola and came to work with the virus on July 21 and 22, Samaritan's Purse Vice President Ken Isaacs said. That staff member died Thursday.

"We think it was in the scrub-down area where the disease was passed to both Nancy and Kent," he said.

Brown, the Liberian information minister, noted Brantly and Writebol's fight in his CNN interview, as well as Liberia's need for more health care workers like them.

"We join the families in prayers that they can come through this and become ... shining examples that, if care is taken, one can come out of this."

Liberian president: 'Ebola kills'

On Monday, the CDC issued an alert warning travelers to avoid hospitals with Ebola patients and funerals for those patients in Liberia, Sierra Leone and Guinea due to the outbreak.

The United States is considering raising the alert to discourage "nonessential" travel to those three countries, a spokesman said.

Liberian President Ellen Johnson Sirleaf has told the Ministry of Health to consider quarantines in some areas and cremating bodies in an attempt to prevent further infection, according to Brown.

The president also urged residents to avoid public amusement and entertainment areas, and set aside Friday "for the disinfection and chlorination of all public facilities."

"My fellow Liberians, Ebola is real. Ebola is contagious. And Ebola kills," Sirleaf said. "All of us must all take extra measures announced by the Ministry of Health to keep ourselves safe.

"The government will do its part. But you must do yours."

READ: Ebola doctor in Sierra Leone dies

READ: What you need to know

CNN's Jonathan Helman, Elise Labott, Pamela Brown and Nana Karikari-apau contributed to this report.