Speaking out against the World Bank comes with consequences, according to Human Rights Watch. The group charges that an atmosphere of intimidation by governments and powerful companies has had a chilling effect. Within the World Bank, most staff members say they fear punishment if they were to report unethical conduct, according to an internal survey that was leaked.

The accusations come at a time when support for World Bank head Jim Yong Kim appears to be waning within the global development organization. Taken together, the information shows an organization struggling to maintain accountability and protections for people speaking out.

People openly critical of World Bank-funded projects “have faced threats, harassment, and trumped-up criminal charges in Cambodia, India, Uganda, Uzbekistan, and elsewhere,” according to HRW. The New York-based rights group accused the bank of doing little to nothing to protect people facing such threats, in the report titled, At Your Own Risk: Reprisals against Critics of World Bank Group Projects.

“The World Bank has long said that public participation and accountability are key to the success of the development efforts it funds,” said Jessica Evans, senior international financial institutions advocate at HRW, in a news release. “But the World Bank’s repeated failure to confront intimidation or harassment of people who criticize its projects risks making a mockery out of these principles.”

A leaked engagement survey of World Bank staff shows the atmosphere inside is not much better. Only 41 percent of people feel that they “can report unethical behavior without fear of reprisal,” shows the survey. That is despite the fact that the majority of staff surveyed, 85 percent, say they feel proud to work at the World Bank. Yet, the International Consortium of Investigative Journalists (ICIJ) reports that there are significant issues within the organization, starting at the top.

“What people tell me is there is a fear of speaking out and disagreeing, because there is a tone of retaliation and payback that emanates from the 12th floor [the president’s office]since Dr. Kim arrived,” Paul Cadario, a former senior manager at the World Bank, told the ICIJ.

A series of reports published by the ICIJ and Huffington Post starting in April show how World Bank-backed projects have displaced tens of thousands of people. In Honduras, the lead environmental specialist for a program raised questions about practices carried out by people leading the project. An internal review showed that the specialist was replaced.

People critical of World Bank projects who are not working for the organization face more critical problems. In Uganda, organizations brought to attention the forced displacement of people living in Mubende and Kiboga districts to make room for pine and eucalyptus plantations. The International Finance Corporation, the private sector lending arm of the World Bank, provided loans for the project.

Accountability is a topic often discussed by Kim. One of his early actions after taking leadership at the World Bank was to establish a mechanism that allowed for 100 percent feedback from people affected by projects. Earlier this year, he admitted that the oft-criticized resettlement plans have failed to deliver for people moved from their homes. Activists say they faced threats and surveillance after publishing report and making a formal complaint.

“My colleague claimed he received threats via his mobile telephone. One day, someone I know said to me, ‘If you don’t do these things, they will kidnap your children.’ He had connections with the government and told me he was just telling me ‘as his friend, to drop the case,'” said an employee with the Uganda Land Alliance, to HRW.

Kim recently recognized the gravity of the problem with the World Bank’s resettlement programs. Activists continue to call attention to the issue because the cases often involve the extremely poor who lack basics like land titles and deeds to protect them from forcible removal.

“We took a hard look at ourselves on resettlement and what we found caused me deep concern,” said Kim in March. “We found several major problems. One is that we haven’t done a good enough job in overseeing projects involving resettlement; two, we haven’t implemented those plans well enough; and three, we haven’t put in place strong tracking systems to make sure that our policies were being followed. We must and will do better.”

He is also shaking up system in one of the world’s oldest development agencies. Reports over the last year show that internal reforms are gaining mixed reviews, unsurprising given that most people are comfortable with the status quo. Changes are needed to improve the work it does and make it more accountable.

“These recent failures at the Bank present a legitimacy crisis. To restore its credibility, the Bank needs to rethink its approach to accountability and the reforms needed to achieve it,” wrote Natalie Bridgeman Fields, founder and executive director of Accountability Counsel, and Kindra Mohr, accountability counsel’s policy director, in the Council on Foreign Relations blog in May.