BP BP -3.48% PLC has bet billions of dollars on Azerbaijan, a resource-rich country sandwiched between Russia and Iran whose giant reserves could help reduce Europe’s reliance on Russian gas.

But in order to extract those resources, BP has had to wage a battle to control inflated costs and root out fraud, internal documents and interviews with former employees of BP and its contractors show.

In 2013, BP’s board of directors learned that an employee of its main logistics contractor, a subsidiary of Switzerland’s Panalpina World Transport Holding Ltd., had allegedly approved roughly $16 million worth of invoices billed by a shell company for transportation services that it didn’t provide, according to internal communications viewed by The Wall Street Journal.

In 2014, BP and its partners on Azerbaijan’s giant Shah Deniz natural-gas project awarded contracts worth $2.5 billion to a consortium of oil-services companies, including firms controlled by Azerbaijan’s government. Their work was plagued by bloated costs, according to documents seen by The Wall Street Journal and people familiar with the matter. The issues prompted at least one written complaint to BP’s ethics department.

BP said it investigated the issues related to Panalpina but “did not establish any misconduct with regard to BP nor any impact on our business.”

On Shah Deniz, BP said, “We take concerns raised with us seriously and investigate serious breaches of our code of conduct.”

The incidents highlighted a struggle faced by many large energy companies: how to pull off large projects in countries with lax corruption controls. BP is the biggest foreign investor in Azerbaijan, which ranks 119th out of 168 in Transparency International’s corruption perceptions index.

“Oil companies have for a long time showed a willingness to operate in environments where there is a lot of corruption or conflict or weak institutions,” said Alexandra Gillies, director of governance programs at the Natural Resource Governance Institute, a group that advocates for transparency in commodity industries. “The profits available seem to outweigh the risks.”

Global energy companies are facing pressure from strengthened anticorruption regulations and stepped-up enforcement of bribery statutes, such as the U.S. Foreign Corrupt Practices Act and the U.K.’s Bribery Act. Other countries have cracked down on once-tolerated practices like paying local officials to expedite imports of key equipment.

“ You can’t get anything through customs [in Azerbaijan] without paying off someone. ” — Former BP manager who worked on its projects in the Caucasus

A two-year slump in oil prices also is increasing pressure on the industry to control costs and bring down spending, ratcheting up the scrutiny on contracts between oil companies and their service providers.

Over the past 20 years BP and its partners—including Azerbaijan’s state-owned oil company, Socar, and several Western firms—have pumped around 3 billion barrels of oil from Azerbaijan’s Azeri-Chirag-Deepwater Gunashli field. BP also is helming a $28 billion project to increase production from Shah Deniz.

To keep the oil and gas flowing, BP frequently imports pipes, equipment and other supplies. Since 2012, Panalpina has arranged thousands of these shipments.

Documents reviewed by the Journal show the manager of Panalpina’s Azerbaijan office approved hundreds of bills sent by Bull Equipment Trading. People familiar with the situation said Bull Equipment Trading, which has a Dubai bank account and address, is a shell company that didn’t perform the work that was billed to Panalpina. In several instances, it sent invoices for cargo-handling services at a Georgian seaport that other companies had already provided, according to the documents and people familiar with the matter.

The payments raised red flags for BP’s compliance department, which worried that money siphoned off from its logistics operation might be used to pay bribes, a former BP manager who worked on its projects in the Caucasus told the Journal.

More XPO Logistics Adjusts Guidance Off Customer Bankruptcy

“You can’t get anything through customs [in Azerbaijan] without paying off someone,” the former BP manager said. A BP representative said this doesn’t reflect the company’s position.

Azerbaijan’s State Customs Committee denied allegations of requiring bribes, and said BP and its partners have never complained about illegal fees.

In a statement, Panalpina said it was a victim of fraud “conducted by a former subcontractor and other third parties.” The company fired the manager of its Azerbaijan office after an internal investigation, but kept him on as a consultant for several months, according to a person familiar with the matter. Panalpina also reported several people to local law enforcement and is auditing its Bull Equipment Trading invoices to determine whether it owes BP, the person said.

The senior manager, Murad Housseinov, denied any wrongdoing. He said Bull Equipment Trading was a legitimate subcontractor which was regularly audited by Panalpina’s head office. He disputed that he was fired and said he resigned to take the consulting role.

In 2010, Panalpina and other companies were fined about $236 million by the U.S. Justice Department for bribing officials in seven countries, including Azerbaijan. A Panalpina spokeswoman said the company has since hired a new CEO and chairman and “successfully implemented a new corporate culture that emphasizes compliance.”

Infrastructure projects are another source of cost inflation for BP in Azerbaijan, where imports tend to be costlier than average due to the country’s landlocked position. Foreign companies often partner with state-controlled firms to gain access to construction yards and other infrastructure.

In April 2014, a consortium that included Bos Shelf LLC and Star Gulf FZCO—both controlled by Socar—and Italian firm Saipem SPM -1.43% SpA was tasked to build and install two massive supports and other infrastructure for Shah Deniz. The consortium routinely charged BP well over the market rate for basic services and key construction work, according to documents and people familiar with the matter.

In May 2014, BP’s ethics department received a complaint from Mark Campbell-Roddis, a BP contractor on the Shah Deniz project who had recently resigned. Among other things, he raised concerns about “blatant attempts at profiteering.” When the local companies came on board, “that’s when costs went through the roof,” he said.

BP has said Shah Deniz’s expansion remains on time and on budget.

Socar referred questions to its subsidiaries. Bos Shelf declined to comment on behalf of itself and Star Gulf.

Saipem is facing a court case in Italy for alleged corruption in Algeria and a separate Italian investigation into its operations in Brazil. Saipem said independent audits found no evidence of corruption in Algeria, and that an internal investigation is under way into the Brazil allegations. It said its Shah Deniz work “is delivered under a fully transparent contract structure where costs are visible to, and approved by, the client.”

Write to Robbie Whelan at robbie.whelan@wsj.com and Sarah Kent at sarah.kent@wsj.com