Retailer RadioShack has warned it may file for bankruptcy. Getty Images

Hedge-fund managers are putting a new twist on credit-default swaps, using the contracts to fortify bets on troubled companies.

The swaps, which work like insurance policies when companies default on bonds and loans, fell out of favor after Wall Street’s outsize bets on the swaps soured during the financial crisis. Now, investors are increasingly combining credit-default-swaps trades with elements of activist investing to push companies toward default in some cases and away in others.

Swaps buyers pay sellers an upfront fee and a steady flow of premiums in exchange for a guaranteed payout from the seller if default occurs. Historically, the corporate credit-default-swaps market was dominated by bets on giant borrowers with billions of dollars of debt outstanding. But that use has declined, while swaps contracts on the debt of smaller companies in financial distress has surged.

The average market capitalization of the five most actively traded nonfinancial companies in the credit-default-swaps market has been about $6 billion since March, according to data from Depository Trust & Clearing Corp. and S&P Capital IQ. That’s down from $20 billion at the end of 2013 and well below the $16 billion average since July 2010, when DTCC began collecting the data.

Swaps trades have “been more prevalent in the distressed-credit world recently,” said Michael Henkin, co-head of restructuring at investment bank Guggenheim Securities.

The trend has renewed a debate about the swaps, which are rarely disclosed by those who use them.

It is also making it harder for companies that have fallen on hard times to gauge the motives of shareholders and creditors, because they seldom know who owns credit-default swaps, or CDS, and might benefit from a default.

Investors using this strategy are attracting comparisons to activist investors, who have long pressured corporate boards to take actions that will boost the value of their stock and bond investments. But the combination of credit-default swaps and activism is a relatively new development that makes it much harder to discern investors’ true aims.

Forest Oil Corp. found itself in such a situation this year. The struggling oil company was trying to merge with a healthier competitor, Sabine Oil & Gas LLC, when it learned from some of its shareholders that investors who bought CDS betting on a default were also buying up stock with plans to vote against the deal, according to a person familiar with the matter.

To stop the CDS holders from blocking the deal, the person said, the company adopted a poison pill that prevented investors that bet against the company from amassing big stakes. Forest shareholders ultimately voted to approve the deal, which helped the company avoid a default.

Forest Oil is among a number of small companies that have seen an uptick in CDS trading. Outstanding CDS contracts on troubled consumer electronics retailer RadioShack Corp. amounted to $23.5 billion last week, dwarfing the company’s roughly $1.4 billion in total debt.

The debt of struggling department-store chain J.C. Penney Co. has credit-default-swaps contracts valued at $19.3 billion, more than double its $8.7 billion in total debt. Gambling company Caesars Entertainment Operating Co., which is negotiating a debt-restructuring plan with creditors, has $26.9 billion in CDS contracts on its debt, about $8.5 billion more than the company’s outstanding debt.

A big reason for the shift: Event-driven hedge funds manage $555 billion in assets, double what they controlled in 2008, according to HFR, but there are far fewer distressed companies to target as default rates hover near historic lows. Credit-default swaps allow hedge funds to substantially increase the sums they bet on troubled companies.

Market participants and advisers to troubled companies say they are concerned that the firms using swaps are, in some cases, gaining at the expense of the companies and their investors by either forcing defaults or dragging out the restructuring process. Hedge funds active in the swaps market declined to comment.

“It’s very difficult to know where anyone stands anymore because of CDS and because CDS investments are not disclosed,” said Stephen J. Lubben, a professor of corporate governance and business ethics at Seton Hall University School of Law in New Jersey.

The complicated role of hedge funds that provided a rescue loan to RadioShack is a case in point.

This year, BlueCrest Capital Management LLP, DW Investment Management LP and Saba Capital Management LP, run by former Deutsche Bank AG trader Boaz Weinstein, sold CDS protection on RadioShack, betting it wouldn’t default, according to people familiar with the matter. When the company ran low on cash this summer, the funds privately offered management new loans to carry it through the holiday season, the people said.

The loan pushed off a RadioShack default, and credit-default-swaps prices plummeted.

The funds, however, had already pocketed the upfront payment from the buyers of the swaps at the higher prices.

RadioShack’s survival is still in question. The company warned this month it may file for bankruptcy or even liquidate. Swaps prices have soared again, while the retailer’s debt sold for 17 cents on the dollar this week, according to MarketAxess. The rescue loan put the hedge funds among the creditors that would be repaid first in a RadioShack bankruptcy.

Critics say the loans that hedge funds make to benefit their swaps trades often benefit the lender more than the borrower.

In 2013, Blackstone Group LP’s debt-investing arm, GSO Capital Partners, bought a €100 million ($122.3 million) loan owed by Spanish gambling company Codere SA, then refused to renew the loan unless management delayed paying interest on other existing debt by a few days. The late payment triggered a payout on Codere credit-default swaps that GSO had previously purchased.

Late last year, a Blackstone spokesman said GSO “provided capital when no one else would, which allowed the company to live and fight another day.”

Codere defaulted on the GSO loan in February. The company reached a debt-restructuring deal with creditors in September.

Write to Matt Wirz at matthieu.wirz@wsj.com, Matt Jarzemsky at matthew.jarzemsky@wsj.com and Tom McGinty at tom.mcginty@wsj.com