Advertising automation firm Rubicon Project Inc. is exploring strategic options, including a potential sale, according to people familiar with the matter.

The Los Angeles-based company, which offers automated online systems that give marketers and publishers the ability to buy and sell advertising in real-time auctions, is working with investment bank Morgan Stanley on the process, the people said.

Rubicon Project, which went public in 2014 with Morgan Stanley as one of its lead underwriters, has a market capitalization of about $375 million. The company is exploring multiple options and there is no guarantee it will ultimately do any deal, one of the people said.

“We do not comment on inquiries of this kind,” said a spokesman for the company.

Shares of Rubicon Project jumped 11% to $8.46 in late morning trading.

News of a possible sale of Rubicon Project comes as the ad tech industry continues to navigate through turbulent times. Ad-tech companies that soared after initial public offerings in recent years have taken a hit as the market has become increasingly commoditized. Investors have also grown wary of the increasing dominance of the online ad market by Facebook Inc. and Google.

Some firms in the sector have experienced sinking stock prices and layoffs. Rocket Fuel, for example, announced its second round of layoffs last week, saying it was letting go 11% of its staff as part of a reorganization of its business.

Rubicon Project hasn’t been immune. The company, whose share price declined almost 50% over the past 12 months, has been stung lately by disappointing business results and the departure of some of its top executives, including Neal Richter, the company’s chief technology officer, who left in November.

The company’s financial performance has been hurt by several factors, including its slow response to so-called “header bidding,” which is technology that allows publishers to get better pricing by being able to have multiple buyers bid on ad inventory at the same time. Rubicon Project shares fell more than 30% in August after the company attributed a decline in desktop ad revenue to a delay in recognizing customers’ desire for header bidding solutions.

When the company reported third-quarter earnings results in November, Rubicon Project lowered its full-year guidance and announced it was shedding almost 20% of its workforce, roughly 125 employees, as part of a reorganization.

“Header bidding is certainly a short-term challenge,” Frank Addante, Rubicon Project’s chief executive officer, said on a conference call with analysts at the time. “We’ve got some work to do on this. We’re working through our plans. This is not an overnight shift.”

For the nine months ended in September, Rubicon Project said revenue rose 33% to $205 million. Net income for the period was $3.1 million, compared with a loss a year earlier.

News Corp, which publishes The Wall Street Journal, owns an almost 10% stake in Rubicon Project. In September, News Corp invested $10 million in Rubicon Project rival AppNexus.

Despite the problems in the sector, the ad-tech space has had some good news lately with several companies getting acquired, including video ad buying software company TubeMogul, which was bought by Adobe for $540 million late last year.

AppNexus has confidentially filed paperwork for its long-expected initial public offering that could happen during the second quarter. The company could seek a valuation between $1.5 billion and $2 billion, The Wall Street Journal has reported.

Write to Suzanne Vranica at suzanne.vranica@wsj.com and Dana Mattioli at dana.mattioli@wsj.com