Since the hit podcast “Serial” blasted into millions of earbuds more than a year ago, a wave of new shows and listeners have flooded the decade-old podcast business.

About 17% of Americans over the age of 12—around 46 million people—listen to at least one podcast each month, up from just 9% in 2008, according to estimates from Edison Research last year.

But while the popularity of podcasts is on the rise, many big-name advertisers are still wary about committing serious portions of their marketing budgets toward the medium.

Advertising revenue growth has been limited by a range of problems not unlike the issues that online media faced in its early days. It is difficult to measure how many people actually tune into a podcast, let alone listen to hosts promoting its sponsors. Ads in podcasts are relatively expensive, so it is tough for brands beyond direct-response advertisers to determine whether the investment pays off. And the process of buying and selling ads in podcasts is still complex and clunky.

That is all keeping large brand advertisers from pouring in money, leaving podcasts as a niche category.

“I certainly believe that it’s such a difficult and convoluted process to buy, manage and verify that [it] certainly has kept a lot of advertisers on the sideline,” said Stephen Smyk, chief executive of Performance Bridge, which buys podcast ads on behalf of direct-response advertisers like glasses company Warby Parker and Harry’s razors.

Advertisers are expected to spend about $35.1 million on podcasts this year, up about 2% from last year, according to ZenithOptimedia, which is owned by ad holding company Publicis. Meanwhile, marketers are still investing some $18 billion a year in radio and $67 billion in TV.

While the number of podcasts has taken off, a small group of publishers dominates the landscape. About 700 unique programs from the top 10 podcast publishers, including NPR, This American Life/Serial and WNYC, amassed 110 million downloads in the U.S. in December, according to podcast advertising firm Podtrac. Those top 10 podcast publishers account for about 40% of all monthly podcast listeners in the U.S., according to Podtrac’s estimates.

“The stand-out success of a few podcasts has tended to cannibalize spending that would otherwise have been spread across less successful podcasts, instead of expanding the market substantially,” Jonathan Barnard, head of forecasting at ZenithOptimedia, said in an email.

In fact, it is hard to know exactly how many people are listening at all. The chief metric in the podcast marketplace is unique U.S. downloads, which counts how many individuals put an episode on their device. Podcast insiders say figures can easily be skewed by counting one person’s attempt to download the same episode multiple times or failing to filter out a company’s own office IP addresses and bot traffic.

A deeper problem is that downloading an episode doesn’t necessarily mean someone listened to the podcast or the advertising within it.

“Over the last year, we’ve all seen there’s been a surge in awareness and listening to on-demand audio, buoyed by shows like ‘Serial,’” said Thomas Hjelm, chief digital officer at New York Public Radio. Yet, “there has not been a common measure for what is a listen and what is a download.”

That is compounded by the fact that the ads aren’t cheap.

Ads in top programs can command $50 to upward of $100 CPMs, which represents the cost of reaching a thousand podcast downloaders. Media buyers compare podcast pricing to very premium online video. YouTube ads sold for an average $18 CPM in December, according to data firm SQAD.

The high price contributes to the fact that most podcast advertisers are direct-response marketers like do-it-yourself website-maker Squarespace, which has spots on podcasts such as “Start Up,” “Serial,” and “99% Invisible.” The company promotes codes to save money on annual website plans, enabling it to easily track its return.

But marketers looking to build awareness for a consumer brand struggle to determine effectiveness, ad buyers say.

“Podcast ads can’t be targeted in the way other digital media can be, and there’s no immediate metric of success—like impressions served or links clicked—to allow advertisers to evaluate return on investment,” Mr. Barnard said.

Still, there are signs large brands may be testing podcasts. Matt Lieber, co-founder of podcast network Gimlet Media, which averages about five million downloads a month across shows such as “Reply All” and “Mystery Show,” said he has worked with brands like Ford, Microsoft and Sonos.

Proponents argue that the medium is particularly effective for advertisers because listeners are trusting and attentive—and the hosts often read the ads. Podcasts like “This American Life” and “WTF with Marc Maron” have fandoms bordering on cultish.

Podcasts may also get a boost from the advancement of ad technology, like the growing ability to “dynamically insert” new ads into old shows and to target based on location.

Meanwhile, New York Public Radio, NPR and other public radio producers are working together to issue new guidelines on what constitutes a unique download, in an effort to standardize measurement across the industry. The Interactive Advertising Bureau, an ad tr ade group, says it is also working to sort out measurement issues.

“It’s the Wild West in many ways for podcast measurement,” said Steve Mulder, senior director of audience insights at NPR. “If we want to unlock the full potential of podcasts, we have to ensure we have rock solid measurement that everyone agrees on and everyone can trust.”

Write to Steven Perlberg at steven.perlberg@wsj.com