Cord-cutting is on the rise, ratings for many networks are in decline, Web video consumption is surging and there’s a new crop of stars on digital media. Surely, marketers are chasing this migration of mostly young viewers by diverting huge chunks of their television advertising budgets to digital video.

But, for the most part, that’s not happening.

Facebook FB -0.90% and Snapchat get all the buzz these days with their billions of daily video views, but there’s really just one game in town when it comes to winning over any TV marketing money to the Web—YouTube.

YouTube has been one of the biggest beneficiaries of advertisers shifting spending from TV to chase viewers who have migrated to online video, but even the Google platform is siphoning away only a small fraction of marketers’ media budgets. YouTube faces several nagging challenges that are holding it back from capturing more TV revenue, including pricing concerns, the way agencies are structured and lingering doubts over the quality of its content, according to ad buyers.

YouTube’s struggles to go the extra mile offer a cautionary tale of the obstacles that could be awaiting the slew of new Web video entrants looking to capitalize on traditional TV’s much-hyped demise.

“Web-only properties are still making a case for a seat at that [TV budget] table versus sitting at the kids table,” said Adam Shlachter, president of VM1 at ad buying agency Zenith Media. “I don’t think it will change drastically this year.”

Overall spending on TV advertising is forecast to rise 0.5% to $63 billion this year, excluding the Olympics and political spending, according to the research and ad buying firm Magna Global.

Meanwhile, Web video advertising is expected to rise 28% to $9.8 billion in 2016, according to eMarketer, still a fraction of the money spent on TV. Even though advertising on Web video is growing rapidly, ad buyers say this growth is mostly not coming at the expense of TV.

“You only have to look at the revenue at a number of TV companies to realize you have not seen a wholesale transfer of dollars,” said Rob Norman, GroupM’s chief digital officer. “But of the new platforms, unlike Facebook and Snapchat, the one most likely to unlock TV dollars is YouTube.”

YouTube is the reigning king of Web video with $1.2 billion in ad revenue in 2014, or about 21% of the total spent on online video ads in the U.S. that year. This year, YouTube is expected to reel in about $2 billion, but its market share will shrink slightly to 20% as more and more companies try to get into Web video, says eMarketer. YouTube’s video ad growth, in addition to stealing some spending away from TV, is bolstered by rising digital budgets and money shifting out of print and display.

The scale of spending on digital video platforms like YouTube still pales in comparison to the TV behemoth. And, it turns out, there are several specific factors that ad buyers say are holding it back.

Ads on YouTube can cost between $15 and $20 per thousand people reached in key demographics, according to ad buyers. Those prices are higher than the average prices for commercials on cable networks, which can cost closer to $10, and comparable to prime-time ads on cable, these buyers say.

The problem with that level of pricing is that some media buyers and their ad clients don’t consider YouTube content to be of the same quality as top-tier cable programming, let alone a prime-time broadcast show.

“There is no doubt that YouTube programming in aggregate speaks to a sizable, passionate, engaged young audience,” said Kris Magel, President, U.S. at the ad buying firm Initiative, adding that YouTube competes with Comedy Central or Adult Swim for some brand’s budgets.

“That connection and engagement is worth a lot to a youth-targeted marketer,” Mr. Magel said. “But it’s not limitless. You have to price competitively to be in the consideration set.”

In some cases, YouTube never has a chance to compete for TV budgets. Some marketers still view digital media and TV as completely separate entities that warrant separate budgets. Thus, some TV ad buyers may not even have the ability to move money from a TV network to YouTube because those budgets are managed separately.

“Few advertisers have subscribed to the idea that their budgets are fully fluid,” said Dave Campanelli, senior vice president and director of national broadcast at the agency Horizon Media.

Over the past few years YouTube has—at times, reluctantly—done everything it can to make buying ads on its platform as similar to TV as possible, from the research it provides to the sheer mechanics of executing ad campaigns.

PewDiePie, aka Felix Kjellberg, is a big YouTube star Photo: YouTube

One of the main arguments for advertising on YouTube is the platform’s established legitimacy as an entertainment destination for young people. YouTube, which has birthed a generation of creators who for many consumers are bigger than movie and TV stars, reaches a billion people each month on average.

According to Google, Nielsen recently found that the service reached more people between the ages of 18 and 49 on mobile devices alone than any TV network in December of last year, while comScore found that 44% of YouTube viewers in that age bracket don’t watch prime-time broadcast TV in an average week.

As a result, Tara Levy, YouTube’s managing director, agency sales, said YouTube is in conversations with many advertisers well in advance of the TV upfront selling season, a marked difference from recent years. The upfront negotiating period, when many TV networks try to lock in ad commitments in advance, kicks off in earnest in a few weeks.

“Everything is just down (in TV),” said Ed McLoughlin, global head of media and digital at HP. Mr. McLoughlin said that the majority of HP’s video ad budgets are now directed toward digital media, and half of the company’s digital media spending is on YouTube. “I see that number going up,” he said.

While YouTube’s experience offers some lessons for digital video rivals, companies like Facebook and Snapchat are still seen as behind YouTube when it comes to winning over slices of those TV ad budgets, despite each boasting of delivering 8 billion video views each day and viral moments like 800,000 people watching an exploding watermelon live on Facebook.

Some of that has to do to with the fact that YouTube has simply been at this longer. Facebook and Snapchat have seen their video numbers explode in the past few years.

But ad buyers say they fundamentally view the way video is consumed on those platforms differently. While YouTube theoretically allows an advertiser to run its ads alongside popular content or talent (it feels a lot more similar to running ads on TV networks and shows), it’s worth pointing out that a huge portion of the video views on Facebook and Snapchat are from user-produced videos that carry no advertising.

And pushing an ad into someone’s social media feed is seen as potentially more intrusive for some buyers compared with TV or YouTube where users expect to see ads alongside video content.

“The more personal an experience, the harder it is to bring in ad messaging in an acceptable way,” said John Swift, Omnicom Media Group’s chief executive of North American investment. “It doesn’t mean the potential isn’t huge.”

It’s also harder to compare Web video ads with TV ads from a pricing and effectiveness perspective, say buyers. While all of these large Web outlets have taken steps to enable tracking from third-party researchers like Nielsen, advertisers say it’s still difficult to make apples-to-apples comparisons on basics like how many people will a Snapchat or Twitter campaign reach versus one on TV.

Many major marketers also have decades of evidence showing how TV ads impact sales. This is something YouTube has just started tracking.

“Measurement is still a challenge,” said Christina Norman, chief executive at the media agency Media Storm. “You need to be able to go to your client or boss and say, I spent X money on this platform and got Y back.”

Snapchat, in particular, is still seen by many advertisers as experimental, and its very young user base isn’t crucial for every brand. Snapchat’s ad prices are also in the neighborhood of premium cable, which makes experimentation challenging for cautious brands.

Facebook’s ad prices are reasonable on the surface, said Mike Law, executive vice president and managing director of video investment at the agency Amplifi U.S. Facebook counts video views after a person watches for just three seconds, though marketers can elect to pay only for video ads that are viewed for at least 10 seconds.

Regardless, ad buyers often make their own calculations based on how many people watch video ads in their entirety. Based on those calculations, Facebook’s prices can look even higher than prime-time TV, he said. “We view them less as a TV replacement,” Mr. Law said.

Even with TV ratings not what they used to be, marketers are finding that replacing TV isn't an easy task.

Write to Mike Shields at mike.shields@wsj.com