As the popularity of smart speakers has risen — with Amazon Echo and Google Home leading the way and Apple's Homepod following — so have concerns about privacy.

To the horror of consumers, researchers and security experts in recent weeks have found ways to hack into and control these speakers' voice assistants with methods including undetectable audio commands, eavesdropping software and targeting devices connected on a network.

The companies say they have solved many of these vulnerabilities already, according to reports, but it's hard not to wonder: How likely is it your smart speaker could be hacked by a bad guy?

Chances are actually pretty low for the average person, according to Jake Williams. He's the founder and president of cybersecurity firm Rendition Infosec, and was formerly a hacker at the National Security Agency for the Department of Defense, where he analyzed the vulnerability of U.S. systems.

"What we're talking about here is a lot of work," Williams tells CNBC Make It of hacking into a smart speaker. "[Would-be attackers] don't care what you're talking about at home, they're looking to monetize data."

That means the work it takes for hackers to listen in often isn't worth the information they might get, he says. It's much more likely a scammer would target your banking information than try to take control of your smart speaker.

"The level of effort to do it is too high in the vast majority of cases," Williams explains. "Your average American just isn't that interesting."

Also, the way smart speakers operate makes them less vulnerable to hacks than your other internet-connected devices, like laptops and smart phones, he says.

"Unlike your laptop — where you install software and you run a lot of complicated programs there, and you have a web browser that has to interpret Javascript and HTML and Flash and ridiculous amounts of processing — ... if you think about your Echo or your always-on assistant, it really only takes input from two places," Williams explains. "It takes [in] you, with your voice, and then it's a stream coming back from the server [at the company.]"

That means hackers only have two options to get at your information: infiltrating the stream to Amazon and Google's servers or your voice communications to the device.

"If you compare that to a web browser, if you think about all of the different inputs to that web browser, all the different millions and millions of ways that I can manipulate that HTML and Javascript and Flash ... your [Echo], your always-on device, your Google Home, the attack surface on them is very, very small, and the way they interact with users makes them very difficult to hack remotely."

Even if you're using your device to make purchases, Williams adds, your credit card information isn't stored on the smart speaker but accessed through the servers of Google or Amazon, meaning it would require a vulnerability of the company's system at large for that information to be hacked.

"It's possible that one of these interfaces ... could itself have vulnerabilities, but that wouldn't be a vulnerability in the Echo, that would be a vulnerability in Amazon's infrastructure, and we wouldn't call that an Echo hack," he says.

For its part, Amazon says security is a high priority.

"Amazon takes customer security seriously and we have full teams dedicated to ensuring the safety and security of our products. We have taken measures to make Echo secure. These include disallowing third party application installation on the device, rigorous security reviews, secure software development requirements and encryption of communication between Echo, the Alexa App and Amazon servers," an Amazon spokesperson told CNBC Make It in a statement. Google could not be reached, and Apple declined to comment for the article.

Amazon and Google say their assistants only start recording after they have heard a wake word, like "Ok Google" or "Hey Alexa," but some smart speaker users maintain privacy concerns.

Fears that always-listening smart speakers record more than just questions and requests were heightened in May when a family in Portland found out a spontaneous recording of a conversation in their home was texted to a random contact.

Amazon told CNBC the event was caused by a misinterpretation: "Echo woke up due to a word in background conversation sounding like 'Alexa.' Then, the subsequent conversation was heard as a 'send message' request," Amazon said in a May statement. "As unlikely as this string of events is, we are evaluating options to make this case even less likely."

But, there are still other data privacy concerns. New "phishing" scams from bad actors looking to lure users into sharing their personal information could soon arise, Ars Technica reported in August.

An Alexa phishing scam might work like this: When a user asks Alexa to open up a "skill" — a software function created by a third party — from a brand they trust, the user would be tricked into opening a similar sounding skill created by the scammers. For example, "a skill called 'Am Express' could be used to hijack initial requests for American Express' Amex skill — and steal users' credentials," the publication reported.

And in the future, companies such as Amazon and Google may have plans to use more of the audio they are privy to. The New York Times reported in March both companies have filed patent applications for technology that might someday gather audio "used to identify a person's desires or interests, which could be mined for ads and product recommendations."

To see everything Amazon's Alexa has recorded about you, check out CNBC's explainer here. (You can also see what information has been collected about you by Google and by Facebook.)

While it is important to be conscious about your digital privacy, Williams says there are lots of areas you're better off securing before worrying about your smart speaker's vulnerabilities — at least when it comes to spying threats from hackers.

"If you really think you've checked the box on everything else — your phone is secure, you've got full encryption on your phone, you've got full encryption on your laptop, you've got endpoint security every place, you have two-factor authentication — if all that's done, then worry about Alexa," Williams says. "Until then, spend your time worrying about literally everything else."

Don't miss: Here's what people actually ask Amazon's Alexa

Like this story? Subscribe to CNBC Make It on YouTube!