From fitness trackers to connected headwear for soldiers on the battlefield, wearable devices stand as one of the fastest-growing segments of the tech industry.

[ Related: Consumers are buying millions and millions of wearable devices ]

But with those always-on devices come a slew of considerations for policy makers, in particular the concern that device manufacturers aren't implementing appropriate security and privacy measures.

Those worries got an airing at a recent House hearing, where industry witnesses urged lawmakers to tread lightly before developing stringent new privacy rules, while at the same time acknowledging that device and application makers need to be vigilant in how they are handling the data collected from users.

[ Related: The CSO IoT survival guide ]

Cisco's Doug Webster, vice president of service provider marketing at the networking giant, positioned wearable devices as a microcosm of the larger Internet of Things (IoT) upheaval, which he described as the next major phase in the evolution of IT.

"The next wave of innovation isn't about moving data from one place to another; it's about connecting physical objects to the Internet on an unprecedented scale. Increasingly, the <em>things</em> connected are the shirts on our backs, the glasses on our foreheads, the watches on our wrists and the jewelry around our necks," Webster said. "It's fair to say that these devices are poised to take off."

Cisco has some numbers to support that seemingly safe prediction.

The firm projects that by 2020, 600 million wearable devices will be online globally, up from 87 million last year. In North America alone, Cisco expects that there will be 180 million wearable devices online, a more than fourfold increase from last year. And those devices will only account for "a tiny trickle" of wireless activity -- estimated at just 1 percent of all mobile traffic.

[ Related: Millennials will accelerate Internet of Things action, IDC predicts ]

"Given this growth, it's important for policymakers to understand the issues affecting wearables," Webster said.

Cisco is appealing for the government to take more action to free up wireless spectrum to ensure sufficient network capacity to support the surge in mobile traffic, and to further incentivize ISPs to invest in building out their networks.

Lawmakers eye security, privacy concerns with wearable devices

But security and privacy remain a challenge. In particular, many of the small, lightweight and mass-produced devices that comprise IoT aren't conducive to robust security protections, according to Scott Peppet, a professor at the University of Colorado Law School.

"That's a technical matter," Peppet said. "Mostly, we know from research that many of these devices have not been secure in the first waves of consumer devices, for example. And the reason is obvious -- they're small, they're generally designed to be relatively inexpensive. It's hard to pack data security measures into a thing with a small processor, very little connectivity; they're hard to update because they don't talk to the Internet that frequently. And so it's been a challenge."

Witnesses also pointed to uneven protections under privacy laws. Most of the data collected through devices like the popular Fitbit and other fitness trackers, for instance, is not covered under the protections and usage dictates of the HIPAA statute that governs health information.

Then there is the general confusion of how the limits of data collection and usage are spelled out in privacy policies, often byzantine legal documents that can be written to afford broad latitude for device makers and application providers. Often times users are surprised to learn that they don't own their own data under those agreements, giving them little or no opportunity to edit or delete information from their profiles.

"There's clearly great potential here," said Rep. Jan Schakowsky (D-Ill.), quickly adding that "we need to make sure consumers fully understand what they're getting into."