Before I tear your idea apart, let me say that it's a really interesting idea and it was super fun to think about.

Please continue to think outside the box and ask interesting questions!

Alright, let's do this!

Let's take a step back and ask why that baby monitor is running Linux in the first place? What if there was no operating system and the application was written in bare microcontroller code (think arduino code)? Then there would be no sudo or ls or even a shell for the attacker to use, right?

I'm not an expert here, but I expect that we, as an industry, have gravitated towards putting Linux on anything big enough to run it largely for developer convenience:

Reducing dev time: When building your WiFi-and-bluetooth-capable web-administered cloud-syncing self-patching whizpopper with companion Android and iOS apps, Linux comes with all the libraries, utilities, and drivers you need to do that. Increasing testability: If the device is running bash or busybox with an SSH port, then it's super easy to connect in and figure out what went wrong during your product testing phase.

For your obfuscation idea to work, you'd need to obfuscate not only the names of command-line utilities like sudo and ls , but also every Linux kernel API to prevent the attacker from dropping in their own compiled binary that calls the kernel directly. So let's take another look at your idea:

Implementation would be fairly easy for compilers. Take the simplest case of "rename this function and all calls to it." You could give an OS compiler and an application compiler the same randomized names and they'd be able to talk to each other.

You'll need to do this randomized compilation yourself; otherwise someone could look up the mappings on google.

So, you'll need to build the kernel from source with your "obfuscating compiler" so that only you know the mappings of the obfuscated kernel APIs. (ever built the linux kernel from source? It's certainly more of a chore than docker pull alpine , which is the direction that dev culture seems to be going).

But an operating system is more than just the kernel. You want drivers for the Broadcom BCM2837 wifi chip that comes on that mini-pc device? You'll need to build that driver against your ofbuscated kernel with your compiler, if Broadcom will even give you the source code. Than you'll need to build the entire GNU wifi and networking software stacks. How many other things will you need to find source for and add do your build pipeline before you have a functioning OS?

Oh, and if the upstream repos of any of those things issues a patch, you're now responsible for re-building it (assuming you saved the compiler obfuscation mapping files that match your kernel binary) and pushing it out to your devices because - by design - your devices can not use patch binaries produced by the vendor.

Oh, and in order to foil hackers, there'll be none of this "Here's the binary files for Whizpopper 1.4.7", oh no, you'll need to build a uniquely obfuscated version of everything from the kernel up per device that you ship.

So to your questions:

Is OS obfuscation as described used widely and I just haven't encountered it? If not used widely, what are the practical or technical barriers to usage?

I think the answer is that what you're describing pretty much completely defeats the purpose of using pre-existing software components if you need to find and build literally everything from source. It might actually be less effort to ditch the operating system entirely, pretend it's 1960, and write your application directly in CPU microcode.

I like security more than most developers, but like f* that.