For a project at my current workplace, I had to modify the firmware image of a Ubiquiti device (an EdgeRouter to be precise. Pretty nice device considering its price, worth trying!).

The firmware image itself is just a .tar file which after being unarchived (it’s not actually gzipped or anything, just tar’d) will output a bunch of files. The one I was interested in is squashfs.tmp, which, as the name suggests, is the squashed filesystem for the router. I did unsquash it by just running the unsquashfs command, modified the files I was interested in modifying, mksquashfs again, generate the md5 checksum (there is a file to verify integrity of the firmware image, called squashfs.tmp.md5 which essentially holds an md5 of the squashfs.tmp file, quite simple) and tar it again. After that, just upload to the router, wait for it to update and happy days! -> WRONG! So turns out that the router upgraded successfully to the new image, but when trying to enter into the ‘configure’ mode (as you may find in a Cisco or Juniper device) it would complain with a message saying:

Failed to set up config session

What?? I hadn’t seen that message before. My first idea was to do a ‘sudo su’ on the router (it’s essentially running a debian and you’re free to mess as much as you want; thanks ubiquiti!) and try running ‘configure’ again and this time it did not complain. Mhhh… looks like a permission issue then… Next step, quick Google search (saves a ton of time usually) and I found this post at Ubiquiti forums: Building custom image for EdgeMax Really helpful, because as you may be able to see at the end of the post (just before my comment to help on the poor souls having the same issue) somebody noticed that this was in fact an issue related to extended file attributes, not being properly preserved. Knowing that, I decided to check If that was really the issue. Mounting the squashfs.tmp file directly gave this result:

# mount -o loop squashfs.tmp/tmp/router-fs-test/ # getcap /tmp/router-fs-test/usr/sbin/ubnt-util # /tmp/router-fs-test/usr/sbin/ubnt-util = cap_net_raw,cap_sys_admin+ep

But then, when unsquashing the filesystem and running getcap, guess what happened?

# unsquashfs squashfs.tmp # getcap squashfs-root/usr/sbin/ubnt-util #

Nothing! Nada! HA! So the issue happens to be when we unsquash the filesystem. I made sure to have the options enabled to preserve the xattr of files, which btw are enabled by default on Linux (be careful Mac users, they’re not there by default on Mac OS X builds of squashfs-tools), so clearly the issue was somewhere else. Some more google searching led to this: squashfs-tools: unsquashfs not preserving file capabilities Oh, that may be why… so there is a bug on the unsquashfs, at least on 4.2. I went straight to the squashfs site and got the latest package (4.3 (2014/05/12)) and unsquashed the filesystem again after compiling the tools. Still no luck, so I guess the bug is still there on 4.3. Pretty easy, just edit the file squashfs-tools/unsquashfs.c and find the function:

<p class="p1"><span class="s1">int</span><span class="s2"> set_attributes(</span><span class="s1">char</span><span class="s2"> *pathname, </span><span class="s1">int</span><span class="s2"> mode, uid_t uid, gid_t guid, </span><span class="s1">time_t</span><span class="s2"> time, </span><span class="s1">unsigned</span> <span class="s1">int</span><span class="s2"> xattr, </span><span class="s1">unsigned</span> <span class="s1">int</span><span class="s2"> set_mode)</span></p>

<p class="p1">

(Should be somewhere around line 800).

Remove the call to

</p>

<p class="p1"><span class="s1">write_xattr(pathname, xattr);</span></p>

<p class="p1">

from the top of the function and place it just before the return TRUE statement of the function. Compile again the unsquashfs tool and done! Now the file capabilities are correctly