---===[ Qubes Security Bulletin #51 ]===--- 2019-09-10 Insufficient validation of backup compression filter on restore Summary ======= The Qubes backup format has an option for user-selected compression filters. While Qubes backups are authenticated and encrypted, the choice of compression filter in a given backup is not further validated on restore. Under specific conditions, an attacker capable of circumventing backup authentication could exploit the lack of compression filter validation in order to execute arbitrary commands when a backup is restored. Impact ====== In Qubes OS, backups are encrypted and authenticated using a passphrase. You must enter your backup passphrase when attempting to restore a backup. If your passphrase successfully authenticates the backup, it will be decrypted and restored. Otherwise, the operation will be aborted before the contents of the backup are further processed. In order to exploit this vulnerability, an attacker would have to create a malicious backup that would successfully pass authentication and induce you to restore it. In practice, this means that the attacker would effectively have to (1) learn your backup passphrase and (2) replace one of your backups with a malicious version. If the attacker is unable to achieve either condition (or an equivalent), the vulnerability cannot be exploited. Therefore, if you have only restored backups protected by strong, secret passphrases, then you are not affected. Likewise, if you have only restored backups from trusted, secure locations, then you are not affected. However, if you have ever restored a backup with a weak or non-secret passphrase that was also stored in an untrusted location, then your system may be compromised. This bug can be exploited reliably and silently only when both conditions are met. (Note: In this context, "restoring" a backup also includes use of the Qubes "verify backup" functionality, since verification works by simulating the restore process.) Details ======= The ability to use a user-specified "compression filter" (in practice: an argument to `tar -I`/`tar --use-compress-program`) was introduced in Qubes Backup Format Version 3. [1] Alternate compression programs are specified by the user via `qvm-backup --compress-filter [...]` and are stored in the backup metadata. Originally, they were stored in the `compressed` header field, then later in the `compression-filter` field. The whole backup, including its metadata, is authenticated via an HMAC with the backup passphrase. There is no further validation of the contents of the specific `compressed` or `compression-filter` field. In order to exploit this vulnerability, an attacker would have to create or modify a backup for the user to restore. In order for the HMAC authentication to succeed, the attacker would have to know the passphrase that the user will enter when attempting to restore, e.g., by cracking the user's passphrase or somehow inducing the user to enter a passphrase provided by the attacker. Furthermore, the attacker would have to induce the user to select the attacker's malicious backup for a restore operation, e.g., by replacing one of the user's genuine backups with a malicious version (which would require that the attacker have write access to the user's backup storage location) or somehow convincing the user to restore a backup provided by the attacker. The internal structure of backups is one outer uncompressed multi-session-concatinated (via `tar -A`) tar archive. The first session contains an uncompressed and unauthenticated `backup-header`. This untrusted backup header file is initially only inspected to determine the backup archive format to determine how to proceed with further extraction. The following tar sessions contain a `backup-header.hmac` file used to authenticate the backup header before deeper inspection, and other files (qubes.xml, vmXX/{private,etc.}.img) which are each wrapped in their own tar archives and then wrapped in their own encryption and authentication. Since qubes.xml is itself contained in an inner potentially-compressed tar archive, the decompression filter is also used when verifying a backup, i.e., when the `--verify-only` flag is passed to `qvm-backup-restore` or when the "Verify backup integrity, do not restore the data" option is selected in the "Restore qubes" GUI tool. Patching ======== Note: Patching is not sufficient to recover from a compromised state. If you suspect you may have restored a malicious backup, see the next section for details and recommendations. The specific package that resolves the problems discussed in this bulletin are as follows: For Qubes 4.0: - qubes-core-admin-client version 4.0.27 The packages are to be installed in dom0 via the Qubes VM Manager or via the qubes-dom0-update command as follows: For updates from the stable repository (not immediately available): $ sudo qubes-dom0-update For updates from the security-testing repository: $ sudo qubes-dom0-update --enablerepo=qubes-dom0-security-testing These packages will migrate from the security-testing repository to the current (stable) repository over the next two weeks after being tested by the community. Securely Restoring from Backups =============================== The safest way to restore from a backup is to do the actual backup processing outside dom0. 1. Install the `qubes-core-admin-client` package in a domU. 2. Authorize the appropriate qrexec policies in the domU: - admin.vm.Create.AppVM - admin.vm.Create.TemplateVM - admin.vm.Create.StandaloneVM - include/admin-local-rwx - include/admin-global-ro 3. Use `qvm-backup-restore` in the domU. In a subsequent update, the above procedure will be automated with a new `qvm-backup-restore --paranoid-mode` option. See "Compromise recovery in Qubes OS" for details about how to use this mode. [2] Indicators of Compromise ======================== It is possible to manually inspect the header of a backup to observe whether the vulnerability has been exploited. To do so, inspect the backup as follows: 1. Verify the backup header integrity according to the "emergency backup restore without Qubes" instructions for your backup. These vary depending on the age of the backup, as the format has changed over time. [3][4][5][6] 2. Check the "compressed" and "compression-filter" header fields for anything anomalous. For example, you may see something like the following: $ tar -ivxf qubes-2019-08-06T121200 backup-header{,.hmac} backup-header backup-header.hmac $ scrypt dec backup-header.hmac backup-header.ok Please enter passphrase: backup-header!<backup-passphrase> $ cmp backup-header.ok backup-header && echo ok || echo wrong ok $ grep -E '^(compressed|compression-filter)=' backup-header | cat -v compressed=True compression-filter=gzip If you see anything other than `True` and a legitimate compression filter like `gzip` or `bzip2`, this may be a reason for suspicion. It is worth noting, however, that depending on how a malicious backup has been stored and/or transferred to the machine on which it is restored -- and depending on the sophistication of an attacker -- a previously malicious backup may have self-modified to appear benign after the fact as part of its exploit payload. Therefore, this should not be considered an infallible way to detect malicious backups. Storing the backup exclusively on immutable media throughout this process can provide further assurance. The possibility of other similar vulnerabilities cannot be completely ruled out, so restoring backups in a deprivileged manner (outside dom0, as described in the previous section) is still recommended. Credits ======= This issue was discovered and reported by Jean-Philippe Ouellet <jpo@vt.edu>, who also provided a fix, a PoC exploit, helped with mitigations for this general class of issue in the future, and wrote the initial draft of this advisory. References ========== [1] https://github.com/QubesOS/qubes-core-admin/commit/0cd8281ac10ee06f4b2fce9f86e27eb25292bc25 [2] https://www.qubes-os.org/news/2017/04/26/qubes-compromise-recovery/ [3] https://www.qubes-os.org/doc/backup-restore/ [4] https://www.qubes-os.org/doc/backup-emergency-restore-v4/ [5] https://www.qubes-os.org/doc/backup-emergency-restore-v3/ [6] https://www.qubes-os.org/doc/backup-emergency-restore-v2/ -- The Qubes Security Team https://www.qubes-os.org/security/