It is time mention about simple sequential or simultaneous upgrade process on lxd guests. It is adapted to Debian-like operating systems using apt as I am using these on daily basis.

Distribution-specific information.

$ lsb_release -a

No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 16.04.3 LTS Release: 16.04 Codename: xenial

The container hypervisor client / daemon version.

$ lxc --version

2.0.10

As you can see the following operations are performed on the Ubuntu 16.04 using the container hypervisor lxc 2.0.10 .

Upgrade guests sequentially

Update package index.

for guest in $(sudo lxc list -c ns | awk 'BEGIN{FS="|"} NR%2==0 && NR>2 && $3 ~ "RUNNING" {print $2}'); do sudo lxc exec $guest -- /bin/bash -c "echo $guest; apt-get update; echo;"; done | tee /tmp/guest_update.log

Inspect the /tmp/guest_update.log log file.

$ tail /tmp/guest_update.log

[...] dokuwiki-tests Hit:1 http://security.ubuntu.com/ubuntu xenial-security InRelease Hit:2 http://archive.ubuntu.com/ubuntu xenial InRelease Hit:3 http://archive.ubuntu.com/ubuntu xenial-updates InRelease Hit:4 http://archive.ubuntu.com/ubuntu xenial-backports InRelease Reading package lists... [...]

This step is purely optional, but you can list packages to be upgraded.

for guest in $(sudo lxc list -c ns | awk 'BEGIN{FS="|"} NR%2==0 && NR>2 && $3 ~ "RUNNING" {print $2}'); do sudo lxc exec $guest -- /bin/bash -c "echo $guest; apt-get --dry-run dist-upgrade | grep ^Inst | cut -d\ -f2 | sort | tr '

' '\ '; echo;echo;"; done | tee /tmp/guest_upgrade_list.log

Inspect the /tmp/guest_upgrade_list.log log file.

$ tail /tmp/guest_upgrade_list.log

[...] dokuwiki-tests apt apt-transport-https apt-utils base-files binutils bsdutils btrfs-tools cloud-init cloud-initramfs-copymods cloud-initramfs-dyn-netconf coreutils iproute2 isc-dhcp-client isc-dhcp-common kmod less libapt-inst2.0 libapt-pkg5.0 libblkid1 libdrm2 libfdisk1 libkmod2 liblxc1 libmount1 libpam-systemd libphp7.0-embed libsmartcols1 libssl1.0.0 libsystemd0 libudev1 libuuid1 linux-libc-dev lxc-common lxcfs lxd lxd-client mount open-iscsi openssl overlayroot php7.0-common php7.0-intl php7.0-phpdbg php7.0-readline php7.0-sqlite3 php7.0-xml python3-distupgrade python3-software-properties python3-update-manager snap-confine snapd software-properties-common sudo systemd systemd-sysv ubuntu-core-launcher ubuntu-release-upgrader-core udev unattended-upgrades update-manager-core util-linux uuid-runtime vlan [...]

Perform upgrade process.

for guest in $(sudo lxc list -c ns | awk 'BEGIN{FS="|"} NR%2==0 && NR>2 && $3 ~ "RUNNING" {print $2}'); do sudo lxc exec $guest -- /bin/bash -c "export DEBIAN_FRONTEND=noninteractive; echo $guest; apt-get upgrade -y; echo;"; done | tee /tmp/guest_upgrade.log

Inspect the /tmp/guest_upgrade.log log file.

$ tail /tmp/guest_upgrade.log

[...] dokuwiki-tests [...] Reading package lists... Building dependency tree... Reading state information... Calculating upgrade... The following package was automatically installed and is no longer required: snap-confine Use 'apt autoremove' to remove it. The following packages will be upgraded: apt apt-transport-https apt-utils base-files binutils bsdutils btrfs-tools cloud-init cloud-initramfs-copymods cloud-initramfs-dyn-netconf coreutils iproute2 isc-dhcp-client isc-dhcp-common kmod less libapt-inst2.0 libapt-pkg5.0 libblkid1 libdrm2 libfdisk1 libkmod2 liblxc1 libmount1 libpam-systemd libphp7.0-embed libsmartcols1 libssl1.0.0 libsystemd0 libudev1 libuuid1 linux-libc-dev lxc-common lxcfs lxd lxd-client mount open-iscsi openssl overlayroot php7.0-common php7.0-intl php7.0-phpdbg php7.0-readline php7.0-sqlite3 php7.0-xml python3-distupgrade python3-software-properties python3-update-manager snap-confine snapd software-properties-common sudo systemd systemd-sysv ubuntu-core-launcher ubuntu-release-upgrader-core udev unattended-upgrades update-manager-core util-linux uuid-runtime vlan 63 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. Need to get 36.6 MB of archives. After this operation, 1794 kB of additional disk space will be used. Get:1 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 base-files amd64 9.4ubuntu4.5 [68.4 kB] Get:2 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 bsdutils amd64 1:2.27.1-6ubuntu3.3 [52.0 kB] Get:3 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 coreutils amd64 8.25-2ubuntu3~16.04 [1174 kB] Get:4 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 util-linux amd64 2.27.1-6ubuntu3.3 [849 kB] [...] Hint: Some lines were ellipsized, use -l to show in full. Setting up snapd (2.25) ... Installing new version of config file /etc/apparmor.d/usr.lib.snapd.snap-confine.real ... Setting up ubuntu-core-launcher (2.25) ... Setting up snap-confine (2.25) ... Setting up iproute2 (4.3.0-1ubuntu3.16.04.1) ... Setting up isc-dhcp-client (4.3.3-5ubuntu12.7) ... Setting up isc-dhcp-common (4.3.3-5ubuntu12.7) ... [...]

You can use additional conditions inside awk program line to match only specific guests.

Upgrade guests in parallel

You can speed up a whole process by using the amazing parallel utility.

$ sudo apt-get install parallel

Update package index simultaneously on four guests.

sudo lxc list -c ns | \ awk 'BEGIN{FS="|"} NR%2==0 && NR>2 && $3 ~ "RUNNING" {print $2}' | \ tr -d ' ' | \ parallel --jobs 4 --progress --no-run-if-empty \ --results /tmp/parallel_update/ --joblog /tmp/parallel_update_status.log \ sudo lxc exec {1} -- apt-get update

Perform upgrade process simultaneously on four guests.

export DEBIAN_FRONTEND="noninteractive"; \ sudo lxc list -c ns | \ awk 'BEGIN{FS="|"} NR%2==0 && NR>2 && $3 ~ "RUNNING" {print $2}' | \ tr -d ' ' | \ parallel --jobs 4 --progress --no-run-if-empty \ --results /tmp/parallel_upgrade/ --joblog /tmp/parallel_upgrade_status.log \ --env DEBIAN_FRONTEND \ sudo lxc exec {1} -- apt-get upgrade -y

List of the executed jobs will be stored in /tmp/parallel_update_status.log and /tmp/parallel_update_status.log log files.

$ cat /tmp/parallel_update_status.log

Seq Host Starttime JobRuntime Send Receive Exitval Signal Command 1 : 1501795764.764 4.626 0 0 0 0 sudo lxc exec dokuwiki-tests -- apt-get update 4 : 1501795764.793 4.599 0 0 0 0 sudo lxc exec jira-tests -- apt-get update 5 : 1501795769.392 4.840 0 0 0 0 sudo lxc exec nextcloud-tests -- apt-get update 6 : 1501795769.395 4.841 0 0 0 0 sudo lxc exec wallabag-tests -- apt-get update 7 : 1501795774.236 4.055 0 0 0 0 sudo lxc exec wekan-tests -- apt-get update 2 : 1501795764.767 20.569 0 0 0 0 sudo lxc exec draw-tests -- apt-get update 3 : 1501795764.775 20.561 0 0 0 0 sudo lxc exec foswiki-tests -- apt-get update

Detailed logs will be located in the specified /tmp/parallel_update/ and /tmp/parallel_upgrade/ directories.

$ cat /tmp/parallel_update/1/jira-tests/std{err,out}

Get:1 http://security.ubuntu.com/ubuntu xenial-security InRelease [102 kB] Hit:2 http://archive.ubuntu.com/ubuntu xenial InRelease Get:3 http://archive.ubuntu.com/ubuntu xenial-updates InRelease [102 kB] Get:4 http://archive.ubuntu.com/ubuntu xenial-backports InRelease [102 kB] Fetched 306 kB in 1s (239 kB/s) Reading package lists...