Introduction

The LXD team is very excited to announce the release of LXD 3.20!

We purposefully accelerated our normal monthly release cycle to just two weeks as a way to quickly integrate the many enhancements and bugfixes that followed the release of LXD 3.19. We intend to keep up this pace until the release of LXD 4.0 at which point we’ll resume the monthly cadence.

This release includes 3 features/improvements contributed by students at the University of Texas in Austin:

Server side support of API collections

New unix-hotplug device type

Reworked background process management

On top of integrating those, we’ve also been focusing on filling in some of the current gaps in our VM story, fixing many bugs since 3.19 came out and now completing its network interface handling, adding support for ppc64le and support for boot device ordering.

Enjoy!

Highlights

Server side support of API collections

As our users are dealing with increasingly larger sets of instances and images, filtering all those records on the client side can become quite expensive. With this release, we’re now adding the infrastructure and initial implementation of server side filtering.

This looks like:

stgraber@castiana:~/data/code/lxc/lxd (lxc/master)$ lxc query '/1.0/instances?filter=config.image.os%20eq%20ubuntu' [ "/1.0/instances/snapcraft", "/1.0/instances/ups-monitor", "/1.0/instances/v1", "/1.0/instances/maas01", "/1.0/instances/steam", "/1.0/instances/lxd-build" ]

This is using config.image.os eq ubuntu as a filter using URL encoding. More details on the filtering options can be found here.

New unix-hotplug device type

This device type is a bit of a cross between usb and unix-char and unix-block .

It allows specifying a specific vendorid/productid and have any resulting unix-char/unix-block devices be automatically passed to the container.

An example with a USB drive:

stgraber@castiana:~$ lxc config device add c1 kingston unix-hotplug vendorid=0951 productid=1666 Device kingston added to c1 stgraber@castiana:~$ lxc exec c1 bash root@c1:~# ls -lh /dev/ total 0 crw--w---- 1 root tty 136, 0 Jan 30 23:00 console lrwxrwxrwx 1 root root 11 Jan 30 22:59 core -> /proc/kcore lrwxrwxrwx 1 root root 13 Jan 30 22:59 fd -> /proc/self/fd crw-rw-rw- 1 nobody nogroup 1, 7 Jan 13 03:59 full crw-rw-rw- 1 nobody nogroup 10, 229 Jan 30 22:59 fuse lrwxrwxrwx 1 root root 25 Jan 30 22:59 initctl -> /run/systemd/initctl/fifo lrwxrwxrwx 1 root root 28 Jan 30 22:59 log -> /run/systemd/journal/dev-log drwxr-xr-x 2 nobody nogroup 60 Jan 30 22:46 lxd drwxrwxrwt 2 nobody nogroup 40 Jan 13 03:59 mqueue drwxr-xr-x 2 root root 60 Jan 30 22:59 net crw-rw-rw- 1 nobody nogroup 1, 3 Jan 13 03:59 null crw-rw-rw- 1 root root 5, 2 Jan 30 22:59 ptmx drwxr-xr-x 2 root root 0 Jan 30 22:59 pts crw-rw-rw- 1 nobody nogroup 1, 8 Jan 13 03:59 random drwxrwxrwt 2 root root 40 Jan 30 22:59 shm lrwxrwxrwx 1 root root 15 Jan 30 22:59 stderr -> /proc/self/fd/2 lrwxrwxrwx 1 root root 15 Jan 30 22:59 stdin -> /proc/self/fd/0 lrwxrwxrwx 1 root root 15 Jan 30 22:59 stdout -> /proc/self/fd/1 crw-rw-rw- 1 nobody nogroup 5, 0 Jan 30 21:23 tty crw-rw-rw- 1 nobody nogroup 1, 9 Jan 13 03:59 urandom crw-rw-rw- 1 nobody nogroup 1, 5 Jan 13 03:59 zero root@c1:~# ls -lh /dev/ total 1.0K drwxr-xr-x 3 root root 60 Jan 30 23:01 bus crw--w---- 1 root tty 136, 0 Jan 30 23:00 console lrwxrwxrwx 1 root root 11 Jan 30 22:59 core -> /proc/kcore lrwxrwxrwx 1 root root 13 Jan 30 22:59 fd -> /proc/self/fd crw-rw-rw- 1 nobody nogroup 1, 7 Jan 13 03:59 full crw-rw-rw- 1 nobody nogroup 10, 229 Jan 30 22:59 fuse lrwxrwxrwx 1 root root 25 Jan 30 22:59 initctl -> /run/systemd/initctl/fifo lrwxrwxrwx 1 root root 28 Jan 30 22:59 log -> /run/systemd/journal/dev-log drwxr-xr-x 2 nobody nogroup 60 Jan 30 22:46 lxd drwxrwxrwt 2 nobody nogroup 40 Jan 13 03:59 mqueue drwxr-xr-x 2 root root 60 Jan 30 22:59 net crw-rw-rw- 1 nobody nogroup 1, 3 Jan 13 03:59 null crw-rw-rw- 1 root root 5, 2 Jan 30 22:59 ptmx drwxr-xr-x 2 root root 0 Jan 30 22:59 pts crw-rw-rw- 1 nobody nogroup 1, 8 Jan 13 03:59 random brw-rw---- 1 root root 8, 0 Jan 30 23:01 sda brw-rw---- 1 root root 8, 1 Jan 30 23:01 sda1 drwxrwxrwt 2 root root 40 Jan 30 22:59 shm lrwxrwxrwx 1 root root 15 Jan 30 22:59 stderr -> /proc/self/fd/2 lrwxrwxrwx 1 root root 15 Jan 30 22:59 stdin -> /proc/self/fd/0 lrwxrwxrwx 1 root root 15 Jan 30 22:59 stdout -> /proc/self/fd/1 crw-rw-rw- 1 nobody nogroup 5, 0 Jan 30 21:23 tty crw-rw-rw- 1 nobody nogroup 1, 9 Jan 13 03:59 urandom crw-rw-rw- 1 nobody nogroup 1, 5 Jan 13 03:59 zero root@c1:~# exit

Above you’ll notice sda and sda1 showing up when I plug my USB key.

And with a yubikey:

stgraber@castiana:~$ lxc config device add c1 yubikey unix-hotplug vendorid=1050 Device yubikey added to c1 stgraber@castiana:~$ lxc exec c1 bash root@c1:~# ykman info Device type: YubiKey 5C Serial number: 11576019 Firmware version: 5.2.4 Form factor: Keychain (USB-C) Enabled USB interfaces: FIDO+CCID Applications OTP Disabled FIDO U2F Enabled OpenPGP Enabled PIV Enabled OATH Disabled FIDO2 Enabled root@c1:~#

Support for standby cluster members

The dqlite-based cluster database logic was extended to make use of newly supported standby and idle roles in dqlite.

With this change in place, all cluster members will now be integrated with dqlite, they will then each hold one of the following three roles:

voting member

standby member

idle member

The configuration is for 3 voting members, followed by 2 standby members with the rest of the members being idle. In the event of a loss of a voting member, after a short timeout a non-voting member will get auto-promoted to voting. Idle members will be promoted to non-voting as needed to keep the set number of voting and standby members.

The end result is a much more resilient LXD cluster which will better handle cluster members going offline for maintenance or power failure.

VM: Extended NIC support

With this release, nic devices for virtual machines now support:

macvlan (using macvtap )

) sriov

physical

p2p

Both SR-IOV and physical NIC passthrough rely on PCI passthrough to the virtual machines. This requires a properly configured IOMMU setup as well as adequate PCI setup.

VM: boot priority

A new boot.priority option is now present on all disk and nic type devices for use with virtual machines. The higher the priority, the more likely the VM is to boot from the device.

This is of particular use when one wants a VM to always boot from the network rather than use the default behavior of booting from its root disk.

VM: ppc64le host support

Initial support for ppc64le hosts has been added. This behaves in much the same way as x86_64 and aarch64 except for the fact that a regular PCI bus is used rather than PCIe and the firmware is SLOF rather than UEFI.

Complete changelog

Here is a complete list of all changes in this release:

Full commit list Fix typo

Add role column to raft_nodes table

Make db.RaftNode an alias for dqlite.NodeInfo, which has a Role field

lxd/cluster: rename raft.go to info.go

Drop legacy newRaft() function, leftover from the hashicorp/raft time

Replace raftAddressProvider intefrace with a simple raftAddress() method

Gateway.currentRaftNodes(): return only voting nodes

Use the ID from the cluster nodes table as raft ID

Make RaftNodesReplace() skip non-voters

Include role when exchanging nodes in join/rebalance internal APIs

Always join the dqlite cluster, possibly as non-voter

Attempt to probe to a member which is considered offline

Connect to the target node before spanwing the image replication goroutine

Only contact voter nodes when searching for the cluster leader

Configure the dqlite client store to only connect to voter nodes

Load the role column of raft_nodes in db.RaftNodes()

Store the role of a node in RaftNodesReplace

Only start the dqlite engine for voter nodes

Include role information in heartbeat messages

Return also non-voter nodes in currentRaftNodes

Fix ineffective heartbeat upon join

Drop unused target variable in Join/Promote/Leave cluster functions

Start the dqlite engine also on non-voter nodes

Add cluster.Handover which finds a node eligible to become voter

Extract logic to POST a promote request into a new changeMemberRole() function

Attempt to transfer leadership to another member when shutting down

Add /internal/cluster/handover endpoint to transfer voter role

Add handoverMemberRole() helper to transfer the responsibilities of a member

Update gateway’s identify info upon role change

Rename cluster.Promote to cluster.Assign

Rename /internal/cluster/promote to /internal/cluster/assign

Remove database role when demoting

When a voter is shutdown, handover the role to another member

Assign up to 2 StandBy roles beyond the initial 3 voters

Return only voter nodes in cluster.ListDatabaseNodes()

When leaving, use currentRaftNodes() instead of querying the raft_nodes table

Redirect member delete requests to the leader

Serialize membership-related requests on the leader

Drop clusterRebalance helper, since it is used only once

Drop ineffective post-join heartbeat

Add rebalanceMemberRoles() helper and use it in the delete member API handler

Drop unused tryClusterRebalance()

Test shutting down two members concurrently

Close dqlite clients after use

Automatically demote offline nodes when running cluster.Rebalance

Automatically promote spare nodes if a voter goes offline

Assign roles to members not part of the raft configuration

Export MaxVoters and MaxStandBys

Trigger rebalance also if there are not enough voters or standbys

Downgrade rebalance error to warning, it should not block node removal

lxd/container: Improves error messages in instanceValidDevices

lxd/container: instance.ValidDevices usage

lxd/container/lxc: instance.ValidDevices usage

lxd/device/config/devices: Improves error messages

lxd/device/disk: Adds support for VM disk devices

lxd/instance/instance/interface: Comment ending consistency

lxd/instance/qemu/vm/qemu: Fixes driver index loop bug

lxd/instance/instance/utils: Introduces constant to indicate profile validation in instance name

lxd/profiles: Switches to use instance.ProfileValidationName during profile validation

lxd/device/disk: Updates Stop device to understand VM disks

Make cluster.Rebalance fail immediately if not leader

Export cluster.ErrNotLeader

Silence warning about failing to rebalance when not leader

lxd/storage/drivers/driver/common: Removes generic vfs functions as not common to all driver types

Re-enable clustering upgrade test

lxd/storage/drivers/generic/vfs: Moves generic VFS drivers into standalone file

lxd/storage/drivers: Updates usage of generic VFS functions

Add upgrade test for an 8-member cluster

Make upgrade notifications more robust

Wait for leadership to settle before running lxc cluster list

lxd/instance: Moves vm qemu pkg into instance/drivers pkg

lxd/instance/drivers/container/lxc: Adds placeholder for future lxc implementation

lxd/instance/drivers/load: Adds instance load functions

lxd/container: Removes unused functions

lxd/db/containers: Renames ContainerToArgs to InstanceToArgs

lxd/container: db.InstanceToArgs usage

lxd/profiles/utils: db.InstanceToArgs usage

lxd/profiles/utils: Updates use of containerLXCInstantiate

lxd/container/lxc: Push containerLXC load functions into instance/drivers package

lxd/container/lxc: containerLXCInstantiate usage

lxd/container/lxc: Makes containerLXCInstantiate compatiable with generic instance load functions

lxd/containers: instance.Load usage

lxd/containers/post: instance.Load usage

lxd/instance/drivers/vm/qemu: Unexport and rename load functions

lxd/instance/instance/utils: Load function comments

lxd/instance/instance/utils: Adds Create instance function placeholder

lxd/instance/instance/utils: db.InstanceToArgs usage

lxd/instance/drivers/vm/qemu: Unexports qemu implementation

lxd/exec: Pass full req through

lxd/exec: Forward control messages

lxd/containers: Fix error handling on stop

lxd/vm: Fix stop race condition

lxd/vm: Add locking for stop and shutdown

lxd/vm: Don’t crash on vm-initiated reboots

lxd/storage: Remove legacy volume.size check

lxc/init: Consider image type for instance type

i18n: Update translation templates

tests: Update volume.size tests

lxd/vm: Store qemu log

[Makefiles] Whitelist ldflags in libcap pkgconfig

lxd/vm: Fix incorrect bootindex

lxd/vm: Implement snapshot restore

lxd/instance: Move LoadAllInternal

lxd/vm: Implement Snapshots

lxd/storage/drivers/utils: Updates ensureVolumeBlockFile to use minimum block boundary size of 8192 bytes

lxd/storage/drivers/driver/lvm/utils: Avoid repetition of 512 bytes in roundedSizeBytesString

doc: Corrects lvm striping options

lxd/instance: Renames driver files for consistency

lxd/instance: Comment clarification

lxd-agent: Fixes bug when agent not seen as started if LXD restarted

lxd/device/device/utils/network: Adds host MTU support for VM tap devices

lxd/device/nic/bridged: Makes VM host side TAP interface name prefixed with “tap”

lxd/instance/drivers/qemu: Switch to template pkg to generate qemu conf

doc/api-extensions: Fix syntax

api: vm_boot_priority

lxd/vm: Add boot.priority

lxd/container/logs: Makes log file retrieval project aware

lxd/container/lxc: Adds devName skipping for startCommon

lxd/device/config/device/runconfig: Adds DevName to MountEntryItem

lxd/device/disk: Adds DevName to MountEntryItem

lxd/device: Adds devName property to network interface run config

lxd/instance/drivers/driver/qemu: Adds support for Disk and NIC device boot.priority setting

Use a light TCP/TLS connection attempt instead of a client request

lxd/container/exec: Removes duplication of env map now its being stored back into post data

Revert “lxd/exec: Forward control messages”

lxd/instance/drivers/driver/qemu/cmd: Makes qemu cmd struct qemu specific

lxd/instance/drivers/driver/qemu: Simplifies Exec with revert

lxd/container/exec: Cleaned up logging

lxd/container/exec: Switches to use instance command for resizing window

lxd/container/lxc/exec/cmd: Adds WindowResize

lxd/instance/instance/exec/cmd: Adds WindowResize function to signature

lxd/instance/drivers/driver/qemu: Reworks command control

lxd/instance/drivers/driver/qemu/cmd: Adds WindowResize support

lxd/instance/drivers/driver/qemu: Sets PID to 0 for VM commands

lxd/instance/drivers/driver/qemu: comment on forwardControlCommand

lxd/device/nic/p2p: Adds VM support

Fix translation of hard-coded address of first node

Close http transports since they might keep connections around

lxd/daemon: Ignore SIGHUP

lxd/instance/drivers/driver/qemu: Switch to unsafe async I/O mode on ZFS pools backed by loop files

lxd/storage: Improves pool init failure messages

lxd/storage/drivers: Indicates DirectIO support for most storage drivers

lxd/storage/drivers/driver/types: Adds DirectIO indicator to driver info struct

shared/version/version: Quotes malformed version string in error message

lxd/storage/drivers/driver/zfs: Adds DirectIO detection based on version

lxd/instance/drivers/driver/qemu: Unmounts volume on start failure if needed

lxd/device: Relaxes requirement for name property when not using containers

lxd/device/nic/macvlan: Clean up valid fields

lxd/device/nic/macvlan: Adds VM support and improves revert

lxd/instance/drivers/driver/qemu: Adds macvtap support

lxd/instance/drivers/driver/qemu/templates: Moves templates to separate file

lxd/instance/drivers/driver/qemu: Updates template usage

lxd/storage/drivers/driver/dir: Adds HostPath support

storage: Fix xfs_growfs command for older versions

shared/simplestreams: Fix architecture filtering

lxd/patches: Reset ZFS mountpoint/canmount

shared/simplestreams: Fix inconsistent sorting

lxd/instances: Don’t rquire type on copy

lxc/config: Tweak argument processing

lxd/vm: Prevent attaching directory as disk

lxd/storage/zfs: Ignore bookmarks

lxd/storage/btrfs: Skip missing quota

doc/instance: Clarifies disk path not available for VMs

lxd/instance/drivers/qmp/monitor: Prevent crashes with races closing closed channel

lxd/instance/drivers/driver/qemu: Improve clean up on start failure

Fix request redirect when removing a cluster member

lxd/storage/backend/lxd: Only detect volume.block.filesystem changes on block backed pool FS volumes

lxd/migration/migration/volumes: Adds support for pre-bidirectional negotiation targets

lxd/container/lxc: Removes VM specific NIC config ignoring

lxd/device: Only return devName NIC config item for VMs

lxd/device/nic/physical: Improves revert and deletion of created VLAN devices

lxd/instance/drivers/driver/qemu/templates: Clarifies qemuNetdevPhysical variables

lxd/device/nic/macvlan: Differentiates config parent from actual parent

lxd/device/device/utils/network: Adds networkGetDevicePCIDevice function

lxd/device/nic/sriov: Updates networkGetVFDevicePCISlot to use networkGetDevicePCIDevice

lxd/instance/drivers/driver/qemu: Adds physical NIC passthrough support

shared/instance: Updates config key checker to allow “.driver” keys

doc/instance: Documents which device types can be used with which instance types

lxd/device/device/utils/network: Adds generic PCI device bind/unbind functions

lxd/device/device/utils/network: Adds networkVFIOPCIRegister

lxd/device/nic/sriov: Switches PCI device bind/unbind to generic functions

lxd/device/nic/physical: Adds VM PCI passthrough support

lxd/device: Unexports NetworkRemoveInterfaceIfNeeded

lxd/instance: Add NetworkUpdateStatic

Add maasRename to VM

lxd/storage/generic: Don’t fail rename on missing path

lxd/storage/zfs: Fix block mounts

lxd/storage/zfs: Fix renames

lxd/vm: Implement Rename

lxd/device/nic/sriov: Adds VM support

lxd/instance/drivers/driver/qemu: Mount VM config vol before generating NVRAM file

lxd/device: Add unix_hotplug device type

lxd/device: Add support for listening to unix char and block udev events

lxd/storage: Pass config when deleting images

lxd/devices: Remove dead xtables code

lxd/iptables: Fix matching of IPv6 link-local

lxd: Updates usage of migration.MatchTypes

lxd/migration/migration/volumes: Updates MatchTypes to return all supported migration types

lxd/migration/migration/volumes: Break after first rsync transport features extracted

shared/subprocess: Cleanup test script

shared/subprocess: Fix Wait, tty and ignore stdin

shared/subprocess: Better handle not running

lxd/networks: Avoid dnsmasq reload on start

lxd: Switch to using the new subprocess module

lxd/patches: Convert PID files

shared/subprocess: Cleanup tests

shared/subprocess: Use channel for Wait

lxd/apparmor: Allow ro,remount,noatime,bind

lxd/storage/drivers: Pass mountPath to xfs_growfs

lxd/container: Removes containerValidName function

lxd/container: Switches to instance.ValidName

lxd/instance/instance/utils: Adds ValidName function

shared/util: Modifies ValidHostname to return specific error

shared/instance: InstanceGetParentAndSnapshotName comments

lxd/storgage/locking/lock: Fixes concurrent access race to map

global: Replace Fatalf by Errorf in tests

shared/generate: Fix regression caused by Fatalf fix

devices: retrieve vendor and product for hidraw devices

lxd/db: Fix for new testify

lxd/main: Adds cmdGlobal.rawArgs function

lxd: Adds forklimits command

lxd/instance/drivers/driver/qemu: Switches to launching qemu via forklimits

devices: substract libudev header

lxd/db: adds unix-hotplug device type to database

lxd/instance/drivers/driver/qemu: Adds qemu binary path lookup

lxd/main/forklimits: Switches forklimits to use syscall.Exec

shared/cert: Replace default IPs with localhost

shared/subprocess: Improve error in test

spelling: yaml should be YAML

spelling: Busybox should be BusyBox

i18n: Update translation templates

doc/storage: Update for snap package

api: Add extension for new device type unix hotplug

doc/instances: added new device type unix hotplug

doc: Add libudev-dev dependency

lxd/vm: Record architecture name

lxd/vm: Cleanup qemu config

lxd/vm: Add ppc64el support

lxd/device/device/common: Splits device common into own file

lxd/device/device: Removes original device.go file

lxd/device/device/interface: Splits device interfaces into own file

lxd/device/device/load: Separates device load functions into own file

lxd/instance/drivers/driver/common: Adds common driver type

lxd/instance/instance/interface: Adds ConfigRead interface

lxd/instance/drivers/load: Updates validDevices() to use device.Validate function

lxd/instance/instance/utils: Removes instanceName from validateDevices function

lxd/instance/drivers/driver/qemu: Embeds common type and removes dupe functionality

lxd: instance.ValidDevices usage

lxd/device/device/utils/instance: Adds instanceSupported function

lxd/device: Updates device validateConfig to support instance.ConfigReader argument

api: Add api_filtering extension

lxd/filter: Add API filtering package

lxd/instance: Add instance list filtering functions

lxd: Make use of filtering for instances and images

doc/rest-api: Document filtering

tests: Add tests for API filtering

lxd/filter: Workaround gofmt bug

lxd/device/disk: Adds a check for mkisofs tool for qemu config drive

lxd/device/nic/sriov: Loads vfio-pci module

tests: Fix BusyBox spelling for filtering

lxd/vm: Fix bad bus name on ppc64el

lxd/vm: Don’t specify addresses for pci on ppc64

i18n: Update translations from weblate

Try it for yourself

This new LXD release is already available for you to try on our demo service.

Downloads

The release tarballs can be found on our download page.

Binary builds are also available for: