Introduction

The LXD team is very excited to announce the release of LXD 3.12!

This is one of the more feature packed releases and if you are a cluster user, there should be a lot to be happy about!

We have taken a look through all LXD commands and how they work against clusters, improved our APIs where they were lacking and tweaked the commands to give cluster operators a better experience.

But cluster improvements are far from the only thing improved with this LXD releases.

We’ve also finally got shiftfs support! This feature we’ve been planning for well over a year is finally there when combined with a suitable kernel. With this, LXD containers don’t need any slow shifting on initial startup, reducing the filesystem delta and making container creation so much faster!

Lastly, resource reporting was significantly improved, both in the API and the CLI. We now have more details about the CPU topology, especially NUMA for multi-socket systems and are also now exposing GPU configuration.

Enjoy!

New features

Cluster: Aggregated DHCP leases

LXD managed networks that span multiple cluster members now show a unified view of their DHCP leases, showing hostname, MAC, address and the cluster member’s name for each lease.

root@edfu:~# lxc network list-leases lxdfan0 +----------+-------------------+--------------+---------+----------+ | HOSTNAME | MAC ADDRESS | IP ADDRESS | TYPE | LOCATION | +----------+-------------------+--------------+---------+----------+ | a1 | 00:16:3e:2b:de:8c | 240.31.0.206 | DYNAMIC | edfu | +----------+-------------------+--------------+---------+----------+ | a2 | 00:16:3e:01:99:58 | 240.34.0.124 | DYNAMIC | djanet | +----------+-------------------+--------------+---------+----------+ | a3 | 00:16:3e:b4:8b:94 | 240.36.0.96 | DYNAMIC | nuturo | +----------+-------------------+--------------+---------+----------+ | a4 | 00:16:3e:52:13:2b | 240.31.0.212 | DYNAMIC | edfu | +----------+-------------------+--------------+---------+----------+ | a5 | 00:16:3e:45:54:80 | 240.34.0.68 | DYNAMIC | djanet | +----------+-------------------+--------------+---------+----------+ | a6 | 00:16:3e:d1:81:e3 | 240.36.0.90 | DYNAMIC | nuturo | +----------+-------------------+--------------+---------+----------+

Cluster: Events now show location

Event messages are now all marked with the name of the originating cluster member as their location.

location: edfu metadata: class: task created_at: "2019-04-05T04:13:21.212580932Z" description: Creating container err: "" id: 0c8e4a7d-ef7b-41a0-b949-7030f9aa6827 location: edfu may_cancel: false metadata: null resources: containers: - /1.0/containers/a10 status: Running status_code: 103 updated_at: "2019-04-05T04:13:21.212580932Z" timestamp: "2019-04-05T04:13:21.223834434Z" type: operation

Additionally LXD will now only forward log messages of importance WARN or higher to other members, keeping the INFO and DBUG messages local to reduce network chatter. This behavior can be changed by starting the LXD daemon in debug mode, at which point all log levels will be broadcasted again.

Cluster: Operations now show location

Another area that now benefits from clear tracking of cluster members is operations, as can be seen in lxc operation list :

root@edfu:~# lxc operation list +--------------------------------------+-----------+-------------------+---------+------------+----------------------+----------+ | ID | TYPE | DESCRIPTION | STATUS | CANCELABLE | CREATED | LOCATION | +--------------------------------------+-----------+-------------------+---------+------------+----------------------+----------+ | 36c11142-52d8-4c1e-a342-63657096cdec | WEBSOCKET | Executing command | RUNNING | NO | 2019/04/05 04:19 UTC | edfu | +--------------------------------------+-----------+-------------------+---------+------------+----------------------+----------+ | 701175cf-df82-4ef5-8078-a25d83b770b3 | WEBSOCKET | Executing command | RUNNING | NO | 2019/04/05 04:19 UTC | djanet | +--------------------------------------+-----------+-------------------+---------+------------+----------------------+----------+

This now makes it clear what cluster member is busy doing what and should simplify making sure that a system isn’t actively used before performing maintenance on it.

Cluster: Support for --target in more commands

The following commands have now grown support for --target :

lxc config edit/get/show/set/unset

lxc info [–resources]

lxc network info

lxc storage info

This makes it possible to configure some member-specific daemon configuration options, query cluster member runtime information and system resources, get detailed network statistics and storage usage.

Shiftfs support

This is a feature we’ve been looking forward to for years and that we are really excited to finally see coming to completion. shiftfs allows for an unprivileged container experience that doesn’t need any shifting of the filesystem, instead having the kernel do it on the fly.

This requires kernel support through the shiftfs filesystem which is currently a custom patchset that will be present in the Ubuntu 19.04 kernel.

LXD automatically detects support for this and will transparently start using it whenever possible.

Kernel features now exported over API

For some time now, LXD has been detecting a number of optional kernel features on startup and would print an overview then. That same information is now exposed over the API and visible in lxc info .

kernel_features: netnsid_getifaddrs: "true" shiftfs: "true" uevent_injection: "true" unpriv_fscaps: "true"

Improved CPU reporting

The server resources API now exposes CPU sockets and NUMA node information, making it easier to do CPU pinning for containers.

root@djanet:~# lxc info --resources --target edfu CPUs: Socket 0: Vendor: GenuineIntel Name: Intel(R) Xeon(R) CPU E5430 @ 2.66GHz Cores: 4 Threads: 4 Frequency: 1999Mhz (max: 2336Mhz) NUMA node: 0 Socket 1: Vendor: GenuineIntel Name: Intel(R) Xeon(R) CPU E5430 @ 2.66GHz Cores: 4 Threads: 4 Frequency: 1999Mhz (max: 2336Mhz) NUMA node: 1 Memory: Free: 18.37GB Used: 557.76MB Total: 18.93GB GPU: Vendor: ASPEED Technology, Inc. (1a03) Product: ASPEED Graphics Family (2000) PCI address: 0000:06:03.0 Driver: ast (4.15.0-47-generic) NUMA node: 0

The output of lxc info --resources has also been tweaked to adapt to the hardware present on the system.

GPU reporting

As you may have noticed in the previous listing, GPUs are now present in the system resources output. Additional information can also be seen for NVIDIA cards:

root@vm10:~# lxc info --resources CPU: Vendor: GenuineIntel Name: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz Cores: 2 Threads: 4 Frequency: 2400Mhz NUMA node: 0 Memory: Free: 8.14GB Used: 225.81MB Total: 8.36GB GPUs: Card 0: Vendor: NVIDIA Corporation (10de) Product: GK208B [GeForce GT 730] (1287) PCI address: 0000:00:07.0 Driver: nvidia (418.56) NUMA node: 0 NVIDIA information: Architecture: 3.5 Brand: GeForce Model: GeForce GT 730 CUDA Version: 10.1 NVRM Version: 418.56 UUID: GPU-6ddadebd-dafe-2db9-f10f-125719770fd3 Card 1: Vendor: NVIDIA Corporation (10de) Product: GK208B [GeForce GT 730] (1287) PCI address: 0000:00:09.0 Driver: nvidia (418.56) NUMA node: 0 NVIDIA information: Architecture: 3.5 Brand: GeForce Model: GeForce GT 730 CUDA Version: 10.1 NVRM Version: 418.56 UUID: GPU-253db1df-f725-a174-99d4-a8933288c39e

Snapshot expiry now visible in lxc info

On top of showing when a snapshot was taken, snapshots that have an expiry will now show their expiry in the listing too.

root@djanet:~# lxc info a1 Name: a1 Location: edfu Remote: unix:// Architecture: x86_64 Created: 2019/04/05 04:07 UTC Status: Stopped Type: persistent Profiles: default Snapshots: snap0 (taken at 2019/04/05 04:20 UTC) (expires at 2019/04/05 05:20 UTC) (stateless) snap1 (taken at 2019/04/05 04:50 UTC) (expires at 2019/04/05 05:50 UTC) (stateless) snap2 (taken at 2019/04/05 04:55 UTC) (expires at 2019/04/05 05:55 UTC) (stateless) snap3 (taken at 2019/04/05 04:52 UTC) (stateless) snap4 (taken at 2019/04/05 05:00 UTC) (expires at 2019/04/05 06:00 UTC) (stateless)

Bugs fixed

client: Optimize copies on same nodes

client: Properly generate events URL

doc: Fix typo in api-extensions.md

doc: Inform about ZFS pool default compression

doc: Introduce volatile.idmap.current

doc: Fix typo in faq.md

doc: Tweak markdown format in storage.md

doc: Update documentation for snapshots.pattern

i18n: Update translations from weblate

i18n: Update translation templates

lxc: Use shared.IsSnapshot

lxc/action: skip containers with intended state

lxc/config: Use shared.IsSnapshot

lxc/launch: Show start progress

lxd: Don’t leak netlink fds

lxd: Drop initialShiftRootfs and always shift on start

lxd/backups: Attempt to delete storage on failure

lxd/backups: Cleanup on failure

lxd/backups: Re-order checks for backup.yaml

lxd/cluster: Export Snapshot function

lxd/cluster: Initialize candid on join

lxd/cluster: Limit log message forwarding

lxd/containers: Cleanup shifting

lxd/containers: Cleanup template application

lxd/containers: Export container location

lxd/containers: Fix crash on refresh of non-existing

lxd/containers: Fix owner/mode of container path

lxd/containers: Handle mid-remap containers

lxd/containers: Properly handle tar shifting

lxd/containers: Stop proxy before storage

lxd/containers: Use LXC hook version 1

lxd/devices: Cleanup GPU structs

lxd/devices: Track vendor/product names and driver

lxd/images: Don’t keep an in-memory simplestreams cache

lxd/internal: Expose raft-snapshot

lxd/internal: Have GC endpoint release memory

lxd/main_forkproxy: Fix epoll

lxd/migration: Shift CRIU files to current map

lxd/migration: Fix handling of missing profiles

lxd/networks: Bring mtu device up

lxd/patches: Fix names of pool volume LVs

lxd/resources: Fix bad CPU reporting

lxd/response: Simplify SmartError

lxd/storage: Make use of shared.IsSnapshot

lxd/storage: Remove setUnprivUserACL

lxd/storage: Rename ShiftIfNecessary to resetContainerDiskIdmap

lxd/storage: Rename shiftRootfs to initialShiftRootfs

lxd/storage: Add helper function to get volume snapshots

lxd/storage: Fix copying and moving volume snapshots

lxd/storage/btrfs: Fix volume copy with snapshots

lxd/storage/ceph: Always unmap after use

lxd/storage/ceph: Fix copying existing volume snap

lxd/storage/ceph: Fix volume copy with snapshots

lxd/storage/ceph: Only freeze if needed

lxd/storage/dir: Fix volume copy with snapshots

lxd/storage/lvm: Fix LV naming

lxd/storage/lvm: Fix volume copy with snapshots

lxd/storage/lvm: Pass nouuid for xfs backups

lxd/storage/zfs: Fix volume copy with snapshots

lxd/storage/zfs: Run rename in clean mntns

lxd/tasks: Avoid races on startup

lxd-p2c: Workaround for broken /proc/self/exe

shared: Switch ParseNumberFromFile to simple read

shared/api: Drop StoragePool from Resources struct

shared/api: Sort ServerEnvironment struct

shared/idmap: Use separate uid and gid entries

shared/osarch: Add Plamo x86 arch

shared/simplestreams: Align JSON struct for images.json

shared/simplestreams: Align JSON struct for index.json

shared/utils: Do not chown terminal master fd

tests: Add volume copy tests

tests: Allow up to 15s for container reboot

tests: Fix race condition in proxy test

tests: Make proxy tests work with shiftfs

tests: Make security tests work with shiftfs

tests: Remove dead code

tests: Update resources test

Try it for yourself

This new LXD release is already available for you to try on our demo service.

Downloads

The release tarballs can be found on our download page.