This is the fourth post in the series.

There is a tool for benchmarking LXD, called lxd-benchmark. We are installing it and running it. It is part of the package lxd-tools .

ubuntu@myserver:~$ sudo apt install -y lxd-tools

Let’s see what we can benchmark with this tool.

ubuntu@myserver:~$ lxd-benchmark

Description:

Benchmark performance of LXD



This tool lets you benchmark various actions on a local LXD daemon.



It can be used just to check how fast a given LXD host is, to

compare performance on different servers or for performance tracking

when doing changes to the LXD codebase.



A CSV report can be produced to be consumed by graphing software.



Usage:

lxd-benchmark [command]



Examples:

# Spawn 20 Ubuntu containers in batches of 4

lxd-benchmark launch --count 20 --parallel 4



# Create 50 Alpine containers in batches of 10

lxd-benchmark init --count 50 --parallel 10 images:alpine/edge



# Delete all test containers using dynamic batch size

lxd-benchmark delete



Available Commands:

delete Delete containers

help Help about any command

init Create containers

launch Create and start containers

start Start containers

stop Stop containers



Flags:

-h, --help Print help

-P, --parallel Number of threads to use (default -1)

--report-file Path to the CSV report file

--report-label Label for the new entry in the report [default=ACTION]

--version Print version number



Use "lxd-benchmark [command] --help" for more information about a command.

ubuntu@myserver:~$



We can mass-init (i.e. create), mass-start, mass-launch (i.e. init and then start) and mass-delete containers.

Basic examples



Launching 20 containers in batches of 4

The first example in the command line help, is to launch (init and start) 20 containers in batches of 4. The default image is the current LTS version of Ubuntu (18.04.1). Here it is.

ubuntu@myserver:~$ lxd-benchmark launch --count 20 --parallel 4

Test environment:

Server backend: lxd

Server version: 3.0.2

Kernel: Linux

Kernel architecture: x86_64

Kernel version: 4.15.0-20-generic

Storage backend: zfs

Storage version: 0.7.5-1ubuntu13

Container backend: lxc

Container version: 3.0.2

Test variables:

Container count: 20

Container mode: unprivileged

Startup mode: normal startup

Image: ubuntu:

Batches: 5

Batch size: 4

Remainder: 0



[Oct 3 19:40:22.687] Found image in local store: c395a7105278712478ec1dbfaab1865593fc11292f99afe01d5b94f1c34a9a3a

[Oct 3 19:40:22.687] Batch processing start

[Oct 3 19:40:28.258] Processed 4 containers in 5.572s (0.718/s)

[Oct 3 19:40:34.459] Processed 8 containers in 11.772s (0.680/s)

[Oct 3 19:40:46.106] Processed 16 containers in 23.419s (0.683/s)

[Oct 3 19:40:53.007] Batch processing completed in 30.320s



ubuntu@myserver:~$

The launching speed is about 0.7 containers per second at a total of 30 seconds.



Deleting 20 containers

Let’s delete those 20 containers.

ubuntu@myserver:~$ lxd-benchmark delete

Test environment:

Server backend: lxd

Server version: 3.0.2

Kernel: Linux

Kernel architecture: x86_64

Kernel version: 4.15.0-20-generic

Storage backend: zfs

Storage version: 0.7.5-1ubuntu13

Container backend: lxc

Container version: 3.0.2

[Oct 3 19:45:28.908] Deleting 20 containers

[Oct 3 19:45:28.908] Batch processing start

[Oct 3 19:45:45.192] Batch processing completed in 16.284s

ubuntu@myserver:~$

It took about 16 seconds to delete 20 containers.

Creating 50 Alpine containers in batches of 10

Next, we create (in fact, we init) 50 Alpine containers in batches of 10. init means that we create the container but not start it yet.

ubuntu@myserver:~$ lxd-benchmark init --count 50 --parallel 10 images:alpine/edge

Test environment:

Server backend: lxd

Server version: 3.0.2

Kernel: Linux

Kernel architecture: x86_64

Kernel version: 4.15.0-20-generic

Storage backend: zfs

Storage version: 0.7.5-1ubuntu13

Container backend: lxc

Container version: 3.0.2



Test variables:

Container count: 50

Container mode: unprivileged

Startup mode: normal startup

Image: images:alpine/edge

Batches: 5

Batch size: 10

Remainder: 0

[Oct 3 19:46:51.279] Found image in local store: c9912a815b2d0ccf91694b3c270938f280bb7cb96bdb15c95b14ca14eb268bb5

[Oct 3 19:46:51.279] Batch processing start

[Oct 3 19:46:52.098] Processed 10 containers in 0.819s (12.214/s)

[Oct 3 19:46:53.449] Processed 20 containers in 2.169s (9.219/s)

[Oct 3 19:46:55.936] Processed 40 containers in 4.657s (8.589/s)

[Oct 3 19:46:56.978] Batch processing completed in 5.699s

ubuntu@myserver:~$

It took about 6 seconds to init 50 Alpine containers at a rate of about 9 containers per second.

Finally, we delete all containers as shown earlier. It took 15.6s to delete those 50 containers.

Launching 100 Ubuntu containers

The next benchmark is to launch one hundred Ubuntu containers. Those containers will get IP addresses from the lxdbr0 bridge.



Here is the command to launch 100 Ubuntu 18.04 containers. We select to launch in batches of 24 containers.



ubuntu@myserver:~$ lxd-benchmark launch --count 100 --parallel 24 ubuntu:18.04

Test environment:

Server backend: lxd

Server version: 3.0.2

Kernel: Linux

Kernel architecture: x86_64

Kernel version: 4.15.0-20-generic

Storage backend: zfs

Storage version: 0.7.5-1ubuntu13

Container backend: lxc

Container version: 3.0.2



Test variables:

Container count: 100

Container mode: unprivileged

Startup mode: normal startup

Image: ubuntu:18.04

Batches: 4

Batch size: 24

Remainder: 4



[Oct 4 18:04:18.836] Found image in local store: d72ae2e5073f20450c5260e6f227484c23452a46c6bb553ffe6be55e48602bb4

[Oct 4 18:04:18.836] Batch processing start

[Oct 4 18:04:32.907] Processed 24 containers in 14.070s (1.706/s)

[Oct 4 18:04:51.117] Processed 48 containers in 32.281s (1.487/s)

[Oct 4 18:05:31.630] Processed 96 containers in 72.793s (1.319/s)

[Oct 4 18:05:40.542] Batch processing completed in 81.706s

ubuntu@myserver:~$

The speed was about 1.3 launched containers per second, created in about a minute and 20 seconds.

Then, we stop these container and finally delete them.

ubuntu@myserver:~$ lxd-benchmark stop

Test environment:

Server backend: lxd

Server version: 3.0.2

Kernel: Linux

Kernel architecture: x86_64

Kernel version: 4.15.0-36-generic

Storage backend: zfs

Storage version: 0.7.5-1ubuntu16.3

Container backend: lxc

Container version: 3.0.2



[Oct 4 18:11:31.106] Stopping 100 containers

[Oct 4 18:11:31.106] Batch processing start

[Oct 4 18:11:38.386] Processed 48 containers in 7.280s (6.593/s)

[Oct 4 18:11:44.291] Processed 96 containers in 13.185s (7.281/s)

[Oct 4 18:11:47.516] Batch processing completed in 16.410s



ubuntu@myserver:~$ lxd-benchmark delete

Test environment:

Server backend: lxd

Server version: 3.0.2

Kernel: Linux

Kernel architecture: x86_64

Kernel version: 4.15.0-36-generic

Storage backend: zfs

Storage version: 0.7.5-1ubuntu16.3

Container backend: lxc

Container version: 3.0.2



[Oct 4 18:12:22.896] Deleting 100 containers

[Oct 4 18:12:22.896] Batch processing start

[Oct 4 18:12:47.550] Processed 48 containers in 24.653s (1.947/s)

[Oct 4 18:13:15.700] Processed 96 containers in 52.804s (1.818/s)

[Oct 4 18:13:19.553] Batch processing completed in 56.657

The 100 container were stopped in 16 seconds and then deleted in 57 seconds.



Launching 1200 Alpine containers

The next benchmark is to launch many Alpine containers. These containers will not get IP addresses from the lxdbr0 bridge because there is a hard Linux kernel limit to the number of IP addresses on a single bridge, and that limit is 1024. It is in BR_MAX_PORTS. One option would be to recompile the kernel and increase the value. Another option would be to use something other than a bridge (like openvswitch ).

To disable networking in LXD, we are going to remove the networking configuration from the LXD default profile. Here is the command to remove the configuration. At the end of the benchmark, we show how to add again.

ubuntu@myserver:~$ lxc profile device remove default eth0

Device eth0 removed from default

ubuntu@myserver:~$

Here is the command to launch 1200 Alpine containers.

ubuntu@myserver:~$ lxd-benchmark launch --count 1200 --parallel 24 images:alpine/edge

Test environment:

Server backend: lxd

Server version: 3.0.2

Kernel: Linux

Kernel architecture: x86_64

Kernel version: 4.15.0-20-generic

Storage backend: zfs

Storage version: 0.7.5-1ubuntu13

Container backend: lxc

Container version: 3.0.2



Test variables:

Container count: 1200

Container mode: unprivileged

Startup mode: normal startup

Image: ubuntu:18.04

Batches: 50

Batch size: 24

Remainder: 0



[Oct 4 19:12:24.761] Found image in local store: 87530b2678f03cbc51a7e1732acfed679c4a15ba1939197cbda733346ef28ad9

[Oct 4 19:12:24.761] Batch processing start

[Oct 4 19:12:27.209] Processed 24 containers in 2.447s (9.806/s)

[Oct 4 19:12:29.892] Processed 48 containers in 5.130s (9.356/s)

[Oct 4 19:12:35.286] Processed 96 containers in 10.525s (9.121/s)

[Oct 4 19:12:47.344] Processed 192 containers in 22.583s (8.502/s)

[Oct 4 19:13:11.431] Processed 384 containers in 46.669s (8.228/s)

[Oct 4 19:14:03.468] Processed 768 containers in 98.707s (7.781/s)

[Oct 4 19:15:13.451] Batch processing completed in 168.690s

ubuntu@myserver:~$



The speed was about 8 containers per second, created in less than 3 minutes.

Now, let’s delete them and bring back the LXD installation to the original state.

ubuntu@myserver:~$ lxd-benchmark delete

Test environment:

Server backend: lxd

Server version: 3.0.2

Kernel: Linux

Kernel architecture: x86_64

Kernel version: 4.15.0-36-generic

Storage backend: zfs

Storage version: 0.7.5-1ubuntu16.3

Container backend: lxc

Container version: 3.0.2



[Oct 4 19:16:56.644] Deleting 1200 containers

[Oct 4 19:16:56.644] Batch processing start

[Oct 4 19:17:00.827] Processed 48 containers in 4.182s (11.477/s)

[Oct 4 19:17:05.152] Processed 96 containers in 8.508s (11.284/s)

[Oct 4 19:17:13.448] Processed 192 containers in 16.804s (11.426/s)

[Oct 4 19:17:28.437] Processed 384 containers in 31.793s (12.078/s)

[Oct 4 19:17:53.225] Processed 768 containers in 56.581s (13.574/s)

[Oct 4 19:18:13.147] Batch processing completed in 76.503s

ubuntu@myserver:~

We managed to delete all the container in a minute and 15 seconds. The deletion speed was about 14 containers per second.

Here we are adding back the eth0 device in the LXD default profile.



ubuntu@myserver:~$ lxc profile device add default eth0 nic nictype=bridged parent=lxdbr0

Device eth0 added to default

ubuntu@myserver:~$

Summary



We installed lxd-benchmark , a tool that helps perform benchmarks for LXD. We run the standard benchmarks that are listed in the command’s help. Then, we launched 100 LXD containers and cleared them up. Finally, we launched 1200 LXD container (without networking due to external caveats) and then deleted them.

List of other posts in this series

Share this: Twitter

Facebook

Reddit

Email

Print



Like this: Like Loading...