Application startup time is one thing. It’s time to run a more critical comparison test. Let’s compare the throughput of both, GraalVM and Oracle JDK runtime environments.

We will start with a small number of requests so that the Oracle JDK won’t have enough time to warm up properly. In this test, we use Apache Bench tool, and we execute 200 concurrent requests with a total of 1000 requests. Let’s start with GraalVM.

~ % ab -c 200 -n 1000 http://localhost:5050/ This is ApacheBench, Version 2.3 <$Revision: 1826891 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking localhost (be patient) Completed 100 requests Completed 200 requests Completed 300 requests Completed 400 requests Completed 500 requests Completed 600 requests Completed 700 requests Completed 800 requests Completed 900 requests Completed 1000 requests Finished 1000 requests Server Software: Server Hostname: localhost Server Port: 5050 Document Path: / Document Length: 27 bytes Concurrency Level: 200 Time taken for tests: 0.090 seconds Complete requests: 1000 Failed requests: 0 Total transferred: 117000 bytes HTML transferred: 27000 bytes Requests per second: 11153.00 [#/sec] (mean) Time per request: 17.932 [ms] (mean) Time per request: 0.090 [ms] (mean, across all concurrent requests) Transfer rate: 1274.32 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 4 1.2 4 6 Processing: 1 7 4.8 5 22 Waiting: 1 6 4.7 4 19 Total: 6 11 4.2 9 23 Percentage of the requests served within a certain time (ms) 50% 9 66% 9 75% 10 80% 12 90% 20 95% 22 98% 22 99% 22 100% 23 (longest request)

That was fast. Now let’s see regular Oracle JDK in action.

I start demo application with the following command java -jar build/libs/ratpack-graalvm-demo-all.jar -Dratpack.epoll.disable=true

ab -c 200 -n 1000 http://localhost:5050/ This is ApacheBench, Version 2.3 <$Revision: 1826891 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking localhost (be patient) Completed 100 requests Completed 200 requests Completed 300 requests Completed 400 requests Completed 500 requests Completed 600 requests Completed 700 requests Completed 800 requests Completed 900 requests Completed 1000 requests Finished 1000 requests Server Software: Server Hostname: localhost Server Port: 5050 Document Path: / Document Length: 27 bytes Concurrency Level: 200 Time taken for tests: 0.335 seconds Complete requests: 1000 Failed requests: 0 Total transferred: 117000 bytes HTML transferred: 27000 bytes Requests per second: 2985.77 [#/sec] (mean) Time per request: 66.984 [ms] (mean) Time per request: 0.335 [ms] (mean, across all concurrent requests) Transfer rate: 341.15 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 1 1.8 0 7 Processing: 5 29 13.3 25 98 Waiting: 5 29 13.3 25 94 Total: 5 30 13.7 25 98 Percentage of the requests served within a certain time (ms) 50% 25 66% 31 75% 36 80% 39 90% 47 95% 56 98% 70 99% 83 100% 98 (longest request)

The difference between the cold Oracle JDK and GraalVM is enormous:

17.93 ms vs. 66.98 ms mean time per request in GraalVM' favour.

11153 vs. 2985 requests per second in GraalVM’s favour.

However, let’s be fair - Oracle JDK shows its full potential when JIT jumps in and runs its optimizations. In the next round, we will let it warm up correctly and then we can compare the results. We will run 800 concurrent requests with a total of 500,000 requests, and we are going to do it twice - the first run is used to warm up the JVM so that we take only the second result into account. Let’s start with GraalVM.

ab -c 800 -n 500000 http://localhost:5050/ This is ApacheBench, Version 2.3 <$Revision: 1826891 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking localhost (be patient) Completed 50000 requests Completed 100000 requests Completed 150000 requests Completed 200000 requests Completed 250000 requests Completed 300000 requests Completed 350000 requests Completed 400000 requests Completed 450000 requests Completed 500000 requests Finished 500000 requests Server Software: Server Hostname: localhost Server Port: 5050 Document Path: / Document Length: 27 bytes Concurrency Level: 800 Time taken for tests: 40.725 seconds Complete requests: 500000 Failed requests: 0 Total transferred: 58500000 bytes HTML transferred: 13500000 bytes Requests per second: 12277.48 [#/sec] (mean) Time per request: 65.160 [ms] (mean) Time per request: 0.081 [ms] (mean, across all concurrent requests) Transfer rate: 1402.80 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 37 110.8 25 3130 Processing: 3 28 9.4 27 117 Waiting: 0 18 8.3 17 90 Total: 18 65 112.0 55 3156 Percentage of the requests served within a certain time (ms) 50% 55 66% 61 75% 63 80% 64 90% 68 95% 72 98% 84 99% 1072 100% 3156 (longest request)

Now let’s do the same with Oracle JDK.

ab -c 800 -n 500000 http://localhost:5050/ This is ApacheBench, Version 2.3 <$Revision: 1826891 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking localhost (be patient) Completed 50000 requests Completed 100000 requests Completed 150000 requests Completed 200000 requests Completed 250000 requests Completed 300000 requests Completed 350000 requests Completed 400000 requests Completed 450000 requests Completed 500000 requests Finished 500000 requests Server Software: Server Hostname: localhost Server Port: 5050 Document Path: / Document Length: 27 bytes Concurrency Level: 800 Time taken for tests: 35.889 seconds Complete requests: 500000 Failed requests: 0 Total transferred: 58500000 bytes HTML transferred: 13500000 bytes Requests per second: 13931.95 [#/sec] (mean) Time per request: 57.422 [ms] (mean) Time per request: 0.072 [ms] (mean, across all concurrent requests) Transfer rate: 1591.83 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 28 8.1 27 1034 Processing: 2 30 7.7 30 249 Waiting: 0 17 6.9 16 242 Total: 6 57 5.3 57 1065 Percentage of the requests served within a certain time (ms) 50% 57 66% 58 75% 59 80% 60 90% 62 95% 64 98% 66 99% 68 100% 1065 (longest request)

It looks like if we give Oracle JDK enough time to warm up, it runs a little bit more efficient than the GraalVM application. Take a look at these two charts to see the main difference.

If we compare RPS between cold Oracle JDK and GraalVM, there is no doubt that GraalVM does better. However, if we only give a regular Oracle JDK a chance to warm up, it turns out that it can handle almost 1700 more requests. It’s a significant difference.

Latency benchmark also reveals interesting details. GraalVM wins when we compare it to cold Oracle JDK, and we let both applications handle reasonably small traffic (200 requests with a total of 1000). When we increase the number of concurrent requests to 800, and we need to handle the total of 500,000 requests, warmed up Oracle JDK works much better. While GraalVM slows down to the ~65ms per request when we increase the traffic, Oracle JDK speeds up to ~57ms per request.

There are also two things worth mentioning. I’ve tried to execute more concurrent requests, but it turned out that GraalVM starts throwing IOException when I increased the number of concurrent requests to 1,000.

[main] INFO ratpack.server.RatpackServer - Starting server... [main] INFO ratpack.server.RatpackServer - Building registry... [main] INFO ratpack.server.RatpackServer - Ratpack started for http://localhost:5050 [ratpack-compute-2-1] WARN io.netty.channel.DefaultChannelPipeline - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception. java.io.IOException: Accept failed at com.oracle.svm.core.posix.PosixJavaNIOSubstitutions$Util_sun_nio_ch_ServerSocketChannelImpl.accept0(PosixJavaNIOSubstitutions.java:1261) at sun.nio.ch.ServerSocketChannelImpl.accept0(ServerSocketChannelImpl.java:1188) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) at io.netty.util.internal.SocketUtils$5.run(SocketUtils.java:110) at io.netty.util.internal.SocketUtils$5.run(SocketUtils.java:107) at java.security.AccessController.doPrivileged(AccessController.java:82) at io.netty.util.internal.SocketUtils.accept(SocketUtils.java:107) at io.netty.channel.socket.nio.NioServerSocketChannel.doReadMessages(NioServerSocketChannel.java:143) at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:75) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) at ratpack.exec.internal.DefaultExecController$ExecControllerBindingThreadFactory.lambda$newThread$0(DefaultExecController.java:137) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:481) at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:193)

Oracle JDK at the same time was able to handle 1,000 concurrent requests without any issue.