Updated Thanks for your comments. I’ve added Undertow to the test ( “Servlet-Only Distribution” on the WildFly download page), corrected some mistakes, but more important, I’ve changed the memory benchmark: I now take a memory usage measure at startup, I then perform GC and wait a bit for memory to stabilize. This gives a nice min and max memory usage.

Damn, I’ve been waiting so long to publish this blog (which is the successor of the same post on Java EE 6 app servers). The idea is to do some “benchmarking”(basically, startup time, disk and memory usage) on application servers implementing Java EE 7. Java EE 7 came out in may 2013 and we have now many application servers that have passed the TCK. So let’s see where we are in terms of Java EE 7 implementations and production support.

State of Java EE Application Servers

Before diving into each implementation, let’s quickly see what happened in the application server land in these last 3 years.

Geronimo is no longer supported and will not run the race of Java EE 7. Not sure about JonAS but it hasn’t been updated since. GlassFish is still developped by Oracle and still is the Java EE reference implementation, but has no longer production support. You will have to look into Payara which is the supported version of GlassFish. Instead, Oracle is focusing on Weblogic which has Java EE 7 production support. JBoss 7 EAP is still in Beta but WildFly (the community edition of JBoss AS) has been supporting Java EE 7 since version 8. Websphere has now a Liberty and a Classic version. Jeus, from TmaxSoft and uCosminexus from Hitachi are still running the Java EE 7 race but are not as known as the other app servers.

The main editors also have a Java EE 7 Web Profile implementation (GlassFish, Payara, Websphere) and some only implement the Web Profile such as Resin, Siwpas, or TomEE. Last but not least, the Servlet containers also run the Java EE 7 race as they implement Servlet 3.1, such as Jetty, Tomcat or Undertow.

The Benchmark

Disclamer: This is not a real benchmark !

The idea of this benchmark is to download a Java EE 7 application server, install it, start it, launch the admin console if any, and take some measures : size of download, ease of installation, size on disk once installed, startup time, memory usage... That’s all. I do not deploy any application, I don’t do fancy twists to gain performance… I’m just concerned about the usability of an application server for a developer in 2016. I’m doing all my tests on a Retina OS X (16 Gb or RAM, SSD), no Docker! I use JDK 1.8.0_66 (when it’s not bundled with the server). No optimization at all is made (I haven’t twisted the JVM, or any application server parameter… everything comes out of the box).

To calculate the startup time, I don’t do any fancy rocket science either. I just start the server a few times, check the logs to see how long it takes and use the best startup time. Also remember that some servers do not load any container at startup, making them very fast to start. That’s why I trigger the web admin console (when there’s one) so I’m sure at least one web application is deployed. To calculate the memory footprint, I use JConsole and take two measures: memory used at server startup, and memory used after few seconds (after the GC calms down a bit).

Java EE 7 Application Servers

Java EE 7 comes with a Web profile and a Full profile. Full profile means that all the Java EE 7 specifications are supported. Not all the application servers support the Full profile. Here I look at GlassFish, Payara, JBoss EAP, WildFly, Weblogic and Websphere Liberty.

GlassFish 4.x

GlassFish 4 is the open source reference implementation for Java EE 7. Being the reference implementation, GlassFish 4.x implements both Java EE 7 and the Web Profile 7, that’s why you have two different bundles you can download. Oracle has dropped commercial support for GlassFish and Payara is now the supported version.

Version GlassFish 4.1.1 GlassFish Web 4.1.1 What do you get ? Java EE 7 Full Profile Java EE 7 Web Profile Who is behind? Oracle Open source? Dual license CDDL and GPL Documentation Documentation Download Zipfile Zipfile Size of the downloaded file 109.7 Mb 60.7 Mb Installation mode Unzip Size once installed on drive 136.9 Mb 82.4 Mb How to start it $GLASSFISH_HOME/bin/asadmin start-domain Location of log files $GLASSFISH_HOME/glassfish/domains/domain1/logs Best startup time 2 821 ms 1 991 ms RAM at startup (later) 140 Mb (44 Mb) 120 Mb (33 Mb) Welcome page http://localhost:8080/ Admin console http://localhost:4848

Payara

Payara is the commercial support of GlassFish (so it also comes with a Full and Web profile). It is optimized for production environment and also has a micro-service distribution called Payara Micro.

Version Payara 4.1.1.154 Payara Web 4.1.1.154 What do you get ? Java EE 7 Full Profile Java EE 7 Web Profile Who is behind? C2B2 Open source? Dual license CDDL and GPL Documentation Documentation Download Zipfile Zipfile Size of the downloaded file 116 Mb 66.6 Mb Installation mode Unzip Size once installed on drive 142 Mb 87.6 Mb How to start it $PAYARA_HOME/bin/asadmin start-domain Location of log files $PAYARA_HOME/glassfish/domains/domain1/logs Best startup time 3 001 ms 1 953 ms RAM at startup (later) 144 Mb (43 Mb) 118 Mb (33 Mb) Welcome page http://localhost:8080/ Admin console http://localhost:4848

JBoss 7 EAP

Unfortunatelly JBoss 7 EAP is still not final at the time of writing this blog, so no commercial support at RedHat. JBoss 7 EAP is based on WildFly, which is the community edition and evolves at a faster pace.

Version JBoss 7 EAP Beta 1 What do you get ? Java EE 7 Full Profile Who is behind? JBoss/RedHat Open source? LGPL Documentation Documentation Download Zipfile Size of the downloaded file 157.8 Mb Installation mode Unzip Size once installed on drive 184.2 Mb How to start it $JBOSS_HOME/bin/standalone Location of log files $JBOSS_HOME/standalone/log Best startup time 2 106ms RAM at startup (later) 337 Mb (41 Mb) Welcome page http://localhost:8080/ Admin console http://localhost:9990/console

WildFly

WildFly is the community name for the JBoss application server. It has been supporting Java EE 7 since version 8.x. Both WildFly and EAP support the Full and the Web profile but there is no separate bundles, only one.

Version WildFly 8.2.1.Final WildFly 9.0.2.Final WildFly 10.0.0.Final What do you get ? Java EE 7 Full Profile Who is behind? RedHat Open source? LGPL Documentation WildFly 8.x WildFly 9.x WildFly 10.x Download Zipfile Zipfile Zipfile Size of the downloaded file 136.9 Mb 136.7 Mb 138.2 Mb Installation mode Unzip Size once installed on drive 140.4 Mb 161.8 Mb 163.4 Mb How to start it $WILDFLY_HOME/bin/standalone Location of log files $WILDFLY_HOME/standalone/log Best startup time 1 718 ms 1 887 ms 1 989 ms RAM at startup (later) 97 Mb (27 Mb) 112 Mb (26 Mb) 127 Mb (28 Mb) Welcome page http://localhost:8080/ Admin console http://localhost:4848

Weblogic 12.x

If Oracle still uses GlassFish as the Java EE reference implementation, its work and support goes into Weblogic. Weblogic 12 has recently been Java EE 7 certified.

Version Weblogic 12.2.1.0.0 What do you get ? Java EE 7 Full Profile Who is behind? Oracle Open source? No Documentation Documentation Download Zipfile + installer Size of the downloaded file 221.3 Mb Installation mode Unzip + install Size once installed on drive 749.9 Mb How to start it $WLS_HOME/user_projects/domains/base_domain/startWebLogic Location of log files %WLS_HOME/standalone/log Best startup time 8 411 ms* RAM at startup (later) 182 Mb (71 Mb) Admin console http://localhost:7001/console

*Weblogic doesn’t show milliseconds in the logs so I had to change the starting script to add a few date +%s%3N.

Websphere and WebSphere Liberty 8.5.x

Today IBM has two versions of Websphere 8.5 : Classic and Liberty, both implementing Java EE 7. As for the Classic, you need to get lost in the IBM website maze and ask for help on Tweeter, to understand that you can’t install it on Mac OS X. So I just concentrated on WebSphere Liberty which is a completely different beast: download, unzip, run. There is no administration console but.

Version WebSphere Liberty 8.5.5.8 WebSphere Liberty Web 8.5.5.8 What do you get ? Java EE 7 Full Profile Java EE 7 Web Profile Who is behind? IBM Open source? Commercial product Documentation Documentation Download Zipfile Zipfile Size of the downloaded file 99.8 Mb 67.5 Mb Installation mode Unzip Size once installed on drive 122 Mb 83.4 Mb How to start it $WLP_HOME/bin/server start Location of log files $WLP_HOME%/usr/servers/defaultServer/logs Best startup time 3 471 ms 2 093 ms RAM at startup (later) 265 Mb (63 Mb) 49 Mb (39 Mb) Welcome page http://localhost:9080/

Java EE Web Profile 7 Application Servers

If GlassFish, Payara and Websphere have two different distributions (Full and Web profile), some application servers only implement the Web Profile. That’s the case of TomEE and Resin.

TomEE 7.x

Apache TomEE is the perfect success story for the Web Profile. TomEE is no more than Tomcat + OpenWebBeans + OpenEJB + OpenJPA + MyFaces + other bits. It really shows that Java EE is a jigsaw puzzle where you can take open standards, bundle them together and become a certified Web Profile application server. This TomEE version is bundled with Tomcat 8.0.29.

Version TomEE 7.0.0-M1 What do you get ? Java EE 7 Web Profile Who is behind? Apache Open source? Apache Software License Documentation Documentation Download Zip, tar.gz Size of the downloaded file 35.5 Mb Installation mode Unzip the file Size once installed on drive 42.1 Mb How to start it $TOMEE_HOME/bin/startup.sh Location of log files $TOMEE_HOME/logs Best startup time 505 ms RAM at startup (later) 50 Mb (24 Mb) Welcome page http://localhost:8080/ Admin console (*) http://localhost:8080/manager/html

(*) To be able to log on to the admin console, you need to change the $TOMCAT_HOME/conf/tomcat-users.xml configuration files and add a manager-gui role to a user.

Resin 5.x

Resin was one of the first servlet containers (such as Jetty or Tomcat) to move to the Web Profile. Again, it’s another success story that confirmed that profiles in Java EE were needed. Resin is very CDI centric and based on the Caucho implementation called CanDI.

Version Resin 5.0.2-alpha What do you get ? Java EE 7 Web Profile Who is behind? Caucho Open source? GPL License Documentation Documentation Download Zip, tgz, deb Size of the downloaded file 10.7 Mb Installation mode Unzip the file Size once installed on drive 11.7 Mb How to start it $RESIN_HOME/bin/ resin start Location of log files /tmp/resin/log/ Best startup time 1 303 ms RAM at startup (later) 2 processes : 61 Mb + 39 Mb Welcome page http://localhost:8080/

Servlet 3.1 Servers

You can see Java EE 7 application servers implementing everything (the Full profile), a bit (Web profile) or just the web server portion (Servlet 3.1). That’s the case of Tomcat, Jetty or Undertow. Unfortunately I do not benchmark Undertow because it doesn’t have a standalone installation: you need to start it up in your code.

Tomcat

Tomcat has been implementing Servlet 3.1 since version 8.

Version Tomcat 8.0.30 Tomcat 9.0.0.M1 What do you get ? Servlet 3.1 Who is behind? Apache Open source? Apache Software License Documentation Documentation Documentation Download Zip, tar.gz Zip, tar.gz Size of the downloaded file 9.7 Mb 9.9 Mb Installation mode Unzip the file Size once installed on drive 14.6 Mb 14.5 Mb How to start it $TOMCAT_HOME/bin/startup.bat Location of log files $TOMCAT_HOME/logs Best startup time 401 ms 404 ms RAM at startup (later) 45 Mb (17 Mb) 40 Mb (15 Mb) Welcome page http://localhost:8080/ Admin console http://localhost:8080/manager/html

Jetty

Jetty started to implement Servlet 3.1 from Jetty 9.

Version Jetty 9.3.6.v20151106 What do you get ? Servlet 3.1 Who is behind? Eclipse Open source? Apache License 2.0 and Eclipse Public License 1.0 Documentation Documentation Download Zip, tgz, deb Size of the downloaded file 11.7 Mb Installation mode Unzip the file Size once installed on drive 13.4 Mb How to start it $JETTY_HOME/bin/jetty.sh start Location of log files $JETTY_HOME/logs Best startup time 479 ms RAM at startup (later) 52 Mb (8 Mb) Welcome page http://localhost:8080/

Undertow

JBoss Application Server used to use Tomcat as the web container. Since version 8 it uses Undertow that can be downloaded on its own.

Version WildFly Servlet 10.0.0.Final What do you get ? Servlet 3.1 Who is behind? RedHat Open source? LGPL Documentation Documentation Download Zip (unser Servlet-Only Distribution) Size of the downloaded file 29.2 Mb Installation mode Unzip the file Size once installed on drive 35.8 Mb How to start it $WILDFLY_HOME/bin/standalone.sh Location of log files $WILDFLY_HOME/standalone/log Best startup time 1 252 ms RAM at startup (later) 93 Mb (16 Mb) Welcome page http://localhost:8080/

Summary

As a summary I will show you two graphs with startup time and memory consumption so it’s easier to compare.

Startup time

When you are a developer, startup up time matters (even when you move your app server to the Cloud). Thanks to tools like JRebel, or hot deployment in our IDEs, we tend to restart our app servers less and less. But still, it’s important to have fast startup time. Something to stress out, is that most of the app servers bootstrap the minimum set of services and then, lazy load services when needed.

From left to right we have Full profile, Web profile and Servlet containers. Of course, Servlet containers start faster. But it’s interesting to notice that most app servers boot in less that 3 seconds, and half, in less than 2 seconds (remember that these measures are taken on a Mac OS X 16Gb RAMP SSD). Weblogic is by far the slowest (> 8 seconds) and WildFly the fastest of the Java EE Full profile app servers. TomEE 7 (Web profile) is slightly slower than Tomcat (Servlet).

Memory Footprint

The memory consumption is also important. In this era of Micro Services, we want fast startup time and low resource consumption. Without any JVM or server twists, most app servers use less than 150Mb of RAM. Here I take the measure at startup, perform a GC and wait a few seconds so it stabilize. So you get a min and a max memory usage.

Conclusion

First of all, as you can see there is no Websphere Classic. Despite being a big player, it is difficult to find, difficult to install, and has no Mac OS X support. I hope one day I’ll be able to add it to the benchmark.

What did I want to show in this blog ? Well, that application servers have changed. Most app servers take few resources, are modular and have nice tricks to start up fast (like lazy loading services).

And why did I want to show this ? Just to show that the “Tomcat is light, Java EE app servers are not” line is outdated. Use the app server that fits your need. You need Servlet ? Go for Jetty or Tomcat. You need some JAX-RS with CDI and JPA ? Choose TomEE, WildFly or Websphere Liberty. You need the full monty ? Use WildFly, Weblogic or GlassFish. You need production support ? Use JBoss EAP (you’ll have to wait a few extra month). You need to create Micro Services ? Wait for my next blog ;o)

References