The previous part of this blog showed how to take a thread dump of Payara Server. This part will go through reading these thread dumps, particularly in regard to identifying deadlocks.

The Format of a Thread Dump

Below is a breakdown of a thread dump.

Thread Name

The thread name is self explanatory. These names are very useful when debugging, however, as it gives something to search the codebase for when debugging code. Payara Server has already added fixes for instances where internal threads were not always named, so thread dump information can be more helpful now.

The thread name is self explanatory. These names are very useful when debugging, however, as it gives something to search the codebase for when debugging code. Payara Server has already added fixes for instances where internal threads were not always named, so thread dump information can be more helpful now. Thread Priority

Each thread is given a priority by both Java, and the OS it's being run on. prio and os_prio show these as reported by Java and the OS respectively.

Each thread is given a priority by both Java, and the OS it's being run on. and show these as reported by Java and the OS respectively. Thread Daemon Status

Threads can either be a daemon or non-daemon thread. A daemon thread will be closed when there are no non-daemon threads running. An example of a daemon thread would be garbage collection.

Threads can either be a daemon or non-daemon thread. A daemon thread will be closed when there are no non-daemon threads running. An example of a daemon thread would be garbage collection. Thread ID

Each thread is given an ID by Java and the OS. the Java thread ID is called tid , and is the one that would be returned by thread.getId() . The OS thread id is called nid , and is useful for referring to a thread when seeing if a thread is taking up too much memory.

Each thread is given an ID by Java and the OS. the Java thread ID is called , and is the one that would be returned by . The OS thread id is called , and is useful for referring to a thread when seeing if a thread is taking up too much memory. Thread State and Detail

Each thread has a state as well as details of what it's doing. There are a few different states that a thread can be in: NEW: A thread has just been created but isn't doing anything. RUNNABLE: A thread is currently processing a task. WAITING: A thread is waiting for external input. TIMED WAITING: A thread is waiting, except is can be freed up by a maximum amount of time as well as an external input. BLOCKED: A thread is waiting for another thread to release it's lock.

Each thread has a state as well as details of what it's doing. There are a few different states that a thread can be in: Thread Stack Trace

Each thread has a call stack, which gives exactly which methods have been called for it to get to the point where it currently is.

Identifying Deadlocks

A deadlock is when two or more threads have a lock over some resources and are both waiting for each other to give up their lock, which causes them to hang. Deadlocks aren't always easy to spot, but they can be extremely problematic when they occur. They are often only stopped by restarting the server and removing the problem.

Below, I have made a simple servlet to demonstrate the problem by creating a deadlock when it receives too many requests at once:

@WebServlet ( "/deadlock" ) public class DeadlockServlet extends HttpServlet { @Inject private Model model ; @Inject private View view ; @Override protected void doGet ( HttpServletRequest req , HttpServletResponse resp ) throws ServletException , IOException { resp . getWriter (). println ( "Request completed.

" ); for ( String key: req . getParameterMap (). keySet ()){ model . putData ( key , req . getParameterMap (). get ( key )[ 0 ]); } view . update ( resp . getWriter ()); } @Override protected void doPost ( HttpServletRequest req , HttpServletResponse resp ) throws ServletException , IOException { doGet ( req , resp ); } }

Although, in reality, for something like this you would normally want to use something like CDI Events, this method suits the purpose of creating a deadlock. There is nothing immediately wrong with this servlet. On a GET request it will add each key-value pair parameter to an injected model, and it will call the view in order to print it to the response.

@ApplicationScoped public class Model implements Serializable { private Map < String , String > data ; @Inject private View view ; @PostConstruct private void init () { data = new HashMap <>(); } public synchronized void putData ( String key , String value ){ data . put ( key , value ); view . update ( new PrintWriter ( System . out )); } public synchronized String getData (){ String content = "" ; for ( String key: this . data . keySet ()){ content += key + ": " + this . data . get ( key ) + "

" ; } return content ; } }

The Model bean is @ApplicationScoped , meaning that there is only one per application instance - in other words, every user will use the same model, and will be able to see and add to the same list. The model is just a map of key-value pairs. it can put data into that map, and return a string representation of it. Both of these operations are synchronized , meaning that multiple threads can't manipulate the model at the same time.

@ApplicationScoped public class View { @Inject private Model model ; public synchronized void update ( PrintWriter out ) { out . println ( model . getData ()); } }

The View bean is also @ApplicationScoped . It has a synchronized method which allows the view to print the model to a given PrintWriter .

Each user of a servlet is given their own thread. Knowing this, there doesn't appear to be any code that would cause a threading problem. If however, one thread accesses the view's update() method whilst another thread accesses the model's putData() method, it will cause a deadlock. This is because the view object is waiting for the model's lock, and the model object is waiting for the view's lock, but neither will give up their lock. This will cause the two threads to hang, and will slow down your server. If you're using VisualVM and you call this servlet enough times in a short space of time, you will see an error like this:

Even if you're not using VisualVM, then take a thread dump and you'll see the following problem.