One thing that’s really great about JFR and JMC is that you’re not limited to the events and data baked into the JVM and platform libraries: JFR also provides an API for implementing custom events. That way you can use the low-overhead event recording infrastructure (its goal is to add at most 1% performance overhead) for your own event types. This allows you to record and analyze higher-level events, using the language of your application-specific domain.

Taking my day job project Debezium as an example (an open-source platform for change data capture for a variety of databases), we could for instance produce events such as "Snapshot started", "Snapshotting of table 'Customers' completed", "Captured change event for transaction log offset 123" etc. Users could send us recordings with these events and we could dive into them, in order to identify bugs or performance issues.

In the following let’s consider a less complex and hence better approachable example, though. We’ll implement an event for measuring the duration of REST API calls. The Todo service from my recent blog post on Quarkus Qute will serve as our guinea pig. It is based on the Quarkus stack and provides a simple REST API based on JAX-RS. As always, you can find the complete source code for this blog post on GitHub.

Event types are implemented by extending the jdk.jfr.Event class; It already provides us with some common attributes such as a timestamp and a duration. In sub-classes you can add application-specific payload attributes, as well as some metadata such as a name and category which will be used for organizing and displaying events when looking at them in JMC.

Which attributes to add depends on your specific requirements; you should aim for the right balance between capturing all the relevant information that will be useful for analysis purposes later on, while not going overboard and adding too much, as that could cause record files to become too large, in particular for events that are emitted with a high frequency. Also retrieval of the attributes should be an efficient operation, so to avoid any unneccessary overhead.

Here’s a basic event class for monitoring our REST API calls:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 @Name ( JaxRsInvocationEvent . NAME ) (1) @Label ( "JAX-RS Invocation" ) @Category ( "JAX-RS" ) @Description ( "Invocation of a JAX-RS resource method" ) @StackTrace ( false ) (2) public class JaxRsInvocationEvent extends Event { static final String NAME = "dev.morling.jfr.JaxRsInvocation" ; @Label ( "Resource Method" ) (3) public String method ; @Label ( "Media Type" ) public String mediaType ; @Label ( "Java Method" ) public String javaMethod ; @Label ( "Path" ) public String path ; @Label ( "Query Parameters" ) public String queryParameters ; @Label ( "Headers" ) public String headers ; @Label ( "Length" ) @DataAmount (4) public int length ; @Label ( "Response Headers" ) public String responseHeaders ; @Label ( "Response Length" ) public int responseLength ; @Label ( "Response Status" ) public int status ; }

1 The @Name , @Category , @Description and @Label annotations define some meta-data, e.g. used for controlling the appearance of these events in the JMC UI 2 JAX-RS invocation events shouldn’t contain a stacktrace by default, as that’d only increase the size of Flight Recordings without adding much value 3 One payload attribute is defined for each relevant property such as HTTP method, media type, the invoked path etc. 4 @DataAmount tags this attribute as a data amount (by default in bytes) and will be displayed accordingly in JMC; there are many other similar annotations in the jdk.jfr package, such as @MemoryAddress , @Timestamp and more

Having defined the event class itself, we must find a way for emitting event instances at the right point in time. In the simplest case, e.g. suitable for events related to your application logic, this might happen right in the application code itself. For more "technical" events it’s a good idea though to keep the creation of Flight Recorder events separate from your business logic, e.g. by using mechanisms such as servlet filters, interceptors and similar, which allow to inject cross-cutting logic into the call flow of your application.

You also might employ byte code instrumentation at build or runtime for this purpose. The JMC Agent project aims at providing a configurable Java agent that allows to dynamically inject code for emitting JFR events into running programs. Via the EventFactory class, the JFR API also provides a way for defining event types dynamically, should their payload attributes only be known at runtime.

For monitoring a JAX-RS based REST API, the ContainerRequestFilter and ContainerResponseFilter contracts come in handy, as they allow to hook into the request handling logic before and after a REST request gets processed:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 @Provider (1) public class FlightRecorderFilter implements ContainerRequestFilter , ContainerResponseFilter { @Override (2) public void filter ( ContainerRequestContext requestContext ) throws IOException { JaxRsInvocationEvent event = new JaxRsInvocationEvent (); if (! event . isEnabled ()) { (3) return ; } event . begin (); (4) requestContext . setProperty ( JaxRsInvocationEvent . NAME , event ); (5) } @Override (6) public void filter ( ContainerRequestContext requestContext , ContainerResponseContext responseContext ) throws IOException { JaxRsInvocationEvent event = ( JaxRsInvocationEvent ) requestContext . getProperty ( JaxRsInvocationEvent . NAME ); if ( event == null || ! event . isEnabled ()) { return ; } event . end (); (7) event . path = String . valueOf ( requestContext . getUriInfo (). getPath ()); if ( event . shouldCommit ()) { (8) event . method = requestContext . getMethod (); event . mediaType = String . valueOf ( requestContext . getMediaType ()); event . length = requestContext . getLength (); event . queryParameters = requestContext . getUriInfo () . getQueryParameters (). toString (); event . headers = requestContext . getHeaders (). toString (); event . javaMethod = getJavaMethod ( requestContext ); event . responseLength = responseContext . getLength (); event . responseHeaders = responseContext . getHeaders (). toString (); event . status = responseContext . getStatus (); event . commit (); (9) } } private String getJavaMethod ( ContainerRequestContext requestContext ) { String propName = "org.jboss.resteasy.core.ResourceMethodInvoker" ; ResourceMethodInvoker invoker = ( ResourceMethodInvoker ) requestContext . getProperty ( propName ); return invoker . getMethod (). toString (); } }

1 Allows the filter to be picked up automatically by the JAX-RS implementation 2 Will be invoked before the request is processed 3 Nothing to do if the event type is not enabled for recordings currently 4 Begin the timing of the event 5 Store the event in the request context, so it can be obtained again later on 6 Will be invoked after the request has been processed 7 End the timing of the event 8 The event should be committed if it is enabled and its duration is within the threshold configured for it; in that case, populate all the payload attributes of the event based on the values from the request and response contexts 9 Commit the event with Flight Recorder

With that, our event class is pretty much ready to be used. There’s only one more thing to do, and that is registering the new type with the Flight Recorder system. A Quarkus application start-up lifecycle method comes in handy for that:

1 2 3 4 5 6 7 @ApplicationScoped public class Metrics { public void registerEvent ( @Observes StartupEvent se ) { FlightRecorder . register ( JaxRsInvocationEvent . class ); } }