Configure Java Flight Recorder in WSO2 Products

This blog post explains how to configure JFR with WSO2 products.

What is a Java Flight Recorder?

In day-today life we are encountering considerable amount of performance issues in production systems mainly due to huge traffic. In order to identify the root cause of the issue, we need a way to figure out what went wrong within the past few hours. Therefore the Java Flight Recorder (JFR) is the ideal tool which can be used to find the potential performance issues during testing.

JFR captures and dumps the diagnostic and profiling data about a running Java application. It is integrated into the Java Virtual Machine (JVM) and causes almost no performance overhead (typically less than 2%), so it can be used even in heavily loaded production environments.

JFR can be enabled by default and continuously collect low level data from Java applications in production environments. When default settings are used, both internal testing and customer feedback indicate that the performance impact is less than one percent. For some applications, it can be significantly lower. However, for short-running applications (which are not running in production environments), relative startup and warmup times can be larger, which might impact the performance by more than one percent.

JFR collects events that occur in the JVM when the Java application runs. The collected events include thread latency events such as sleep, wait, lock contention, I/O, GC, and method profiling. JFR saves data about the events in a single output file, flight.jfr.

How to configure Java Flight Recording?

In order to use JFR, we should activate it. We may achieve this in two ways:

1. When starting a Java application.
2. Passing diagnostic commands of the jcmd tool when a Java application is already running.

JFR doesn’t have a standalone tool. Therefore, we use Java Mission Control (JMC), which contains a plugin that allows us to visualize the data collected by JFR.

Note: If we have various versions of Java installed on our computer, it’s important to make sure that the Java compiler (javac), the Java launcher (Java) and the components (JFR, jcmd and JMC) are from the same Java distribution. Otherwise, there’s a risk of not being able to see any useful data because the JFR data formats of different versions might be not compatible.

Here we are discussing how to enable JFR in WSO2 products.

There are few ways to enable the JFR. Since JFR is a commercial feature, available only in the commercial packages based on the Java Platform, Standard Edition (Oracle Java SE Advanced and Oracle Java SE Suite), we must enable commercial features before trying to run JFR.

For that, we need to add the following code lines to the wso2server.sh file which resides in the bin directory of the WSO2 server and start the server.

-XX:+UnlockCommercialFeatures \
-XX:+FlightRecorder \

If we are using Java 8 update 40 (8u40) or later, we can now dynamically enable JFR. So, we don’t need to have above arguments if we need to run JFR. This really helps if we are seeing a problem and there is no need to restart the server.

How to run Java Flight Recorder?

There are two types of recordings available in JFR. They are;

1. Time Fixed Recordings
2. Continuous Recordings

Time Fixed Recording — This type is used to profile the Java application for a given duration. JFR will collect events for a fixed duration and then it will dump the recording to the specified file. If the recording was started in JMC, the recording will be opened automatically.

Continuous Recordings — This recording type gets a Flight Recording dump only when we see an issue. This type does not have an end time. JFR collects events and write data to a global buffer. When this global buffer fills up, the oldest data is discarded. When we request a dump, we will get the data which currently available in the buffer. We can specify maximum buffer size and the maximum allowed age of events.

We can run multiple recordings concurrently and configure each recording using different settings. Moreover, we can configure different recordings to capture different sets of events. However, in order to make the internal logic of JFR as streamlined as possible, the resulting recording always contains the union of all events for all recordings active at that time. That means we can gather more information in the recording than we wanted if more than one recording is running.

There are two ways we can run JRF. Those are,

1. Using the command line
2. Using the Diagnostic Command

Using the Command Line

We can start both the Time Fixed Recording and Continuous recording using the command line options as below.

Starting a Time Fixed Recording (Profiling Recording)

We can start and configure the Time Fixed Recording also known as Profiling Recording by using the “-XX:StartFlightRecording” option of the Java command in the application startup. Please make sure to enable the commercial features and enable the Flight Recorder before start the JFR.

Following example shows how to start a Profiling Recording of “JavaApp” application for 60 seconds after 20 seconds of starting the JVM. The output will be saved to “recording.jfr” file.

-XX:StartFlightRecording=delay=20s,duration=60s,name=Test, filename=recording.jfr,settings=profile JavaApp

Starting a Continuous Recording (Default Recording)

We can start the Continuous Recording also known as the Default Recording by using “-XX:FlightRecorderOptions=defaultrecording=true”.

In Continuous Recording we can dump data only if needed. There are two options to limit the amount of data needed;

1. Set the maximum size (in bytes) of disk data to keep for the default recording using the maxsize parameter.
2. Set the maximum age of disk data to keep for the default recording using the maxage parameter.

If both size limit and an age are specified, the data is deleted when either limit is reached.

Following is an example of starting a Continuous Recording with 2 hours of data. The temporary data will be stored in the disk.

-XX:FlightRecorderOptions=defaultrecording=true,disk=true, repository=/tmp,maxage=2h JavaApp

Continuous Recording data can be dumped either manually using JMC or diagnostic command. In order to dump the recording, we need to specify a new location as the files in the repository are only temporary. Or else we can dump get a dump on exit.

We can use the following options to get a flight recording when the program exits.

-XX:FlightRecorderOptions=defaultrecording=true,dumponexit=true, dumponexitpath=/tmp/dumponexit.jfr

Using the Diagnostic Command

The diagnostic command can be executed using the jcmd tool which is located in the Java installation directory. To issue a command, we have to pass the process identifier of the JVM (or the name of the main class) and the actual command as arguments to jcmd.

Following is an example to start a 60-second recording of the running Java process and save it to myrecording.jfr in the current directory;

jcmd `cat wso2carbon.pid` JFR.start duration=60s filename=myrecording.jfr

The commands relevant to the Java Flight Recorder are:

• JFR.start: Start a recording.
• JFR.check: Check the status of all recordings running for the specified process, including the recording identification number, file name, duration, and etc.
• JFR.stop: Stop a recording with a specific identification number (by default, recording 1 is stopped).
• JFR.dump: Dump the data collected so far from the recording with a specific identification number (by default, data from recording 1 is dumped).
• VM.check_commercial_features: Check whether commercial features are unlocked
• VM.check_commercial_features: Show the status of individual features

Visualize Data

JFR runtime engine dumps recorded data to files with *.jfr extension. These binary files can be viewed from Java Mission Control (JMC), which is part of the JDK distribution. This helps to visualize the data about the events in an intuitive way. Following is a sample *.jfr dump file.

sample *.jfr file

The main screen shows the information about how the program was using the CPU during its execution. The other tab groups show certain aspects of the JVM and the Java application runtime such as Memory, Code, Threads, I/O, System and Events.

• General: Details of the JVM, the system, and the recording.
• Memory: Information about memory & garbage collection.
• Code: Information about methods, exceptions, compilations, and class loading.
• Threads: Information about threads and locks.
• I/O: Information about file and socket I/O.
• System: Information about environment
• Events: Information about the event types in the recording

I hope you will find this information useful. :)