So, a new year and a new set of challenges are already here. Over the past few weeks, I have been trying to understand how to profile the software performance of applications while they are running in production. While there are a huge number of softwares that are already available for running software performance profiling, one of the major challenges of doing this in production environment in comparison to your local development machine or testing environment is the fact that these additional softwares may introduce a considerable overhead in terms of performance for the application being profiled. This overhead may appear in terms of increased memory utilisation or increased usage of CPU cycles.

So, with the basic picture in place, let me try to elaborate on what I have been busy exploring in the past few weeks. The major target for my exploration has been the profiling of application written using the Java platform. These software applications targeting the JAVA platform usually don’t compile into binaries, but are rather executed inside the virtual machine known as Java Virtual Machine (JVM). Once the developer has finished writing the code, they usually compile that code into class files by using the tools provided by the Java Development Kit (JDK). These generated class files consist of the bytecode that can be understood by the JVM and can be executed.

Now, to understand about the performance of these applications running inside the JVM, we can take multiple approaches:

• Monitor system metrics: Once the application is launched, we can utilise a certain set of tools that are provided by the operating system to monitor the application resource utilisation. These metrics consist of the CPU usage by the application, how much memory the application is using and how many threads are being spawned by the application. But these metrics are too high level and do not help us to gather the knowledge of how exactly our code is behaving and which part of the code is spending more time executing on the CPU and which thread is running for too long. To answer these questions, we need something which can go more deeper into the JVM.
• Profiling codebase using annotations: Code annotations provide a useful option when we want to profile our whole application or certain parts of it. Usually, we annotate the code paths which we want to profile and rest of the task is handled by the profiling framework which we are using (for example, spf4j). The problem with this type of approach is that, the profiling code is also executed alongside our application code inside the same JVM. This causes an increased overhead and also limits the extent of profiling that we may want to do in an application deployed in production.
• Native profiling: Now, there is another option. How about getting the performance metrics directly from the JVM directly. For example, getting to know about when JVM starts executing a certain function or schedules a thread or allocates an object on the heap. This kind of profiling can be very useful and if it can be executed outside of the bounds of the regular constraints of the JVM, it can prove to be a low overhead option for running performance profiling in a production environment.

In this post, we will take a look at one of the ways through which we can exercise the third type of option.

Introducing JVMTI

The JVM Tool Interface (JVMTI) is a programming interface which helps us interact and control the execution of a JVM. While a lot of the APIs exposed by the JVMTI are very powerful and are already being used by a number of useful tools such as debuggers for the JVM or code coverage analyzers, these APIs can also be used to understand and monitor the application performance while not generating too much overhead in the production phase.

To utilise the JVMTI, the developers can build agents which can listen to the events of their interest, which are generated by the JVM and then act on those events. These agents can be built as a static library or as a full fledged java agent which can be attached to the JVM.

Once the agent has been loaded, the agent executes inside the same JVM in which the application is running and communicates with the application through the use of the native interface.

This post serves the purpose of introducing how to build a super simple agent using the JVMTI interface and how to load it in our Java application.

What We Are Going to Build

The aim of this post will be to write very simple Java agent using the JVMTI libraries and will introduce us to the basic concepts of the APIs exposed by JVMTI. The agent we build in this sample will do nothing more than printing the names of the methods that are being executed by JVM while running a sample Java program.

So, let’s get started.

Setting Up the Development Environment

For the purpose of this tutorial, we are going to use Red Hat Enterprise Linux. If you are using some other Operating system, you might want to adjust these steps a little bit based on your environment.

• The first requirement will be to have the JDK on your system. For this tutorial, you will need to have OpenJDK installed on your machine. Execute the following command to get the required packages on your development machine:

sudo yum install java-1.8.0-openjdk java-1.8.0-openjdk-devel

• Now, with the JDK setup done, we will need a few more tools that can compile our native code into a shared library. For this, we are going to settle of with the GNU GCC compiler collection. If not already on the system, you can install it by running the following command:

sudo yum install gcc

To validate if our package install worked fine, let’s execute the below commands one by one:

gcc --version

java -version

javac -version

If everything works fine, we are ready to go ahead 😀

Writing the Agent

So, now it’s the time for us to get our hands dirty. Let’s start with the development of our JVMTI based agent. Let’s create a new file named sample_agent.c in which we are going to write all of our code.

To build the JVMTI based agent, we need to have access to the functions and features exposed by JVMTI. The definitions of these functions is provided in the “jvmti.h” header file that comes as a part of the openjdk-devel package we installed.

Also, we are going to use a certain set of C library functions for handling memory allocation for certain variables and printing of text to our console. So, we may also want to include the header files which provide the declaration for these functions.

So, the first part of our agent code needs to include the required header files:

#include <stdio.h>

#include <stdlib.h>

#include "jvmti.h"

Now, we have the header file inclusion sorted. With this done, we can now look into how to handle the start-up of our agent inside the JVM. The JVMTI based agents can be attached to the JVM by passing the -agentlib or -agentpath command line option to the JVM during while starting the JVM. These agents once attached, can be started during either the OnLoad phase or the live phase of the JVM. For this article, we are going to target the OnLoad phase of the JVM.

For our agent to startup during the OnLoad phase, the agent needs to export a definition for the Agent_OnLoad function as defined by the JVMTI. The function has the following method signature:

JNIEXPORT jint JNICALL

Agent_OnLoad(JavaVM *jvm, char *options, void *reserved);

When the JVM starts, it calls this function to start up the agent. This call is made quite early in the JVM startup cycle when:

• No bytecode has been executed yet
• No classes have been loaded
• No objects have been created
• Full set of JVM capabilities are available
• System properties can still be set

The first parameter of the function call provides a reference to the JavaVM object which recognises the JVM to which the agent is attached and allows the agent to communicate with the JVM. The second parameter, options is populated with the options that were passed to the JVM at the time of the startup. The third parameter is a reserved parameter and not in use currently.

The return value indicated by the Agent_OnLoad if other than zero, indicates an error and causes the JVM to terminate.

One interesting fact to note here is, nearly all the methods exported by JVMTI returns an integer as a value which is used to signify an error in the operation. We can utilise this to our advantage and build a single function that can help us log the errors in our agent.

So, let’s move on to implement a function that can check the errors for us and provide us with some kind of output when one occurs. We call this function, “check_jvmti_errors“. The definition of the function looks like:

static void check_jvmti_errors(jvmtiEnv *jvmti, jvmtiError errnum, const char *str) {

    if (errnum != JVMTI_ERROR_NONE) {

       char *errnum_str;

        errnum_str = NULL;

        (void) (*jvmti)->GetErrorName(jvmti, errnum, &errnum_str);

        printf("ERROR: JVMTI: [%d] %s - %s", errnum, (errnum_str == NULL ? "Unknown": errnum_str), (str == NULL? "" : str));

    }

}

In the definition of the above function, we take in 3 parameters, namely, a pointer pointing to the current JVMTI Environment(more on this shortly), a error number as returned by the JVMTI operation and a custom string that we want to pass as a helpful message.

Inside the function, we first check if the errnum, actually points to some error or not. If it points to an error, we call the GetErrorName function of the jvmti environment passing it, the current jvmti environment pointer, the error number for which the error name needs to be retrieved and the pointer to which the function can copy the name of the error.

Now, with the common error checking function implemented, the next thing to implement is our Agent_OnLoad function. So, let’s get going.

Since, we have already seen how the signature of the function looks like, we can now get to the definition directly:

JNIEXPORT jint JNICALL

Agent_OnLoad(JavaVM *jvm, char *options, void *reserved) {

    jvmtiEnv *jvmti;

    jvmtiError error;

    jint res;

    jvmtiEventCallbacks callbacks;

    jvmtiCapabilities capa;

    jrawMonitorID rawMonitorID;



    // Get the JVMTI environment

    res = (*jvm)->GetEnv(jvm, (void **) &jvmti, JVMTI_VERSION_1_0);

    if (res != JNI_OK || jvmti == NULL) {

        printf("Unable to get access to JVMTI version 1.0");

    }

    (void) memset(&capa, 0, sizeof(jvmtiCapabilities));

    // Let's initialize the capabilities

    capa.can_generate_method_entry_events = 1;

    error = (*jvmti)->AddCapabilities(jvmti, &capa);

    check_jvmti_errors(jvmti, error, "Unable to add the required capabilities");

    // Setup event notification

    error = (*jvmti)->SetEventNotificationMode(jvmti, JVMTI_ENABLE, JVMTI_EVENT_METHOD_ENTRY, (jthread) NULL);

    check_jvmti_errors(jvmti, error, "Unable to set the event notification mode");

    // Setup the callbacks

    (void) memset(&callbacks, 0, sizeof(callbacks));

    callbacks.MethodEntry = &callbackMethodEntry;

    error = (*jvmti)->SetEventCallbacks(jvmti, &callbacks, (jint) sizeof(callbacks));

    check_jvmti_errors(jvmti, error, "Unable to set event callbacks");

    // Get the raw monitor

    error = (*jvmti)->CreateRawMonitor(jvmti, "JVMTI agent data", &rawMonitorID);

    check_jvmti_errors(jvmti, error, "Unable to create a Raw monitor");

    return JNI_OK;

}

Now, we are done with the definition of our Agent_OnLoad function. Let’s try to understand what we have done there.

As a first, we try to get the JVMTI environment which we can use from the running JVM. For this to happen, we make a call to the GetEnv function of the JVM passing it the pointer where it should return the JVMTI version and the version of JVMTI Environment which we want to load.

Once, we have the environment, we can now work on to set the capabilities that our JVMTI need to have. Since the JVM is still in its early initialization phase, we can set these capabilities quite easily.

For our example, the only capability we need is that the JVM should be able to generate method entry events. We set this capability by setting:

capa.can_generate_method_entry_events = 1

Once we have set the value, we can make a call to the AddCapabilities function of the JVMTI environment passing it the capabilities structure.

Now, with the capabilities set, the next thing to do is to setup the event notifications. To do this, we make a call to the JVMTI environment’s SetEventNotificationMode function. We tell the function to enable(JVMTI_ENABLE) the notifications for the method entry event (JVMTI_EVENT_METHOD_ENTRY).

Once the notifications are set, we can now register our callback function, which will be called everytime a method entry event is generated. This is done by setting up the required value in the callbacks structure:

callbacks.MethodEntry = &callbackMethodEntry

and then passing this structure to the SetEventCallbacks function of the JVMTI environment.

With all of this done, let’s do the final step and create a raw monitor which can be used to take care of the cases when we want to enter into critical sections which may need to be synchronized. This can be done by making a call to the CreateRawMonitor function.

Now, one thing to note here is that, we have setup a callback function which will be called everytime a method starts executing inside the JVM. But, we haven’t provided a definition for this callback yet. So, let’s move on and define the callback function.

Defining the Callback function

Inside the world of JVMTI, for every function that can handle a certain JVMTI event, the JVMTI has already provided a function signature. Our method entry callback is no exception, which has the following signature defined:

void JNICALL

MethodEntry(jvmtiEnv *jvmti, JNIEnv *jni, jthread thread, jMethodID method);

The signature defines multiple parameters, which are passed to the callback when a method entry event occurs. These parameters include, a pointer to the current jvmti environment, a pointer to the JNI environment inside the JVM, a thread id of the thread where the method is executing and the id of the method that is executing.

Now, for our example, we are going to print the name of the method that is being entered. For this, our callback definition looks something like the one given below:

static void JNICALL

MethodEntry(jvmtiEnv *jvmti, JNIEnv *jni, jthread thread, jMethodID method) {

    char *name_ptr;

    char *signature_ptr;

    char *generic_ptr;



    jvmtiError error;

    error = (*jvmti)->GetMethodName(jvmti, method, &name_ptr, &signature_ptr, &generic_ptr);

    check_jvmti_errors(jvmti, error, "Unable to get the method name");

    printf("Entered method %s\n", name_ptr);

}

Inside our definition, we didn’t do too many things. We just defined some pointers to a char array and then made a call to the GetMethodName function of the jvmti environment. The function provides us with three values for the passed method id, namely, the name of the method, the signature of the method, and the generic signature of the method. All these values are pointed by the pointers passed to the function call.

Once we have these details, we simply print the name of the function by making a call to printf.

With this done, we are now ready to compile our agent into a shared library.

Creating a Shared Library

Once we have developed the agent, we need to convert it into a static library which the JVM can load. This can be done quite easily. But before that, we need to find out the location of where our jvmti.h header file is located. To do this, execute the following command:

find / -type f -name "jvmti.h"

and note the directory where the header file is located.

For example, if my header file is located inside /usr/lib/java-1.8.0-openjdk/include/jvmti.h, then I need to remember the following path: /usr/lib/java-1.8.0-openjdk/include. This path will be required while making a call to our compiler.

Now, we are ready to create an object file out of our C code. This can be done by running the following command:

gcc -Wl, -g -fno-strict-aliasing -fPIC -fno-omit-frame-pointer -W -Wall  -Wno-unused -Wno-parentheses -I '/usr/lib/java-1.8.0-openjdk/include/' -I '/usr/lib/java-1.8.0-openjdk/include/linux' -c -o sample_agent.o sample_agent.c

After this step, we will have an object file named sample_agent.o which we can use to build our shared library. To build this shared library, execute the following command:

gcc -shared -o libsample_agent.so sample_agent.o

This will take up our sample_agent.o object file and perform the required linking and will provide us with the shared library.

Once this is done, we are now ready to take our sample agent for a test run. But for that, we need a sample program. So, let’s quickly build a sample program.

Building a Sample Program

To test our JVMTI agent, we need to have a sample program that we can run it on. The following code sample, provides us the sample program which we may want to run. Let’s call this file HelloWorld.java with the following contents inside it:

class HelloWorld {

    public static void main(String[] args) {

       System.out.println("Hello world!");

    }

}

Let’s compile this program by executing the following command:

javac HelloWorld.java

Once the command is executed, we now have a classfile which we can execute along with our agent.

Executing Our Agent

Now, we have the required shared library which contains our agent code as well as a sample program which we can test the agent with. Assuming that we are inside a directory which contains both our JVMTI agent as well as the Java classfile which we want to execute, here is the command which you may want to execute:

LD_LIBRARY_PATH=`pwd` java -agentlib:sample_agent HelloWorld

Here, the LD_LIBRARY_PATH tells java the path to find the shared libraries it may require. The -agentlib parameter is used to tell the name of the shared library that needs to be executed.

Once the command is executed, we get to see an output which resembles the one given below:

Entered <init>

Entered <init>

Entered <init>

Entered init

Entered currentThread

Entered getSecurityManager

Entered checkAccess

Entered getSecurityManager

Entered addUnstarted

Entered isDaemon

Entered getPriority

Entered toCharArray

And with this, we are now done with building and running our sample JVMTI agent.

What’s Next?

In this article we saw how we can build a simple JVMTI agent and how it can be used to understand the execution of our application inside the JVM. One major thing to note in this example was, we loaded the agent at the start of our application by passing the -agentlib parameter to our JVM. But, what if we wanted to load this agent into a running JVM instead.

In the next article, we will take a look at how we can load an agent into a running JVM to get us the required data while also understanding the performance impact of using a JVMTI based agent.

Till then, happy hacking! 🙂