Ezio Melotti @ Flickr CC 2.0

Stackdriver Profiler is a dynamic code analysis tool. It allows you to collect information about your program as it runs, which in turn, helps you to discover possible bugs and problems directly on production. It’s so called sampling profiler which gathers the data as samples, creating minimum load overhead on your production system.

Depending on the programming language, multiple types of profiling are available (CPU, Heap, Allocated Heap, Contention, Threads, Wall time). In case of JVM we can use CPU Time , Heap and Wall Time (where Heap is only available for Java 11+)

CPU Time — the time CPU was running while executing your function, it doesn’t count the time the CPU was idle while in that function.

Wall Time — the total time spend while executing the function (including waiting for locks and other resources). It is always ≥ CPU Time.

Heap — amount of memory allocated on the heap space. With this metric you can observe if your blocks of code don’t consume excessive amounts of memory or if they hold on memory for too long.

Heap allocation — total amount of memory that was allocated during the program execution (including the one which has been freed).

Threads — information about threads created, blocked, threads created and never run (thread leaks). Not available for JVM at the time of writing.

Contention — time spent waiting by a thread for a shared resource. Not available for the JVM at the time of writing.

Stackdriver Profiler gathers so called Profiles . A Profile is a single snapshot of your running application instance where data is collected over ~10 seconds with 1 minute interval. Collecting data in this way reduces performance impact on the running production application to less than 5%. Taking into account multiple replicas of your service you usually have running on production environment, the impact on performance is usually less than 0.5%.

To gather the data from our production applications we need to install the agent on each one of them. The agent collects the data which you can browse on the Google Console web interface.

In this tutorial, we switch to Java 11 to enable Heap data collection, as well as create a service which will consume the memory by creating multiple copies of an object faster than the finalizer function on those objects can be called by garbage collector.

App Modifications

Modifications we need to make to enable collecting data for Stackdriver Profiler are a bit troublesome but luckily just a tiny bit, at least in case where we deploy our application instances on a Kubernetes cluster. We need to somehow embed the Stackdriver Profiler agent with our release artifacts as well as provide JVM arguments for the running program inside Docker container. On top of all, we need to modify our Docker file using sbt as the docker image is build by it.

There are 4 points you need to do, all in build.sbt file:

1.Add some commands to the docker file so our image will be created with the agent included.

Those commands will create a directory, download the agent tar.gz into it and unpack it for us to use. It created the following entries in Dockerfile :

2. Pass JVM arguments for the running instance of your application so appropriate flags will be set for the Stackdriver agent installed.

More information about profiling Java applications with Stackdriver Profiler can be found here.

Important flags for us are:

-cprof_service=stackdriver-test5-profiler and -cprof_service_version=1.0.0 — setting those two flags will allow you to browse profiles by service name and service version; as we are using sbt you can of course modify it to be dynamically replaced by the app name and the currently released version.

-cprof_enable_heap_sampling=true — this flag is available only when using Java 11+ VM and will profile heap memory for our running app.

3. Choose your base docker image type wisely — there are reports that running Stackdriver profiler agent will not work when used together with some docker base images, like alpine example. I have choosen openjdk:11-jdk and it works fine.

4. Remember to use Java 11+ in your project if you want to have Heap profiling on the Stackdriver web console.

Create the Cluster

Same as in the previous parts of this series we are creating a simple Kubernetes cluster with Stackdriver enabled using gcloud commands:

NAME=stackdriver-test3

ZONE=us-west2-a gcloud config set compute/zone $ZONE

gcloud container clusters create $NAME --num-nodes=2 --enable-stackdriver-kubernetes

Release the App

Download GCP-Goodies repository, navigate to part-7 and execute sbt clean release to build your project and release the image to Google Container Registry. Remember to change the softwaremill-playground-2 to the project name you use on the GCP.

kubectl create deployment stackdriver-test3-profiler --image=eu.gcr.io/softwaremill-playground-2/play-scala-stackdriver-profiler:1.5

Expose our application to the world for easier testing:

kubectl expose deployment stackdriver-test3-profiler --type=LoadBalancer --port 80 --target-port 9000

If something gets wacky you can always rebuild, release and update the pod version if necessary:

kubectl set image deployment/stackdriver-test3-profiler play-scala-stackdriver-profiler=eu.gcr.io/softwaremill-playground-2/play-scala-stackdriver-logging:1.6

After the successful release we can run our memory intensive endpoint:

Navigate to Google Cloud console and select Profiler and Stackdriver section. You should be able to see the latest profiles collected already.

As you can see, we can filter our collected profiles by service name, version, zone etc., as well as observe 3 main types like CPU, Wall time and Heap memory (all of the available types for the JVM).

Another nice feature is the ability to focus on specific block of code/function which highlights the frame on the chart for us:

Example frames from the Stackdriver Profiler for our executememoryIntensiveOperation function in ProfilerService implementation.

Profiling can help you discover new information about your applications and sometimes can be an invaluable tool in your toolbox. Watching the changes in heap memory or analyzing CPU/Wall times in a very granular way, especially on the production system with minimum overhead on performance due to sampling nature of Stackdriver profiler, can open a whole new world of possibilities. I hope that with this post, installing an agent and starting up a profiler on any JVM based application running in Kubernetes will become a breeze.

Happy Profiling!