Cgroup Metrics – now available in Javacore

Posted on by bharathappali
 
Eclipse OpenJ9 can now provide Linux cgroup information as part of a Java dump file, commonly known as a javacore. This can be very handy to debug clearly by having the information about the environment in which JVM is running and if JVM is aware of the limits imposed by the environment etc. In this article we’ll show you which cgroup information is captured in a javacore for the following scenarios:
 
The cgroup section in a `javacore` can be found before the subcomponent dump and after the CPU Information  section. The first line of the cgroup section (2CICONTINFO)  tells you whether the JVM was running in a container environment. The second line (2CICGRPINFO) tells you whether the JVM was started with container support enabled. Subsequent lines provide the detailed cgroup information. 
 
Container support is enabled by setting the `​-XX:+UseContainerSupport` option on the command line when you run an application (Note: This is turned on by default in the latest releases of OpenJ9). This option turns on some additional intelligence about containerized environments in the OpenJ9 JVM. The JVM allocates a larger fraction of the available memory to the Java heap and is able to detect any limits that are imposed on the container by the host environment.
 
 

I.  An application running on a traditional Linux system

 
In the first scenario, we produced a javacore file on a standard Linux environment by specifying the following `-Xdump` option for the `java -version` 
 

java -Xdump:java:events=vmstop -version

 
Here is the section of the Javacore that contains the cgroup information:
 
 
2CICONTINFO    Running in container : FALSE
2CICGRPINFO    JVM support for cgroups enabled : FALSE
1CICGRPINFO    Cgroup Information 
NULL           ————————————————————————
2CICGRPINFO    subsystem : cpu
2CICGRPINFO    cgroup name : /
3CICGRPINFO        CPU Period : 100000 microseconds
3CICGRPINFO        CPU Quota : Not Set
3CICGRPINFO        CPU Shares : 1024 
3CICGRPINFO        Period intervals elapsed count : 0 
3CICGRPINFO        Throttled count : 0 
3CICGRPINFO        Total throttle time : 0 nanoseconds
2CICGRPINFO    subsystem : cpuset
2CICGRPINFO    cgroup name : /
3CICGRPINFO        CPU exclusive : 1 
3CICGRPINFO        Mem exclusive : 1 
3CICGRPINFO        CPUs : 0-3 
3CICGRPINFO        Mems : 0 
2CICGRPINFO    subsystem : memory
2CICGRPINFO    cgroup name : /
3CICGRPINFO        Memory Limit : Not Set
3CICGRPINFO        Memory + Swap Limit : Not Set
3CICGRPINFO        Memory Usage : 4697415680 bytes
3CICGRPINFO        Memory + Swap Usage : 4697415680 bytes
3CICGRPINFO        Memory Max Usage : 0 bytes
3CICGRPINFO        Memory limit exceeded count : 0 
3CICGRPINFO        Memory + Swap limit exceeded count : 0 
3CICGRPINFO        OOM Killer Disabled : 0 
3CICGRPINFO        Under OOM : 0 
NULL           
 
As expected for this scenario, the first line indicates that the JVM was not running in a container and the second line indicates that the JVM was started without enabling container support. The cgroup information shown describes the various cgroup subsystems and there respective attributes and values.
 
 

II.  Javacore “Cgroup Information Section” on JVM running inside a container with no container limits set and container support not enabled.

 
In this scenario we create a container from a docker image named `openj9`, and pass to it the same command used in scenario 1.
 

docker run -it openj9

java -XX:-UseContainerSupport -Xdump:java:events=vmstop -version

 
Output :
2CICONTINFO    Running in container : TRUE
2CICGRPINFO    JVM support for cgroups enabled : FALSE
1CICGRPINFO    Cgroup Information 
NULL           ————————————————————————
2CICGRPINFO    subsystem : cpu
2CICGRPINFO    cgroup name : /
3CICGRPINFO        CPU Period : 100000 microseconds
3CICGRPINFO        CPU Quota : Not Set
3CICGRPINFO        CPU Shares : 1024 
3CICGRPINFO        Period intervals elapsed count : 0 
3CICGRPINFO        Throttled count : 0 
3CICGRPINFO        Total throttle time : 0 nanoseconds
2CICGRPINFO    subsystem : cpuset
2CICGRPINFO    cgroup name : /
3CICGRPINFO        CPU exclusive : 0 
3CICGRPINFO        Mem exclusive : 0 
3CICGRPINFO        CPUs : 0-3 
3CICGRPINFO        Mems : 0 
2CICGRPINFO    subsystem : memory
2CICGRPINFO    cgroup name : /
3CICGRPINFO        Memory Limit : Not Set
3CICGRPINFO        Memory + Swap Limit : Not Set
3CICGRPINFO        Memory Usage : 26279936 bytes
3CICGRPINFO        Memory + Swap Usage : 26279936 bytes
3CICGRPINFO        Memory Max Usage : 26513408 bytes
3CICGRPINFO        Memory + Swap Max Usage : 26513408 bytes
3CICGRPINFO        Memory limit exceeded count : 0 
3CICGRPINFO        Memory + Swap limit exceeded count : 0 
3CICGRPINFO        OOM Killer Disabled : 0 
3CICGRPINFO        Under OOM : 0 
NULL           
 
In this cgroup section of the javacore file you can see that the dump was produced from a JVM that ran in a container. However, because there was no container limit set we get the default cgroup information.  Also, we did not enable container support.
 

III.  Javacore “Cgroup Information Section” on JVM running inside a container with no container limits set and container support enabled.

 
In this scenario we create a container from a docker image named `openj9`, and add    `-XX:+UseContainerSupport` to the command in scenario 2.
 
docker run -it openj9
java -XX:+UseContainerSupport -Xdump:java:events=vmstop -version
 
Output :
2CICONTINFO    Running in container : TRUE
2CICGRPINFO    JVM support for cgroups enabled : TRUE
1CICGRPINFO    Cgroup Information 
NULL           ————————————————————————
2CICGRPINFO    subsystem : cpu
2CICGRPINFO    cgroup name : /
3CICGRPINFO        CPU Period : 100000 microseconds
3CICGRPINFO        CPU Quota : Not Set
3CICGRPINFO        CPU Shares : 1024 
3CICGRPINFO        Period intervals elapsed count : 0 
3CICGRPINFO        Throttled count : 0 
3CICGRPINFO        Total throttle time : 0 nanoseconds
2CICGRPINFO    subsystem : cpuset
2CICGRPINFO    cgroup name : /
3CICGRPINFO        CPU exclusive : 0 
3CICGRPINFO        Mem exclusive : 0 
3CICGRPINFO        CPUs : 0-3 
3CICGRPINFO        Mems : 0 
2CICGRPINFO    subsystem : memory
2CICGRPINFO    cgroup name : /
3CICGRPINFO        Memory Limit : Not Set
3CICGRPINFO        Memory + Swap Limit : Not Set
3CICGRPINFO        Memory Usage : 26796032 bytes
3CICGRPINFO        Memory + Swap Usage : 26796032 bytes
3CICGRPINFO        Memory Max Usage : 26800128 bytes
3CICGRPINFO        Memory + Swap Max Usage : 26800128 bytes
3CICGRPINFO        Memory limit exceeded count : 0 
3CICGRPINFO        Memory + Swap limit exceeded count : 0 
3CICGRPINFO        OOM Killer Disabled : 0 
3CICGRPINFO        Under OOM : 0 
NULL           
 
In this cgroup section of the javacore file you can see that the dump was produced from a JVM that ran in a container. However, because there was no container limit set we get the default cgroup information.  Also, we have enable container support.
 

IV.  Javacore “Cgroup Information Section” on JVM running inside a container with container limits set and container support enabled

 
In this scenario we create a container from a docker image named `openj9`, and add container limits (-m, –cpu-quota, –cpu-period) to the command in scenario 3.
 
docker run -m2g –cpu-quota=”100000″ –cpu-period=”200000″ -it openj9
java -XX:+UseContainerSupport -Xdump:java:events=vmstop -version
 
Output :
2CICONTINFO    Running in container : TRUE
2CICGRPINFO    JVM support for cgroups enabled : TRUE
1CICGRPINFO    Cgroup Information 
NULL           ————————————————————————
2CICGRPINFO    subsystem : cpu
2CICGRPINFO    cgroup name : /
3CICGRPINFO        CPU Period : 200000 microseconds
3CICGRPINFO        CPU Quota : 100000 microseconds
3CICGRPINFO        CPU Shares : 1024 
3CICGRPINFO        Period intervals elapsed count : 16 
3CICGRPINFO        Throttled count : 2 
3CICGRPINFO        Total throttle time : 164156409 nanoseconds
2CICGRPINFO    subsystem : cpuset
2CICGRPINFO    cgroup name : /
3CICGRPINFO        CPU exclusive : 0 
3CICGRPINFO        Mem exclusive : 0 
3CICGRPINFO        CPUs : 0-3 
3CICGRPINFO        Mems : 0 
2CICGRPINFO    subsystem : memory
2CICGRPINFO    cgroup name : /
3CICGRPINFO        Memory Limit : 2147483648 bytes
3CICGRPINFO        Memory + Swap Limit : 4294967296 bytes
3CICGRPINFO        Memory Usage : 32083968 bytes
3CICGRPINFO        Memory + Swap Usage : 32083968 bytes
3CICGRPINFO        Memory Max Usage : 32083968 bytes
3CICGRPINFO        Memory + Swap Max Usage : 32083968 bytes
3CICGRPINFO        Memory limit exceeded count : 0 
3CICGRPINFO        Memory + Swap limit exceeded count : 0 
3CICGRPINFO        OOM Killer Disabled : 0 
3CICGRPINFO        Under OOM : 0 
NULL           
 
In this cgroup section of the javacore file you can see that the dump was produced from a JVM that ran in a container. As container limits are set the cgroup information is displayed accordingly to the limits set.
 

Summary:

 
This diagnostic information helps developers to debug clearly by knowing if the JVM is running inside the container and if JVM is using container support to consider the limits set to the container. For more on other diagnostic techniques for containerized Java, check out this post on how to gather diagnostic data from your containerized Java application on the fly. To learn more about OpenJ9 enhancements that optimize for running in containerized environments, please read this Eclipse OpenJ9 in Containers article . As always, if you have questions or comments, please connect with us in our OpenJ9 slack workspace.

Leave a Reply