Debugging OpenJ9 in Docker with GDB

Recently, one of my OpenJ9 commits caused intermittent segmentation faults with Open Liberty. I created the test environment to recreate the failure in a Docker container, and I decided to use GDB to further investigate and identify the source of the segmentation fault. To my surprise, I found that GDB doesn’t work by default in a Docker container. After some researching, I was finally able to get GDB to work in a Docker container. When I attached the Java process to GDB, the native stack was unresolved with no references to source code. NO …. feeling frustrated … another blocker. After digging around, I found the steps to compile OpenJ9 with debug symbols. After recompiling OpenJ9 with debug symbols, I again attach the Java process to GDB. Now, the native stack was resolved but still no references to source code. Banging my head on my desk … another blocker … haven’t even started investigating the segmentation fault … a whole day wasted. Next day, I found that “–strip-debug” lines need to be removed from makefiles to get references to source code in GDB. After removing “–strip-debug” lines, I recompile OpenJ9 and attach the Java process to GDB. Finally, I could see the native stack with references to source code. Tears of joy in my eyes.

I decided to write this blog so others can easily setup GDB and OpenJ9 for debugging without any suffering and frustration. Please follow the steps below to debug OpenJ9 with GDB in a Docker container.

Building OpenJ9 with debug symbols

1) Download the Linux 64-bit (x86_64) DockerFile to build OpenJDK8 with OpenJ9.

wget https://raw.githubusercontent.com/eclipse/openj9/master/buildenv/docker/jdk8/x86_64/ubuntu16/Dockerfile

2) Create a docker image using the above DockerFile.

docker build -t openj9-jdk8 -f Dockerfile .

3) Create a docker container.

docker run --privileged --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -it openj9-jdk8

4) Get openj9-openjdk-jdk8 source.

git clone https://github.com/ibmruntimes/openj9-openjdk-jdk8

5) Get OpenJ9 and OMR source.

cd openj9-openjdk-jdk8
bash ./get_source.sh

6) Set environment variables to enable debugging.

export UMA_DO_NOT_OPTIMIZE_CCODE="1"
export VMDEBUG="-g3 -fno-inline -O0"
export VMLINK="-g -O0"
export enable_optimized=no
export enable_optimize=no
export CXXFLAGS="-O0 -g3"
export CFLAGS="-O0 -g3"
export BUILD_CONFIG=debug

For Java 9+, the Just-In-Time (JIT) compiler crashes with debug symbols and assertions enabled. A workaround is to disable debug symbols and assertions for the JIT by not setting the BUILD_CONFIG environment variable. This issue is being tracked over here: https://github.com/eclipse/openj9/issues/2088.

7) Remove “–strip-debug” lines from Linux makefiles.

sed -i.bak "/--strip-debug/d" openj9/runtime/makelib/targets.mk.linux.inc.ftl
sed -i.bak "/--strip-debug/d" omr/omrmakefiles/rules.linux.mk

sed should remove “–strip-debug” from the makefile and create a backup (*.bak) for the original makefile.

8) Run ./configure with “–with-debug-level=slowdebug”.

bash ./configure --with-freemarker-jar=/root/freemarker.jar --with-debug-level=slowdebug

9) Build OpenJDK8.

make all

10) After the build has completed,

>>>> Java 8 <<<<

• JDK should be available here: “/root/openj9-openjdk-jdk8/build/linux-x86_64-normal-server-slowdebug/images/j2sdk-image“
• Debug symbols should be available here: “/root/openj9-openjdk-jdk8/build/linux-x86_64-normal-server-slowdebug/vm“
• JDK’s build directory: “/root/openj9-openjdk-jdk8/build/linux-x86_64-normal-server-slowdebug/images/j2sdk-image/jre/lib/amd64/compressedrefs“

>>>> Java 9+ <<<<

• JDK should be available here: “/root/openj9-openjdk-jdk/build/linux-x86_64-normal-server-slowdebug/images/jdk“
• Debug symbols should be available here: “/root/openj9-openjdk-jdk/build/linux-x86_64-normal-server-slowdebug/vm“
• JDK’s build directory: “/root/openj9-openjdk-jdk/build/linux-x86_64-normal-server-slowdebug/images/jdk/lib/compressedrefs“

Setting up GDB

1) Download package list from the repositories.

apt-get update

2) Install GDB.

apt-get install gdb

3) Only for Java 8, copy shared libraries with debug symbols into JDK’s build directory.

cp /root/openj9-openjdk-jdk8/build/linux-x86_64-normal-server-slowdebug/vm/*.so* /root/openj9-openjdk-jdk8/build/linux-x86_64-normal-server-slowdebug/images/j2sdk-image/jre/lib/amd64/compressedrefs/.

For Java 9+, the above step is not needed.

4) Run a Java process with GDB.

gdb --args /root/openj9-openjdk-jdk8/build/linux-x86_64-normal-server-slowdebug/images/j2sdk-image/bin/java ...

Now, GDB is setup and ready for debugging OpenJ9.