DDR (Direct Dump Reader) is a Java implementation of the DTFJ (Diagnostic Tooling Framework for Java) API. It works by walking the J9 structures inside a dump to extract the VM and application state. It’s useful for inspecting Java objects at a certain point of execution of a program. For example, you can list all the threads and inspect the stack trace of a thread.
In Java 19, the Virtual Thread was introduced by Project Loom. A virtual thread is a lightweight implementation of java.lang.Thread which allows people to write highly concurrent applications. In contrast to a platform thread, a virtual thread is not tied to any OS thread but is mounted on a platform thread. A virtual thread can be unmounted if a blocking operation is encountered, and then its continuation is no longer associated with any OS thread.
With more and more applications using virtual threads, supporting virtual threads in DDR will become essential for debugging programs with virtual threads. To support virtual threads, three new DDR commands were added: vthreads, continuationstack and continuationstackslots.
An Example
We will introduce the new command with the following example:
public static void main(String[] args) throws InterruptedException {
Thread t1 = Thread.ofVirtual().name("vthread 1").start(() -> {
System.out.println("virtual thread 1 starts");
LockSupport.park();
System.out.println("virtual thread 1 ends");
});
while (t1.getState() != Thread.State.WAITING) {
Thread.sleep(500); // let vthread 1 park
}
Thread t2 = Thread.ofVirtual().name("vthread 2").start(() -> {
System.out.println("virtual thread 2 starts");
com.ibm.jvm.Dump.systemDumpToFile();
LockSupport.unpark(t1);
System.out.println("virtual thread 2 ends");
});
t1.join();
t2.join();
}In the example, two virtual threads are created and one of them is parked, thus unmounted.
In order to get the desired core file, you may need to run the example with -Xdump:system:request=exclusive+prepwalk option.
The vthreads Command
The vthreads command lists all the virtual threads in the core file.
Let’s open the generated core file with jdmpview, and call the vthreads command by typing !vthreads and we get the output:
> !vthreads
!continuationstack 0x00007f1d94003410 !j9vmcontinuation 0x00007f1d94003410 !j9object 0x00000007FFF9A848 (Continuation) !j9object 0x00000007FFF9A7A0 (VThread) - vthread 2
!continuationstack 0x00007f1d94010110 !j9vmcontinuation 0x00007f1d94010110 !j9object 0x00000007FFF8C598 (Continuation) !j9object 0x00000007FFF8C490 (VThread) - vthread 1In each line, four commands are listed to:
- inspect the stack trace of the continuation
- inspect the native continuation struct
- inspect the continuation object
- inspect the virtual thread object
respectively. The name of the virtual thread will also be shown at the end of the line (if the name is set).
The vthreads command is usually the first step that you want to do after opening the core file. You can identify the virtual thread that you want to work on either by looking at the names or by checking the stack trace which we will cover in the following section.
The continuationstack and continuationstackslots Commands
The stack trace is useful for quickly obtaining diagnostic information after a crash or at a certain point of program execution. The continuationstack command outputs the stack trace of a continuation. Running !continuationstack 0x00007f1d94003410 (the continuation of virtual thread 2) will output:
> !continuationstack 0x00007f1d94003410
<7f1d94003410> !j9method 0x00000000000A58F0 jdk/internal/vm/Continuation.enterImpl()Z
<7f1d94003410> !j9method 0x00000000000A5770 jdk/internal/vm/Continuation.run()V
<7f1d94003410> !j9method 0x0000000000096118 java/lang/VirtualThread.runContinuation()V
<7f1d94003410> !j9method 0x000000000026D388 java/lang/VirtualThread$$Lambda/0x0000000000000000.run()V
<7f1d94003410> !j9method 0x000000000026F388 java/util/concurrent/ForkJoinTask$RunnableExecuteAction.exec()Z
<7f1d94003410> !j9method 0x000000000026E108 java/util/concurrent/ForkJoinTask.doExec()I
<7f1d94003410> !j9method 0x000000000025E440 java/util/concurrent/ForkJoinPool$WorkQueue.topLevelExec(Ljava/util/concurrent/ForkJoinTask;Ljava/util/concurrent/ForkJoinPool$WorkQueue;)V
<7f1d94003410> !j9method 0x000000000025BC10 java/util/concurrent/ForkJoinPool.scan(Ljava/util/concurrent/ForkJoinPool$WorkQueue;II)I
<7f1d94003410> !j9method 0x000000000025BBF0 java/util/concurrent/ForkJoinPool.runWorker(Ljava/util/concurrent/ForkJoinPool$WorkQueue;)V
<7f1d94003410> !j9method 0x00000000001FFF30 java/util/concurrent/ForkJoinWorkerThread.run()V
<7f1d94003410> JNI call-in frame
<7f1d94003410> Native method frameNote that this is actually the stack trace of the carrier thread. This is because virtual thread 2 is mounted and during the mounting process, some of the continuation’s variables, such as the stack pointer, are swapped with the carrier thread’s. So the continuationstack command will output the stack trace of the carrier thread for a mounted virtual thread. To view the stack trace of the virtual thread, use the stack command with the carrier thread:
> !stack 0x0023d700
<23d700> !j9method 0x0000000000286A20 com/ibm/jvm/Dump.triggerDumpsImpl(Ljava/lang/String;Ljava/lang/String;)Ljava/lang/String;
<23d700> !j9method 0x00000000002868C0 com/ibm/jvm/Dump.triggerDump(Ljava/lang/String;Ljava/lang/String;)Ljava/lang/String;
<23d700> !j9method 0x00000000002867C0 com/ibm/jvm/Dump.systemDumpToFile()Ljava/lang/String;
<23d700> !j9method 0x0000000000246800 VTTest2.lambda$main$1(Ljava/lang/Thread;)V
<23d700> !j9method 0x0000000000246DE8 VTTest2$$Lambda/0x000000001436ed78.run()V
<23d700> !j9method 0x00000000000961D8 java/lang/VirtualThread.runWith(Ljava/lang/Object;Ljava/lang/Runnable;)V
<23d700> !j9method 0x00000000000961B8 java/lang/VirtualThread.run(Ljava/lang/Runnable;)V
<23d700> !j9method 0x000000000026C770 java/lang/VirtualThread$VThreadContinuation$1.run()V
<23d700> !j9method 0x00000000000A5750 jdk/internal/vm/Continuation.enter(Ljdk/internal/vm/Continuation;)V
<23d700> JNI call-in frame
<23d700> Native method frameFor an unmounted virtual thread, the continuationstack command will output the stack trace of the virtual thread:
> !continuationstack 0x00007f1d94010110
<7f1d94010110> !j9method 0x00000000000A5910 jdk/internal/vm/Continuation.yieldImpl(Z)Z
<7f1d94010110> !j9method 0x00000000000A57B0 jdk/internal/vm/Continuation.yield0()Z
<7f1d94010110> !j9method 0x00000000000A5790 jdk/internal/vm/Continuation.yield(Ljdk/internal/vm/ContinuationScope;)Z
<7f1d94010110> !j9method 0x0000000000096298 java/lang/VirtualThread.yieldContinuation()Z
<7f1d94010110> !j9method 0x0000000000096378 java/lang/VirtualThread.park()V
<7f1d94010110> !j9method 0x00000000000B9198 java/lang/Access.parkVirtualThread()V
<7f1d94010110> !j9method 0x0000000000278110 jdk/internal/misc/VirtualThreads.park()V
<7f1d94010110> !j9method 0x000000000008F2C8 java/util/concurrent/locks/LockSupport.park()V
<7f1d94010110> !j9method 0x0000000000246820 VTTest2.lambda$main$0()V
<7f1d94010110> !j9method 0x0000000000246AE8 VTTest2$$Lambda/0x000000001436ec28.run()V
<7f1d94010110> !j9method 0x00000000000961D8 java/lang/VirtualThread.runWith(Ljava/lang/Object;Ljava/lang/Runnable;)V
<7f1d94010110> !j9method 0x00000000000961B8 java/lang/VirtualThread.run(Ljava/lang/Runnable;)V
<7f1d94010110> !j9method 0x000000000026C770 java/lang/VirtualThread$VThreadContinuation$1.run()V
<7f1d94010110> !j9method 0x00000000000A5750 jdk/internal/vm/Continuation.enter(Ljdk/internal/vm/Continuation;)V
<7f1d94010110> JNI call-in frame
<7f1d94010110> Native method frameAdditionally, similar to the stack and stackslots commands, we also have the continuationstackslots command which outputs slots along with the stack trace.