TestKitGen: One Framework to Rule Them All

Apologies in advance, during this brief overview of the test framework that drives the testing of JVMs at OpenJ9 and AdoptOpenJDK projects, I may occasionally lapse into Tolkienesque references.

One of our OpenJ9 developers and expert in all things ‘TestKitGen’-related, Renfei Wang delivers a brief overview of TestKitGen that conveys essential information for any developer needing to test JVMs (at both the Eclipse OpenJ9 or AdoptOpenJDK projects).

While Renfei references the 20,000+ functional tests that live in the Eclipse OpenJ9 repository, this framework drives the various types of testing done at AdoptOpenJDK (6600+ openjdk regression tests, 1000’s of system tests, 10’s of 1000’s application tests, a growing set of performance benchmarks, and 100,000+ JCK tests).

Given the power of this ‘simple’ framework, you can understand the struggle to avoid reciting:

One Ring Framework to rule them all, One Ring Framework to find them,
One Ring Framework to bring them all and in the darkness bind them

I digress…

The beauty of this approach is that based on (much) feedback from developers, it is as transparent and intuitive as possible, a thin veneer to make each of the types of tests (functional, openjdk, system, external, perf and jck) feel similar, despite their great differences.  There is no hiding of what is running, no skirting away of variables or output…  no levels of indirection.  You can see easily what are the generated test targets by looking at the <testCaseName> elements of playlist.xml files found in the test directories.

As a contributor to Eclipse OpenJ9, if you find yourself needing to add, edit, execute, exclude, interpret or understand the tests, Renfei’s overview and the OpenJ9 test documentation can help get you started.  As you run OpenJ9 testing, if you hit issues or encounter problems that cause you to proclaim:

You Shall Not Pass!

Think again… you can and will pass… now that you have a basic understanding of the steps to compile and run tests locally.  If you have questions, please post them at the OpenJ9 #general Slack channel or at the AdoptOpenJDK #testing Slack channel.