OpenJ9 JCL Preprocessor (JPP) is extended to the extensions for OpenJDK

Posted on by jasonfengj9

OpenJ9 has been using the JPP to produce a custom set of libraries from a single JCL codebase since its inception. The customized libraries are used to build OpenJ9 JDKs. Initially, only the OpenJ9 JCL codebase is structured to support various JPP configurations.

Recently the JPP is extended to preprocess some of the JCL code in the extensions for OpenJDK. For the relation between OpenJ9 and the extensions, refer to What is Eclipse OpenJ9?

This blog first gives a short overview of JPP usage, then shows how it is extended to the extensions.

JPP expressions

There are a set of JPP expressions that appear as part of the JCL source code. Each expression evaluates to either true or false. If you are familiar with the use of #ifdef and #if in C compiler pre-processors then this will be recognizable to you.

The JPP expressions are written directly in the source code as comments.

Decide if a file is included

/*[INCLUDE-IF (JAVA_SPEC_VERSION >= 8) & CRIU_SUPPORT]*/

Only include the following code if the condition is met

/*[IF JAVA_SPEC_VERSION >= 11]*/

Only include the following code if the condition is NOT met

/*[ELSE] JAVA_SPEC_VERSION >= 11 */

End a directive

/*[ENDIF] JAVA_SPEC_VERSION >= 11 */

There are other JPP expression usages that are not covered here.

JPP configuration file

jpp_configuration.xml is used to keep configuration information. JPP first reads this file and loads configurations such as output path, flags, parameters, etc. before it builds a customized library.

A sample configuration for Java 17

	<configuration
		  label="JAVA17"
		  outputpath="JAVA17/src"
		  flags="OPENJDK_METHODHANDLES"
		  dependencies="JAVA11"
		  jdkcompliance="1.8">
		<classpathentry kind="src" path="src/java.base/share/classes"/>
		<classpathentry kind="src" path="src/java.management/share/classes"/>
		<classpathentry kind="src" path="src/jdk.attach/share/classes"/>
		<classpathentry kind="src" path="src/jdk.jcmd/share/classes"/>
		<classpathentry kind="src" path="src/jdk.management/share/classes"/>
		<classpathentry kind="src" path="src/openj9.criu/share/classes"/>
		<classpathentry kind="src" path="src/openj9.cuda/share/classes"/>
		<classpathentry kind="src" path="src/openj9.dataaccess/share/classes"/>
		<classpathentry kind="src" path="src/openj9.dtfj/share/classes"/>
		<classpathentry kind="src" path="src/openj9.dtfjview/share/classes"/>
		<classpathentry kind="src" path="src/openj9.gpu/share/classes"/>
		<classpathentry kind="src" path="src/openj9.jvm/share/classes"/>
		<classpathentry kind="src" path="src/openj9.sharedclasses/share/classes"/>
		<classpathentry kind="src" path="src/openj9.traceformat/share/classes"/>
		<classpathentry kind="src" path="src/openj9.zosconditionhandling/share/classes"/>
		<classpathentry kind="lib" path="/binaries/common/ibm/ibmjzos.jar"/>
		<classpathentry kind="lib" path="/binaries/vm/third/rt-compressed.sunJava17.jar"/>
		<source path="src"/>
		<parameter name="macro:define" value="JAVA_SPEC_VERSION=17"/>
		<parameter name="msg:outputdir" value="java.base/share/classes/com/ibm/oti/util"/>
		<parameter name="jxerules:outputdir" value="java/lang"/>
	</configuration>

The set of flags defined by flags= and those inherited from dependencies= are used in JPP expressions. JAVA_SPEC_VERSION is a special variable, always available, that equates to the version being built. Otherwise, only the flags found in the jpp_configuration.xml  (and the -tag:define command line option) are valid for use in expressions. Anything else results in an error when pre-processing.

JPP is built from source

JPP is built from the source in each OpenJ9 build.

A sample log output

Building OpenJ9 Java Preprocessor
Building /home/jenkins/workspace/Build_JDK17_x86-64_linux_OpenJDK17/build/linux-x86_64-server-release/support/j9tools/jpp.jar

JPP headless

OpenJ9 builds run JPP headless to generate the JCL code for its JDK level. As explained above, the source code is augmented by JPP expressions that define various subsets for the supported java versions and components.

Command line to preprocess OpenJ9 JCL source files

java -cp jpp.jar -Dfile.encoding=US-ASCII com.ibm.jpp.commandline.CommandlineBuilder -verdict -config JAVA17 -baseDir openj9-openjdk-jdk17/openj9/ -srcRoot jcl/ -xml jpp_configuration.xml -dest build/linux-x86_64-server-release/support/j9jcl -tag:define CRIU_SUPPORT;PLATFORM-xa64

The -config JAVA17 corresponds to <configuration label="JAVA17" in jpp_configuration.xml.

The flags defined by -tag:define are added in addition to the flags="OPENJDK_METHODHANDLES" that is part of the JPP configuration, and also those inherited from dependencies="JAVA11".

Command line to preprocess DDR source files

java -cp jpp.jar -Dfile.encoding=US-ASCII com.ibm.jpp.commandline.CommandlineBuilder -verdict -config GENERIC -baseDir openj9-openjdk-jdk17/openj9/debugtools/DDR_VM/ -srcRoot src/ -xml jpp_configuration.xml -dest build/linux-x86_64-server-release/support/j9jcl/openj9.dtfj/share/classes -tag:define CRIU_SUPPORT;PLATFORM-xa64

JPP Plugin

The JPP Plugin is an Eclipse front-end to the OpenJ9 JCL Preprocessor.

The JPP plugin generates a pConfig Eclipse Java project by using the outputpath of the config i.e. pConfig JAVA17 for the JAVA17 configuration, which contains the preprocessed OpenJ9 Java code for Java 17.
Any changes in OpenJ9 JCL are preprocessed and the pConfig project is updated as source files are saved.

JPP is extended to the Extensions for OpenJDK for OpenJ9

All source files within the closed folder are preprocessed by JPP.

java -cp jpp.jar -Dfile.encoding=US-ASCII com.ibm.jpp.commandline.CommandlineBuilder -verdict -config JAVA17 -baseDir openj9-openjdk-jdk17/ -srcRoot closed/ -xml jpp_configuration.xml -dest build/linux-x86_64-server-release/support/j9jcl -tag:define CRIU_SUPPORT;PLATFORM-xa64 -includeIfUnsure -noWarnIncludeIf

All source files copied into an overlay folder are preprocessed by JPP.

java -cp jpp.jar -Dfile.encoding=US-ASCII com.ibm.jpp.commandline.CommandlineBuilder -verdict -config JAVA17 -baseDir openj9-openjdk-jdk17/build/linux-x86_64-server-release/support/ -srcRoot overlay/ -xml jpp_configuration.xml -dest build/linux-x86_64-server-release/support/j9jcl -tag:define CRIU_SUPPORT;PLATFORM-xa64 -includeIfUnsure -noWarnIncludeIf

Because JPP doesn’t support the whole OpenJDK codebase, the selected Java files to be preprocessed by JPP are required to be copied using the following script. If you want to preprocess a new file, modify this script code snippet.

$(eval $(call SetupCopyFiles,COPY_OVERLAY_FILES, \
      SRC := $(TOPDIR), \
      DEST := $(SUPPORT_OUTPUTDIR)/overlay, \
      FILES := \
            src/java.base/share/classes/java/lang/ClassValue.java \
            src/java.base/share/classes/java/security/Security.java \
            src/java.base/share/classes/java/util/Timer.java \
            src/java.base/share/classes/java/util/TimerTask.java \
            src/java.base/share/classes/java/util/concurrent/ConcurrentHashMap.java \
            src/java.base/unix/classes/java/lang/ProcessEnvironment.java \
      ))

There is a special case to handle StandardCharsets.java since it is generated from StandardCharsets.java.template separately by GensrcCharsetMapping.gmk.

      $(CP) $(CHARSET_GENSRC_JAVA_DIR_BASE)/StandardCharsets.java $(SUPPORT_OUTPUTDIR)/overlay-gensrc/src/java.base/sun/nio/cs/
      $(call RunJPP, JAVA$(VERSION_FEATURE), $(SUPPORT_OUTPUTDIR)/overlay-gensrc, /overlay-result)
      $(CP) $(SUPPORT_OUTPUTDIR)/overlay-result/java.base/sun/nio/cs/StandardCharsets.java $(CHARSET_GENSRC_JAVA_DIR_BASE)/
java -cp jpp.jar -Dfile.encoding=US-ASCII com.ibm.jpp.commandline.CommandlineBuilder -verdict -config JAVA17 -baseDir openj9-openjdk-jdk17/build/linux-x86_64-server-release/support/ -srcRoot overlay-gensrc/ -xml jpp_configuration.xml -dest build/linux-x86_64-server-release/support/overlay-result -tag:define CRIU_SUPPORT;PLATFORM-xa64 -includeIfUnsure -noWarnIncludeIf

Summary

OpenJ9 JPP is extended to preprocess some selected folders and files within the OpenJ9 extensions of OpenJDK. Note the JPP Eclipse plugin doesn’t preprocess those files yet.

If you need a hand adding java files to be preprocessed by JPP, feel free open an OpenJ9 issue or reach out to me directly.

References:

[1] CRIU JPP for unix/classes/java/lang/ProcessEnvironment.java
[2] Apply JPP to JCL patch files within $(TOPDIR)/closed
[3] Add IncludeIfUnsure for $(SUPPORT_OUTPUTDIR)/overlay

Leave a Reply