Jetty logs warning with stacktrace when annotation parser encounters module-info.class file inside elasticsearch-x-content jar

[d2a-pnagel]

Jetty version(s)
11.0.13

Java version/vendor (use: java -version)
java version "18.0.1.1" 2022-04-22
Java(TM) SE Runtime Environment (build 18.0.1.1+2-6)
Java HotSpot(TM) 64-Bit Server VM (build 18.0.1.1+2-6, mixed mode, sharing)

OS type/version
Windows 11

Description
When Jetty loads a .war file which includes the elasticsearch, or more specifically, elasticsearch-x-content jar of version 8.2.0 and above, (4) warnings with stack traces are logged to the console by the annotation parser. Example:

[WARNING] Failed to handle org.eclipse.jetty.annotations.AnnotationParser$ClassInfo@1972bfd8
java.lang.NullPointerException: Cannot invoke "Object.hashCode()" because "key" is null
    at java.util.concurrent.ConcurrentHashMap.get (ConcurrentHashMap.java:936)
    at org.eclipse.jetty.annotations.ClassInheritanceHandler.addToInheritanceMap (ClassInheritanceHandler.java:70)
    at org.eclipse.jetty.annotations.ClassInheritanceHandler.handle (ClassInheritanceHandler.java:57)
    at org.eclipse.jetty.annotations.AnnotationParser$MyClassVisitor.visit (AnnotationParser.java:482)
    at org.objectweb.asm.ClassReader.accept (ClassReader.java:569)
    at org.objectweb.asm.ClassReader.accept (ClassReader.java:424)
    at org.eclipse.jetty.annotations.AnnotationParser.scanClass (AnnotationParser.java:897)
    at org.eclipse.jetty.annotations.AnnotationParser.parseJarEntry (AnnotationParser.java:881)
    at org.eclipse.jetty.annotations.AnnotationParser.lambda$parseJar$2 (AnnotationParser.java:840)
    at java.util.TreeMap$ValueSpliterator.forEachRemaining (TreeMap.java:3215)
    at java.util.stream.ReferencePipeline$Head.forEach (ReferencePipeline.java:762)
    at org.eclipse.jetty.annotations.AnnotationParser.parseJar (AnnotationParser.java:836)
    at org.eclipse.jetty.annotations.AnnotationParser.parse (AnnotationParser.java:737)
    at org.eclipse.jetty.annotations.AnnotationConfiguration$ParserTask.call (AnnotationConfiguration.java:159)
    at org.eclipse.jetty.annotations.AnnotationConfiguration$1.run (AnnotationConfiguration.java:540)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob (QueuedThreadPool.java:934)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run (QueuedThreadPool.java:1078)
    at java.lang.Thread.run (Thread.java:833)

After some debugging (the logged ClassInfo.toString is not very helpful here), I discovered that the warning is about the following class files from the elasticsearch-x-content jar file:

• IMPL-JARS/x-content/es-jackson-core-8.6.1.jar/META-INF/versions/9/module-info.class
• IMPL-JARS/x-content/jackson-dataformat-cbor-2.14.1.jar/META-INF/versions/9/module-info.class
• IMPL-JARS/x-content/jackson-dataformat-smile-2.14.1.jar/META-INF/versions/9/module-info.class
• IMPL-JARS/x-content/jackson-dataformat-yaml-2.14.1.jar/META-INF/versions/9/module-info.class

It seems to have been introduced by Elasticsearch in this PR: elastic/elasticsearch#83705 , introducing an embedded classloader. Since then, the IMPL-JARS directory with module-info.class files is included in the elasticsearch-x-content jar file.

Jetty issue #1692 / commit e81e17d addresses a similar issue, skipping module-info.class files on the root level of the jar file.

A few suggestions/considerations:

• Skipping any file with the same name (not path) module-info.class in the scanned jar would effectively get rid of the warning.
• Logging a clear, single line warning reporting the failed jar and class file paths (instead of a stacktrace of a NullPointerException) would make it more clear what the issue and its impact is.
• The annotation parser should probably skip the entire IMPL-JARS directory, but not sure which criteria should be used for this (IMPL-JARS is a name very specific for the elasticsearch-x-content jar file). Maybe Jetty can be configured to skip specific file patterns when scanning for annotations?

How to reproduce?
Run mvn jetty:run in a directory with a pom.xml with content as below.

Actual result:

• Warnings logged to the console

Expected result:

• No warnings logged, or more informative output. Normal startup.

<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://maven.apache.org/POM/4.0.0"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/maven-v4_0_0.xsd">

    <modelVersion>4.0.0</modelVersion>

    <groupId>group</groupId>
    <artifactId>api</artifactId>
    <packaging>war</packaging>
    <version>0.0.1</version>
    <name>Api</name>

    <properties>
        <maven.compiler.source>17</maven.compiler.source>
        <maven.compiler.target>17</maven.compiler.target>
        <jetty-maven-plugin.version>11.0.13</jetty-maven-plugin.version>
        <jetty.version>11.0.13</jetty.version>
    </properties>

    <build>
        <plugins>
            <plugin>
                <groupId>org.eclipse.jetty</groupId>
                <artifactId>jetty-maven-plugin</artifactId>
                <version>11.0.13</version>
                <configuration>
                    <webApp>
                        <war>${project.build.directory}/api.war</war>
                    </webApp>
                    <scan>1</scan>
                </configuration>
            </plugin>
        </plugins>
    </build>

    <dependencies>
        <dependency>
            <groupId>org.elasticsearch</groupId>
            <artifactId>elasticsearch-x-content</artifactId>
            <version>8.2.0</version>
        </dependency>
    </dependencies>
</project>

[janbartel]

@d2a-pnagel while I take a look into this, have a read of https://www.eclipse.org/jetty/documentation/jetty-11/operations-guide/index.html#og-annotations

If you know you've got everything defined in web.xml you don't have to scan, you can set metadata-complete=true to prevent it. Or, for a more fine-grained solution, you can use the org.eclipse.jetty.server.webapp.ContainerIncludeJarPattern context attribute to define patterns for the names of jars effectively in WEB-INF/lib that will be included in the scan - just leave out the elastic search jars.

[d2a-pnagel]

@janbartel thanks for the info. I will look into it. Issue seems to be resolved in the mvn jetty:run situation by setting org.eclipse.jetty.server.webapp.WebInfIncludeJarPattern. Still need to look (will be next week) into setting a context attribute from the .war.

[d2a-pnagel]

Indeed, the warning can be prevented when loading the .war, by adding a jetty context xml file as described in https://www.eclipse.org/jetty/documentation/jetty-11/operations-guide/index.html#og-the-webapp-classpath . That helps. As added bonus, it decreases application startup time.

The jetty context xml file increases configuration maintenance a bit, so adjusting the scanning code is still a nice solution.

[janbartel]

Fixed via #9433