InputStream become closed (fatjar/überjar over OpenJ9_11.0.8.10 debian x64)

Question

I'm facing a strange behavior at my enviroment.

I have a fatjar of my spring application and I running it in a docker container based on adoptopenjdk/openjdk11-openj9:jdk-11.0.8_10_openj9-0.21.0-debian-slim and using the garbage collector G1GC.

The java options are: -Xmx128m -XX:+UseG1GC -Xss256K

The fatjar is generated by maven-assembly-plugin with the jar-with-dependencies descriptor.

My application uses the aws-java-sdk-core:1.11.955 who depends on jackson-core:2.6.7.

My application runs normally, but randomically starts to throw a NoClassDefFoundError. The only way to stop this to continue to happen is restarting my container/application.

This error occurs when the com.amazonaws.internal.config.InternalConfig.Factory static block runs and load a json resource file from the classpath (/com/amazonaws/internal/config/awssdk_config_default.json) and it exists.

This is the stacktrace:

java.lang.NoClassDefFoundError: com.amazonaws.auth.DefaultAWSCredentialsProviderChain (initialization failure)
    at java.base/java.lang.J9VMInternals.initializationAlreadyFailed(J9VMInternals.java:136) 
    ....... (ommited by me, my code calling: new DefaultAWSCredentialsProviderChain())
Caused by: java.lang.IllegalStateException: Fatal: Failed to load the internal config for AWS Java SDK
    at com.amazonaws.internal.config.InternalConfig$Factory.\u003cclinit\u003e(InternalConfig.java:340)
    at com.amazonaws.util.VersionInfoUtils.userAgent(VersionInfoUtils.java:142)
    at com.amazonaws.util.VersionInfoUtils.initializeUserAgent(VersionInfoUtils.java:137)
    at com.amazonaws.util.VersionInfoUtils.getUserAgent(VersionInfoUtils.java:100)
    at com.amazonaws.internal.EC2ResourceFetcher.\u003cclinit\u003e(EC2ResourceFetcher.java:44)
    at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.\u003cinit\u003e(InstanceMetadataServiceCredentialsFetcher.java:36)
    at com.amazonaws.auth.InstanceProfileCredentialsProvider.\u003cinit\u003e(InstanceProfileCredentialsProvider.java:102)
    at com.amazonaws.auth.InstanceProfileCredentialsProvider.\u003cinit\u003e(InstanceProfileCredentialsProvider.java:82)
    at com.amazonaws.auth.InstanceProfileCredentialsProvider.\u003cinit\u003e(InstanceProfileCredentialsProvider.java:66)
    at com.amazonaws.auth.InstanceProfileCredentialsProvider.\u003cclinit\u003e(InstanceProfileCredentialsProvider.java:49)
    at com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper.initializeProvider(EC2ContainerCredentialsProviderWrapper.java:64)
    at com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper.\u003cinit\u003e(EC2ContainerCredentialsProviderWrapper.java:53)
    at com.amazonaws.auth.DefaultAWSCredentialsProviderChain.\u003cinit\u003e(DefaultAWSCredentialsProviderChain.java:49)
    at com.amazonaws.auth.DefaultAWSCredentialsProviderChain.\u003cclinit\u003e(DefaultAWSCredentialsProviderChain.java:43)
    ....... (ommited by me, my code calling: new DefaultAWSCredentialsProviderChain())
Caused by: java.io.IOException: Stream closed
    at java.base/java.util.zip.InflaterInputStream.ensureOpen(InflaterInputStream.java:68)
    at java.base/java.util.zip.InflaterInputStream.read(InflaterInputStream.java:143)
    at java.base/java.io.FilterInputStream.read(FilterInputStream.java:133)
    at com.fasterxml.jackson.core.json.ByteSourceJsonBootstrapper.ensureLoaded(ByteSourceJsonBootstrapper.java:489)
    at com.fasterxml.jackson.core.json.ByteSourceJsonBootstrapper.detectEncoding(ByteSourceJsonBootstrapper.java:126)
    at com.fasterxml.jackson.core.json.ByteSourceJsonBootstrapper.constructParser(ByteSourceJsonBootstrapper.java:215)
    at com.fasterxml.jackson.core.JsonFactory._createParser(JsonFactory.java:1240)
    at com.fasterxml.jackson.core.JsonFactory.createParser(JsonFactory.java:780)
    at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2679)
    at com.amazonaws.internal.config.InternalConfig.loadfrom(InternalConfig.java:250)
    at com.amazonaws.internal.config.InternalConfig.load(InternalConfig.java:263)
    at com.amazonaws.internal.config.InternalConfig$Factory.\u003cclinit\u003e(InternalConfig.java:336)

After debugging the JsonFactory and ByteSourceJsonBootstrapper, there is no code calling the "InputStream.close()".

So I started to thinking this can be a JVM (OpenJ9) bug... Or some strange behavior about the Garbage Collector (G1GC).

Another strange think is, when I run the same jar in local environment, I cannot get this FilterInputStream instance and the InflaterInputStream, I just get a sun.net.www.protocol.jar.JarURLConnection$JarURLInputStream.

Ps.: There is no ClassLoader instrumentation/manipulation.

Can someone give my a light?

Answer 1

You could try adding the following option to the command line to see why the stream is being closed:

"-Xtrace:print=mt,trigger=method{java/util/zip/InflaterInputStream.close,jstacktrace}"