How to quickly detect and remove log4j classes from our code base and the base image? "mvn dependency:tree" does not check base image

Question

We are building an app based on Red Hat JBoss AMQ 6. We wrap some Java code around the base image to provide extra functionalities which are lacking in AMQ 6.

Now, when the CVE of Log4j stroke, we found that this component is vulnerable because it uses log4j 1.x. Now I don't just mean our Java code uses it, but also the base image of Red Hat AMQ 6 uses it. As AMQ 6 is EOL now, Red Hat does not provide support anymore, so there will not be official releases with fix.

So, how do I remove vulnerable log4j 1.x classes from:

• my Java code
• the base image

?

I am building with jib maven plugin.

It seems mvn dependency:tree only gives info about our Java wrapper code, not about the base image. And I don't understand what does the "+" and "-" mean while it gives output like this:

[INFO] +- org.jboss.resteasy:resteasy-jaxrs:jar:3.7.0.Final:compile
[INFO] |  +- org.jboss.spec.javax.ws.rs:jboss-jaxrs-api_2.1_spec:jar:1.0.2.Final:compile
[INFO] |  +- org.jboss.spec.javax.xml.bind:jboss-jaxb-api_2.3_spec:jar:1.0.1.Final:compile
[INFO] |  +- org.reactivestreams:reactive-streams:jar:1.0.2:compile
[INFO] |  +- javax.validation:validation-api:jar:2.0.1.Final:compile
[INFO] |  +- org.jboss.spec.javax.annotation:jboss-annotations-api_1.3_spec:jar:1.0.1.Final:compile
[INFO] |  +- javax.activation:activation:jar:1.1.1:compile
[INFO] |  +- org.apache.httpcomponents:httpclient:jar:4.5.4:compile
[INFO] |  |  +- org.apache.httpcomponents:httpcore:jar:4.4.7:compile
[INFO] |  |  +- commons-logging:commons-logging:jar:1.2:compile
[INFO] |  |  \- commons-codec:commons-codec:jar:1.10:compile
[INFO] |  +- commons-io:commons-io:jar:2.5:compile
[INFO] |  +- net.jcip:jcip-annotations:jar:1.0:compile
[INFO] |  \- org.jboss.logging:jboss-logging:jar:3.3.2.Final:compile

Does + mean that it can be expanded further but is not shown here?

Some background here: https://nvd.nist.gov/vuln/detail/CVE-2021-44228

Answer 1

Step 1: Dealing with our wrapping Java code base

For our code, we do:

mvn dependency:tree | grep log4j

And we found some dep from other teams bringing in transitive log4j 1.17. Informed that team and they fixed that in a recent version, we just change in our pom the version to be it, and our pom is fixed.

If your dependency is not maintained anymore, you can enter the artifactory of your organization, and manually look for the classes below in all jars you need(the list is long, because apart from CVE-2021-4104 which mentions JMSAppender, I found log4j 1.x has a lot of other vulnerabilities and more classes should be removed)

• org/apache/log4j/net/SocketServer.class
• org/apache/log4j/net/SimpleSocketServer.class (just in case)
• org/apache/log4j/net/SocketAppender.class
• org/apache/log4j/net/SMTPAppender$1.class
• org/apache/log4j/net/SMTPAppender.class
• org/apache/log4j/net/JMSAppender.class
• org/apache/log4j/net/JMSSink.class
• org/apache/log4j/net/JDBCAppender.class
• org/apache/log4j/chainsaw/*.class

If you cannot fix your internal Nexus repo/artifactory, you can find the jar of log4j in local Maven registry (under ~/.m2) and remove the class; then you build your app again; but remember don't use -U to redownload the jar from remote registry.

Step 2: Dealing with base image jars

To find other libs in the base image containing log4j is more complicated.

• Tampering the layers by removing the classes files cannot go undetected by Docker daemon. The sha256 value changes, you have to replace the sha256 value in the json file in the main dir with new sha256sum layer.tar; but even with that, Docker daemon will give error when you load the tar: Cannot open /var/lib/docker/tmp-xxxx/...: file not found or so.

• Then I tried to create a script to remove the classes at runtime, right before running the app, and define a new entrypoint in jib to run it before running the app.

#!/bin/sh
/opt/amq/bin/fix_log4j_1.x_cves.sh
/opt/amq/bin/launch.sh # the original, inherited entrypoint in jib

But then I found it will slow down pod startup; unresponsive pods may be restarted by Openshift, causing unwanted delay and errors. But the output of this script gives me an idea of which jars contain the classes to remove, which is a solid basis for my next solution.

At last, I came up with a perfect solution

1. Implement the previous solution, docker run the image, and note down the jars' names in the output of the script.

Starting to fix all CVEs regarding Log4j 1.x...
>>>>> Removing class file from '/opt/amq/lib/optional/log4j-1.2.17.redhat-1.jar':
removed 'org/apache/log4j/chainsaw/ControlPanel$1.class'
removed 'org/apache/log4j/chainsaw/ControlPanel$2.class'
...
>>>>> Removing class file from '/opt/amq/activemq-all-5.11.0.redhat-630495.jar':
...

2. Define these two as provided dependencies in pom.xml
3. Use maven copy-dependency plugin to copy them into build folder (by default target/dependency)
4. Use maven exec plugin to run the same script against target/dependency dir to remove vulnerable classes, while building the image with jib
5. Use maven jib plugin to copy the fixed jars into the container, so that they will be on a new layer on top of all previous layer, to shadow/whiteout the unfixed jar (see answer to my other question)

By doing this, we eliminate the vulnerable classes while building the image, pod startup speed is not compromised, and the binary transferred to production image registry is already safe.

An advantage of this approach is we are not limited by available tools provided by the container because the script runs now in our local environment. We can install whatever tool we need and use them in the script. For example, in the original script I defined function extract_remove_repackage to complete a simple task of extracting+remove classes+repackaging, only because zip is not installed in the base image. But in my local machine, this can be done by zip in one line,

You have to make sure to bind 3, 4 and 5 to different maven build phases, so that they happen in such order. I bind 3) to compile, 4) to process-classes and 5) to package.

---

Implementation details below:

• The script (in my previous solution, put under src/main/jib/opt/amq/bin, so it could be copied into container. Also you needed the new entrypoint script here in the same folder. Now in this solution, moved to src/main/scripts)

fix_log4j_1.x_cves.sh:

#!/bin/bash
# Script to fix log4j 1.x CVEs. Initially it is only for CVE-2021-4104, but
# since there are multiple CVEs regarding log4j 1.x, they are all fixed here:

# Class File                                        CVE
# org/apache/log4j/net/SocketAppender.class         CVE-2019-17571
# org/apache/log4j/net/SocketServer.class           CVE-2019-17571
# org/apache/log4j/net/SMTPAppender$1.class         CVE-2020-9488
# org/apache/log4j/net/SMTPAppender.class           CVE-2020-9488
# org/apache/log4j/net/JMSAppender.class            CVE-2021-4104
# org/apache/log4j/net/JMSSink.class                CVE-2022-23302
# org/apache/log4j/net/JDBCAppender.class           CVE-2022-23305
# org/apache/log4j/chainsaw/*.class                 CVE-2022-23307

cves=(
'CVE-2019-17571'
'CVE-2019-17571'
'CVE-2020-9488'
'CVE-2020-9488'
'CVE-2021-4104'
'CVE-2022-23302'
'CVE-2022-23305'
'CVE-2022-23307'
)

size() {
    stat -c %s "$1"
}

extract_remove_repackage() {
    before=$1
    # jar xf -C some_dir only extract to current dir, we have to cd first
    jar_dir=$(dirname "$2")
    jar_file=$(basename "$2")
    temp_dir=$jar_dir/temp
    mkdir "$temp_dir"
    cp list.txt "$temp_dir"/ && cp "$2" "$temp_dir"/
    cd "$temp_dir"
    jar xf "$jar_file"
    # provide file and dir names to rm with list.txt
    xargs rm -rvf < list.txt && rm list.txt "$jar_file"
    jar cf "$jar_file" .
    mv "$jar_file" ../
    # go back and clean up
    cd "$before" && rm -rf "$temp_dir"
}

find_vulnerable_jars() {
    cd "$root_dir"
    jar -tvf "$1" | grep -E "$pattern" | awk '{ print $8 }' > list.txt
    if [ "$(size list.txt)" -gt 0 ]; then
        echo ">>>>> Removing class file from '$(realpath "$1")'":
        extract_remove_repackage "$(pwd)" "$1"
    else
        return 0
    fi
}
remove_classes_from_jars() {
    echo Starting to fix all CVEs regarding Log4j 1.x...
    # exclude jolokia.jar(link)
    # xargs can return error level to "if", when any of execution fails, while "find -exec" cannot
    # because we use custom function, xargs needs "bash -c"; thus we have to use "_" to pass each arg
    if find "$root_dir" -name "*.jar" -not -type l -print0 | xargs -0 -n1 bash -c 'find_vulnerable_jars "$@"' _; then
        echo All vunerable classes removed. CVE addressed:
        printf '%s\n' "${cves[@]}"
    else
        echo "Error while removing classes; exiting..."
        return 1
    fi
}

# to be able to use in find -exec child shell, we need to export all vars and functions
# $1: where to search jars, should match copy-dependency output dir.
export root_dir=$1
export pattern=".*(JMS|JDBC|SMTP|Socket)Appender.*.class|.*SocketServer.class|.*JMSSink.class|org/apache/log4j/chainsaw/.*"
export -f size
export -f extract_remove_repackage
export -f find_vulnerable_jars
remove_classes_from_jars

• Define provided dependencies:

<dependency>
    <groupId>org.apache.activemq</groupId>
    <artifactId>activemq-all</artifactId>
    <version>${version.activemq-all}</version>
    <scope>provided</scope>
</dependency>
<dependency>
    <groupId>log4j</groupId>
    <artifactId>log4j</artifactId>
    <version>${version.log4j}</version>
    <scope>provided</scope>
</dependency>

• copy-dependency plugin

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-dependency-plugin</artifactId>
    <executions>
        <execution>
            <id>copy-cve-jars</id>
            <phase>compile</phase>
            <goals>
                <goal>copy-dependencies</goal>
            </goals>
            <configuration>
                <includeArtifactIds>activemq-all,log4j</includeArtifactIds>
                <includeScope>provided</includeScope>
                <includeTypes>jar</includeTypes>
                <outputDirectory>${project.build.directory}/dependency</outputDirectory> <!-- default value -->
                <excludeTransitive>true</excludeTransitive>
            </configuration>
        </execution>
    </executions>
</plugin>

• exec-plugin:

<plugin>
    <artifactId>exec-maven-plugin</artifactId>
    <groupId>org.codehaus.mojo</groupId>
    <executions>
        <execution>
            <id>remove-cve-classes</id>
            <phase>process-classes</phase>
            <goals>
                <goal>exec</goal>
            </goals>
        </execution>
    </executions>
    <configuration>
        <executable>${project.build.scriptSourceDirectory}/log4j_cve_fix.sh</executable>
        <arguments>
            <!-- should match copy-dependency output dir -->
            <argument>${project.build.directory}/dependency</argument>
        </arguments>
    </configuration>
</plugin>

jib plugin: (needs to be > 3.0.0 to be able to use <path><inclueds>)

<plugin>
    <groupId>com.google.cloud.tools</groupId>
    <artifactId>jib-maven-plugin</artifactId>
    <configuration>
        <from>
            <image>${docker.base.image}</image>
        </from>
        <to>
            <image>${docker.image}</image>
            <tags>
                <tag>${project.version}</tag>
            </tags>
        </to>
        <container>
            <appRoot>/dev/null</appRoot>
            <entrypoint>INHERIT</entrypoint> <!-- customized entrypoint not needed anymore, just revert to the way it was -->
        </container>
        <containerizingMode>packaged</containerizingMode>
        <extraDirectories>
            <paths>
                <path>${project.basedir}/src/main/jib</path>
                <path>${project.build.directory}/jib</path>
                <path>
                    <from>target/dependency</from>
                    <into>/opt/amq/lib/optional</into>
                    <includes>log4j-${version.log4j}.jar</includes>
                </path>
                <path>
                    <from>target/dependency</from>
                    <into>/opt/amq</into>
                    <includes>activemq-all-${version.activemq-all}.jar</includes>
                </path>
            </paths>
            <permissions>
                <permission>
                    <!-- don't forget to restrict writing to prevent tampering -->
                    <file>/opt/amq/conf/log4j.properties</file>
                    <mode>444</mode>
                </permission>
                <!-- the copied jars need to be executable -->
                <permission>
                    <file>/opt/amq/lib/${application.executable}</file>
                    <mode>755</mode>
                </permission>
                <permission>
                    <file>/opt/amq/activemq-all-${version.activemq-all}.jar</file>
                    <mode>755</mode>
                </permission>
                <permission>
                    <file>/opt/amq/lib/optional/log4j-${version.log4j}.jar</file>
                    <mode>755</mode>
                </permission>
            </permissions>
        </extraDirectories>
    </configuration>
    <executions>
        <execution>
            <id>jib-build</id>
            <phase>package</phase>
            <goals>
                <goal>${jib.goal}</goal>
            </goals>
        </execution>
    </executions>
</plugin>

Answer 2

Remember to always check for the latest information from the resources listed below

---

Answering the question directly:
Checking Log4J dependencies in code:

• I think WesternGun's answer is fine... but personally I think the easiest thing to do is probably to just build your app (if you haven't already) and then recursively search the built application's directory structure for JAR files matching the REGEX log4j-core-2.([0-9]+\.){1,2}jar (will detect versions vulnerable to CVE-2021-45046 ... CVE-2021-44228 ... CVE-2021-45105). If you want to detect older versions as well (which have their own critical severity CVEs and also need to be upgraded) then the REGEX would just be log4j and you'd have to manually figure out which specific jars are the vulnerable ones.
  • it may actually be able to be refined further... but i'm not sure of the specific jar file is the problem for these other CVEs: CVE-2019-17571 and CVE-2021-4104
• Reddit thread: log4j_0day_being_exploited cntl+f for .class and .jar recursive hunter will give you some tools to help with this recursive search

Detecting Log4J use on running applcation (in container or not doesn't matter):

• Go to Reddit thread: log4j_0day_being_exploited and cntl+f for Vendor Advisories. Search the lists there for any software/plugins you are running. If you are running something in the list and there is an update available, update.
• Then go to the same website and cntl+f for Vulnerability Detection. Use the tools there. If you detect the vulnerability, remediate.
• Then go to the same website and cntl+f for Exploitation Detection. Use the tools there. These will detect if you have already been attacked. If you detect that you have, then remediate and respond to that attack as necessary.

---

More resources

• https://www.reddit.com/r/blueteamsec/comments/rd38z9/log4j_0day_being_exploited/
  • This one has TONS of useful info including detectors, even more resource links, very easy to understand remediation steps, and more
• https://www.cisa.gov/uscert/apache-log4j-vulnerability-guidance
• https://github.com/cisagov/log4j-affected-db
• https://logging.apache.org/log4j/2.x/security.html

---

Remediation:
CVE-2021-45046 ... CVE-2021-44228 ... CVE-2021-45105
While most people that need to know probably already know enough to do what they need to do, I thought I would still put this just in case...

• Follow the guidance in those resources... it may change, but

As of 2021-12-18

It's basically

• Remove log4j-core JAR files if possible
  • From both running machines for immediate fix AND
  • in your source code / source code management files to prevent future builds / releases / deployments from overwriting the change
• If that is not possible (due to a dependency), upgrade them
  • If you are running Java8, then you can upgrade to log4j 2.17.0+
  • If you are running an earlier version of Java, then you can upgrade to log4j 2.12.3
  • If you are running an older version of Java, then you need to upgrade to the newest version of Java, and then use the newest version of Log4J
  • Again, these changes have to happen both on running machine and in code
• If neither of those are possible for some reason... then there is the NON-remediation stop gap of removing the JndiLookup.class file from the log4j-core JARs.
  • There is a one-liner for the stop gap option on Linux using the zip command that comes packaged with most Linux distros by default.
    • zip -q -d "$LOG4J_JAR_PATH" org/apache/logging/log4j/core/lookup/JndiLookup.class
  • At time of writing, most of the guides online for the stop gap option on Windows say to do the following (again... assuming you can't do one of the remove JAR or upgrade options above):
    • Install something like 7-zip
    • Locate all of your log4j-core JAR files and for each one do the following...
    • Rename the JAR to change the extension to .zip
    • Use 7-zip to unzip the JAR (which now has a .zip extension)
    • Locate and remove the JndiLookup.class file from the unzipped folder
      • The path is \\path\\to\\unzippedFolder\\org\\apache\\logging\\log4j\\core\\lookup\\JndiLookup.class
    • Delete the old JAR file (which now has an extension of .zip)
    • Use 7-zip to RE-zip the folder
    • Rename the new .zip folder to change the extension to .jar
  • There are also some options to use Power Shell
    • Reddit thread: log4j_0day_being_exploited
    • ctrl+f for "PowerShell"

This is fine if you only have 1 or 2 JAR files to deal with and you don't mind installing 7-zip or you have PowerShell available to do it. However, if you have lots of JAR files, or if you don't want to install 7-zip and don't have access to Power Shell, I created an open-source VBS script that will do it for you without needing to install any additional software. https://github.com/CrazyKidJack/Windowslog4jClassRemover

Read the README and the Release Notes https://github.com/CrazyKidJack/Windowslog4jClassRemover/releases/latest

Answer 3

You can use:

mvn dependency:tree -Dincludes=*log4j*

It will find any dependencies and transitive dependencies having "log4j" anywhere in its groupId.

Example of output:

\- org.springframework.boot:spring-boot-starter-web:jar:2.6.0:compile
[INFO]    \- org.springframework.boot:spring-boot-starter:jar:2.6.0:compile
[INFO]       \- org.springframework.boot:spring-boot-starter-logging:jar:2.6.0:compile
[INFO]          \- org.apache.logging.log4j:log4j-to-slf4j:jar:2.14.1:compile
[INFO]             \- org.apache.logging.log4j:log4j-api:jar:2.14.1:compile

Each pattern segment is optional and supports full and partial * wildcards. An empty pattern segment is treated as an implicit wildcard.

For example, org.apache.* would match all artifacts whose group id started with org.apache., and :::*-SNAPSHOT would match all snapshot artifacts.

See also maven doc

EDIT

You then most probably would want then exclude these dependencies with:

<dependency>
    <groupId>your dep groupId</groupId>
    <artifactId>your dep artifactId</artifactId>
    <exclusions>
        <exclusion>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-api</artifactId>
        </exclusion>
        <exclusion>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-to-slf4j</artifactId>
            </exclusion>
    </exclusions>
</dependencies>

Notes

At the time of writting, it is any version < 2.17.1

Log4j versions with vulnerabilites are available on Maven Repository