How to create a jar of jars including only required 3rd party libraries outside of war file

Question

Environment:

• Java 17
• Tomcat 9.x
• Maven 3.8.4 (with Codehaus Cargo plugin)
• Gitlab CI/CD

I'm looking for input from anyone that has separated all of their 3rd party libraries into catalina lib (or a custom directory specified via extending catalina.properties 'common-loader') so that the application's war file can be considerably smaller.

Background: We have five webapps which share about 100 3rd party libraries that combined are about 100MB in size. While the applications may have a few unique dependencies the overlap in 3rd dependencies is over 90%.

Our code itself is only 5-15MB and te dependencies don't change frequently compared to our own code. So there's a lot of unnecessary file transfer and unpacking each day/week in our CI/CD.

I'm wanting to improve both deploy startup times as well as remote deploy times (via cargo) as deploying ~115MB remotely to a cluster seems unnecessarily time-consuming given < 15% of the code changes on a typical sprint.

My plan would be to:

1. Create a new git project and move all the dependencies that are currently in a parent pom to the new project
2. Update the parent pom to reference the new project as a dependency but set scope as provided so those jars aren't included in the war file.
3. Write a maven script to package all the 3rd party jars into a fat jar (not a shaded jar)
4. Write a sftp/shell script (or use a Maven ssh plugin) to publish the fat jar to the production and staging servers into $(catalina)/lib or a custom directory (added to the common-loader attribute) whenever libraries have version or bug updates which require publishing.
5. The existing maven cargo task should run much faster; needing to only transfer 10-15MB. And I suspect that Tomcat should deploy the updated war files faster too since there is so much less to unpack.

Anyone have experience doing this or have any tips/gotchas to be concerned with?

Answer 1

I have a similar situation, which requires common libraries to be on Tomcat classpath. While, there are potentially many approaches, we follow the below process.

We kind of 'build' the Tomcat, not from source code perspective but create directories and put the JARs in specific folders (within catalina.base directory). We have a profile in pom which packages the dependencies of the libraries during building the library.

<profile>
    <id>make-bundle</id>
    <build>
        <plugins>
            <!-- 
            1) Copies the project 'compile' dependencies to the staged_lib directory
            under the 'target' directory.
            -->
            <plugin>
                <artifactId>maven-dependency-plugin</artifactId>
                <executions>
                    <execution>
                        <id>copy-dependencies-compile</id>
                        <phase>package</phase>
                        <goals>
                            <goal>copy-dependencies</goal>
                        </goals>
                        <configuration>
                            <outputDirectory>${staged.lib.directory}</outputDirectory>
                            <includeScope>compile</includeScope>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</profile>

The dependency in the WAR for the libraries is 'provided'. The WAR does not pack the library JARs or their dependencies.

We then use a script to copy the generated library JARs and their dependencies in the specific folders in the Tomcat. The process is a bit long to explain, below is the script that we use, it is self-explanatory:

#!/bin/bash

#Declare colour constants

BRed='\033[1;31m'
NC='\033[0m'


# This batch file takes an existing aRCO Tomcat installation and prepares another new Tomcat installation with 
# new updated platform JAR files. The build process steps for the Tomcat are as follows:
# 1) Unzip the tomcat zip file as found in the 'tczippath' to the 'tcnewrootdirpath'
# 2) Remanes the unzipped folder to apache-tomcat-xxx_aRCODevX and deteles the unnecessary directories from webapp folder
# 3) Copies the following files from the old Tomcat (as provided by 'oldtcpath') to the new Tomcat
#       i) server.xml
#       ii) context.xml
#       iii) catalina.properties
# 4) Creates the platform_lib folder and it's sub-folers as necessary
# 5) Build the following libraries and copies the generated JAR files to 'generated_lib' and the dependencies to 'dependency_lib'
#       i) platform-core
#       ii) tomcat-dist
#       iii) platform-connector
#       iv) wfm-delegate-dist
# 6) If deployment of the built tomcat is needed to dev-share, the following additional steps are performed:
#       i) The entire new Tomcat folder is zipped with 7zip
#       ii) The 7zip file is uploaded to dev-share
# 
# The following arguments need to be provided to the script in order as mentioned below:
#       i) The full path to the Tomcat home folder of the old Tomcat
#       ii) The fill path of the zipped (.zip) file of the Tomcat binary
#       iii) The directory in which the above operations will be performed, can be any directory
#       iv) The workspace directory path under which the individual above library projects are available

echo Give Old aRCO Tomcat Home Directory path
read oldtcpath
    if test -z "$oldtcpath"
    then 
    oldtcpath=~/SW/apache-tomcat-9.0.72_aRCODevX
    fi
echo -e ${BRed}Using old aRCO Tomcat Home Directory path:${NC} $oldtcpath
echo

echo Give the full path of the Tomcat zip file
read tczippath
    if test -z "$tczippath"
    then
    tczippath=~/Install/apache-tomcat-9.0.72.zip
    fi
echo -e ${BRed}Using Tomcat Zip file Path:${NC} $tczippath
echo

echo Give the full path of the diretcory for new Tomcat - this is where the new tomcat will be built
read tcnewrootdirpath
    if test -z "$tcnewrootdirpath"
    then
    tcnewrootdirpath=~/dev-tomcats
    fi
echo -e ${BRed}Using Tomcat Zip file Path:${NC} $tcnewrootdirpath
echo

echo Give Workspace Root Directory Path [Eclipse workspace directory]
read workspaceroot
    if test -z "$workspaceroot"
    then
    workspaceroot=~/workspaces/optimasprime
    fi
echo -e ${BRed}Using Workspace Path [Eclipse Workspace path]:${NC} $workspaceroot
echo

echo eploy zipped Tomcat [y/n]
read deploy
    if test -z "$deploy"
    then
    deploy=n
    fi
echo -e ${BRed}Upload built Tomcat:${NC} $deploy
echo


# Create the complete paths of the new unzipped tomcat
# Get the name of the tomcat zip file without .zip extension, this is the name of the unzipped folder, 
# add to it '_aRCODevX' for the full directory name

tczipfilenameonly=$(basename $tczippath)
newtcpath=$tcnewrootdirpath/${tczipfilenameonly%.*}_aRCODevX
echo The new tomcat path: $newtcpath

# Delete any old folder with the same name of the new tomcat path, if it exists
rm -r $newtcpath

# Extract the provided zip file to old tomcat path
7z x $tczippath -o$tcnewrootdirpath

# Rename the zipped tomcat to the new tomcat name
unzippedtcpath=$tcnewrootdirpath/${tczipfilenameonly%.*}
mv $unzippedtcpath $newtcpath



# Start duilding the new Tomcat with required files from the old tomcat and platform jars
echo Deleting the directories in new Tomcat webapp directory which are not required
rm -r $newtcpath/webapps/docs
rm -r $newtcpath/webapps/examples
rm -r $newtcpath/webapps/host-manager
rm -r $newtcpath/webapps/manager

echo Copying the catalina.properties, server.xml and context.xml from conf of old Tomcat to new Tomcat
cp $oldtcpath/conf/catalina.properties $newtcpath/conf/catalina.properties
cp $oldtcpath/conf/server.xml $newtcpath/conf/server.xml
cp $oldtcpath/conf/context.xml $newtcpath/conf/context.xml

echo Creating the platform_lib directory in new Tomcat root and generated_lib and staged_lib directories under them
mkdir $newtcpath/platform_lib
mkdir $newtcpath/platform_lib/generated_lib
mkdir $newtcpath/platform_lib/staged_lib



# Build the binaries and copy to required locations
echo Building platform-core jar gathering it\'s dependencies and copying files
cd $workspaceroot/platform-core
mvn clean package -Dmaven.test.skip=true -Pmake-bundle -U
cp -rT ./target/staged_lib $newtcpath/platform_lib/staged_lib
cp ./target/platform-core*.jar $newtcpath/platform_lib/generated_lib

echo Building platform-connector jar, gathering it\'s dependencies and copying files
cd $workspaceroot/platform-connector
mvn clean package -Dmaven.test.skip=true -Pmake-bundle -U
cp -rT ./target/staged_lib $newtcpath/platform_lib/staged_lib
cp ./target/platform-connector*.jar $newtcpath/platform_lib/generated_lib

echo Building tomcat-dist jar, gathering it\'s dependencies and copying files
cd $workspaceroot/tomcat-dist
mvn clean package -Dmaven.test.skip=true -Pmake-bundle -U
cp -rT ./target/staged_lib $newtcpath/platform_lib/staged_lib
cp ./target/tomcat-dist*.jar $newtcpath/platform_lib/generated_lib 

echo Building wfm-delegate-dist jar, gathering it\'s dependencies and copying files
cd $workspaceroot/wfm-delegate-dist
mvn clean package -Dmaven.test.skip=true -Pmake-bundle -U
cp -rT ./target/staged_lib $newtcpath/platform_lib/staged_lib
cp ./target/wfm-delegate-dist*.jar $newtcpath/platform_lib/generated_lib 


if [ $deploy == "y" ]
then
    echo Deploying the aRCO Tomcat to Dev-share
    
    echo Calculate the zip file name for the zip file to be uploaded
    uploadzipfilename=$(basename $newtcpath).7z
    echo Upload Zip File Name :: $uploadzipfilename

    echo Change to the new tomcat root directory
    cd $tcnewrootdirpath
    
    echo Delete the 7zip file if it already exists from an earlier run
    rm $uploadzipfilename
    
    echo Create the zip file of the new tomcat with contents of the new tomcat
    7z a $uploadzipfilename $newtcpath/
    
    echo Upload the created 7zip file to Dev-share folder in Engg-subnet
    echo Upload URL :: http://hitman3.server:8080/sw-repo/tomcat/$uploadzipfilename
    curl --upload-file $uploadzipfilename http://hitman3.server:8080/sw-repo/tomcat/$uploadzipfilename -v
fi

Currently we run the above script manually but it can also be dome with Jenkins other CI/CD.

Please not, we don't make an uber-jar