Is it possible to use several jars to specify job in hadoop?

Question

I usually have common/shared libraries and actual job code in different jar files. Is it possible to recompile only one job jar file to execute hadoop command hadoop jar asd? If not is there workaround to simplify jar packaging?

Answer 1

I am using Ant to make the job jar. To include all common/shared libraries in the buildConfig.xml file you have to add such line:

<zipgroupfileset dir="pathToAllCommonAndSharedLibraries" includes="**/*.jar" />

Here is the simplest example of build config file.

<?xml version="1.0" encoding="UTF-8"?>
<project name="Example.makejar" default="jar" basedir=".">
    <target name="jar">
        <jar destfile="Example.jar" basedir="bin">
            <manifest></manifest>
            <zipgroupfileset dir="pathToAllCommonAndSharedLibraries" includes="**/*.jar" />
        </jar>
    </target>
</project>

Answer 2

I am not sure about the hadoop support for this, I will ask later, but here's the workaround: if you use maven to build your job projects, use maven shade plugin, or maven assembly plugin to embed all your dependencies into the jar file, so that you will deploy only one file.