Hive with TEZ failed to start Hive CLI

Question

Dears,

Apache Hive 3.1.2 with Hadoop 3.1.1 was working fine with until i configured Hive with Tez based on this doc :[ https://github.com/NitinKumar94/Installing-Apache-Tez]

it always gives this error, i tried many solutions with no luck

     Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/BatchListingOperations
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:473)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:405)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:348)
    at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:370)
    at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
    at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
    at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:3268)
    at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3313)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3352)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3403)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3371)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:477)
    at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
    at org.apache.hadoop.hive.common.FileUtils.getJarFilesByPath(FileUtils.java:1006)
    at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:5198)
    at org.apache.hadoop.hive.conf.HiveConf.<init>(HiveConf.java:5099)
    at org.apache.hadoop.hive.common.LogUtils.initHiveLog4jCommon(LogUtils.java:97)
    at org.apache.hadoop.hive.common.LogUtils.initHiveLog4j(LogUtils.java:81)
    at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:699)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:232)

Any help. i tried the following also with no luck

Copy hive-exec jar from $HIVE_HOME/lib to hdfs dir /user/tez/

Answer 1

If any of below errors and yours you can try this solution

Error: Could not find or load main class

org.apache.tez.dag.app.DAGAppMaster

You need to export your tez confs in hive server instance ( hive env sh)

export TEZ_CONF_DIR=/etc/tez/conf/
export TEZ_JARS=/your_path_2_tez/tez-0.10.2/ 
export HADOOP_CLASSPATH=${TEZ_CONF_DIR}:${TEZ_JARS}/*:${TEZ_JARS}/lib/*:${HADOOP_CLASSPATH} 

Then you need to copy your tez folder to hdfs but not zip , you need to copy uncompressed tez package

hdfs dfs -mkdir /tez/tez-0.10.2

Then copy your files

hdfs dfs -copyFromLocal /your_path_2_tez/tez-0.10.2/* /tez/tez-0.10.2/

Then define this path in tez-site xml

<property>
    <name>tez.lib.uris</name>
    <value>${fs.defaultFS}/tez/tez-0.10.2,${fs.defaultFS}/tez/tez-0.10.2/lib</value>
</property>

Also you need to copy your hadoop version jars (yours 3.1.1) to tez lib folder :

hadoop-common-3.3.1.jar
hadoop-yarn-common-3.3.1.jar
hadoop-yarn-client-3.3.1.jar

...... .....

the main idea is which class is not exist looking your exception you can copy the jar from your hadoop share jars from your hadoop installation directory. You need to copy this jars to hdfs as well

Then you can restart your hive server

Here you can use tez without error