Exception NoClassDefFoundError JniBasedUnixGroupsMapping while upgrading Drill version to > 1.14.0

Question

For our drill cluster, installed on Hadoop EMR core nodes, we use PAM based authentication. This is the below configuration.

security.user.auth: {
            enabled: true,
            packages += "org.apache.drill.exec.rpc.user.security",
            impl: "pam4j",
            pam_profiles: [ "sudo", "login" ]
     }

We run our jobs through YARN, where the yarn application makes a jdbc connection to drillbits & produces the results. Drill+Yarn Jobs were working fine with 1.6.0.

We tried to upgrade our drill to 1.14.0 & greater, started seeing exception as below. A normal JDBC sqlline connection was fine though in all cases. Only the Yarn apps failing.

When the 'security.user.auth' is disabled, Yarn jobs were fine.

Exception

Caused by: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at oadd.org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:134) at oadd.org.apache.hadoop.security.Groups.(Groups.java:79) at oadd.org.apache.hadoop.security.Groups.(Groups.java:74) at oadd.org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:303) at oadd.org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:283) at oadd.org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:311) at oadd.org.apache.drill.exec.rpc.security.plain.PlainFactory.createAndLoginUser(PlainFactory.java:63) at oadd.org.apache.drill.exec.rpc.user.UserClient.prepareSaslHandshake(UserClient.java:452) at oadd.org.apache.drill.exec.rpc.ConnectionMultiListener$HandshakeSendHandler.success(ConnectionMultiListener.java:160) ... 33 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at oadd.org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:132) ... 41 more Caused by: java.lang.NoClassDefFoundError: Could not initialize class oadd.org.apache.hadoop.security.JniBasedUnixGroupsMapping at oadd.org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.(JniBasedUnixGroupsMappingWithFallback.java:39) ... 46 more

EMR Version 5.36.1 Please let us know for any more info

Answer 1

I believe that your trouble comes from the Hadoop native library. The version of that library installed by EMR 5.36.1 might not be compatible with the Java Hadoop libraries shipped with Drill. Check the Hadoop library version in your Drill installation with a command like this.

➜  ls /opt/apache-drill-1.20.3/jars/3rdparty/hadoop-common*.jar
/opt/apache-drill-1.20.3/jars/3rdparty/hadoop-common-3.2.4.jar

Finally, download a version of libhadoop.so corresponding to Drill's Hadoop library version found above and use techniques like LD_LIBRARY_PATH to ensure that Drill loads it and not the one that was installed by EMR.