Oozie Hive Job status changes to KILLED but the underlying job is succes

146 Views Asked by At

I'm running Hive 3.1.3, Hue 4.10.0, Oozie 5.2.1

Through Oozie, I'm running a periodic Hive job which takes data from an HDFS path, and adds it to an existing hive table.

It works fine when I'm running it using Hive shell. However, when I run it using Oozie, the underlying data in Hive gets updated correctly, but the Oozie job fails without giving any meaningful error.

enter image description here This is a screenshot explaining the situation. The job ran for 1hr 11m, the data in Hive got updated, but Oozie responded with a failed status.

Here are the applicationLogs :



Parameters:
------------------------
  DATE=2023-04-12
------------------------

Hive command arguments :
             --hiveconf
             hive.log4j.file=/mnt/yarn/usercache/ec2-user/appcache/application_1675167116593_12260/container_1675167116593_12260_01_000001/hive-log4j.properties
             --hiveconf
             hive.exec.log4j.file=/mnt/yarn/usercache/ec2-user/appcache/application_1675167116593_12260/container_1675167116593_12260_01_000001/hive-exec-log4j.properties
             --hiveconf
             hive.log.trace.id=oozie:0005857-230202071432750-oozie-oozi-W
             -f
             dr_update_hive.hql
             --hivevar
             DATE=2023-04-12

Fetching child yarn jobs
tag id : oozie-2621f368aebc0324f3d78a262abee63e
No child applications found
=================================================================

>>> Invoking Hive command line now >>>


<<< Invocation of Hive command completed <<<

No child hadoop job is executed.
java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.oozie.action.hadoop.LauncherAM.runActionMain(LauncherAM.java:412)
    at org.apache.oozie.action.hadoop.LauncherAM.access$400(LauncherAM.java:54)
    at org.apache.oozie.action.hadoop.LauncherAM$2.run(LauncherAM.java:225)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
    at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:219)
    at org.apache.oozie.action.hadoop.LauncherAM$1.run(LauncherAM.java:155)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
    at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:143)
Caused by: java.lang.SecurityException: Intercepted System.exit(1)
    at org.apache.oozie.action.hadoop.security.LauncherSecurityManager.checkExit(LauncherSecurityManager.java:57)
    at java.lang.Runtime.exit(Runtime.java:107)
    at java.lang.System.exit(System.java:973)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:685)
    at org.apache.oozie.action.hadoop.HiveMain.runHive(HiveMain.java:310)
    at org.apache.oozie.action.hadoop.HiveMain.run(HiveMain.java:294)
    at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:107)
    at org.apache.oozie.action.hadoop.HiveMain.main(HiveMain.java:69)
    ... 16 more
Intercepting System.exit(1)
Failing Oozie Launcher, Main Class [org.apache.oozie.action.hadoop.HiveMain], exit code [1]
Oozie Launcher, uploading action data to HDFS sequence file: hdfs://ip-10-162-8-160.ap-south-1.compute.internal:8020/user/ec2-user/oozie-oozi/0005857-230202071432750-oozie-oozi-W/hiveScript--hive/action-data.seq
Stopping AM
Callback notification attempts left 0
Callback notification trying http://ip-10-162-8-160.ap-south-1.compute.internal:11000/oozie/callback?id=0005857-230202071432750-oozie-oozi-W@hiveScript&status=FAILED
Callback notification to http://ip-10-162-8-160.ap-south-1.compute.internal:11000/oozie/callback?id=0005857-230202071432750-oozie-oozi-W@hiveScript&status=FAILED succeeded
Callback notification succeeded
 


I need this because the job is correct and I'm getting unnecessary FAILURE alerts.

P.S. : Please tell me if any other info is needed. I'll be happy to provide

I tried reading the application logs but it didn't work

0

There are 0 best solutions below