LSF (bsub) exit code inconsistent in airflow SSHOperator

49 Views Asked by At

TL;DR

bsub -K exit 1 || echo Result : $?

Results

  • Result : 1 in interactive shell
  • Result : 0 using Airflow SSHOperator

More details

When I run the following command in an interactive shell, I get the expected output:

bsub -K exit 1 || echo Result : $?
Job <3567228> is submitted to default queue <batch_a100>.
<<Waiting for dispatch ...>>
<<Starting on my_cluster>>
<<Job is finished>>
Result : 1

If I do the same using airflow's SSHOperator


run_command = SSHOperator(
    task_id="run_command",
    ssh_conn_id="my_cluster",
    command="bsub -K exit 1 || echo Result : $?",
    dag=dag,
)

run_command

I get the following:

[2023-07-14, 08:08:16 UTC] {ssh.py:469} INFO - Running command: bsub -K exit 1 || echo Result : $?
[2023-07-14, 08:08:16 UTC] {ssh.py:501} INFO - Job <3567227> is submitted to default queue <batch_a100>.
[2023-07-14, 08:08:16 UTC] {ssh.py:505} WARNING - <<Waiting for dispatch ...>>
[2023-07-14, 08:08:17 UTC] {ssh.py:505} WARNING - <<Starting on my_cluster>>
[2023-07-14, 08:08:21 UTC] {ssh.py:505} WARNING - <<Job is finished>>
[2023-07-14, 08:08:21 UTC] {ssh.py:501} INFO - Result : 0

I have verified, that both shells are bash using echo $SHELL.

I would have expected the same result for both ways, and I think the correct result is 1.

Where does this difference come from and how can I get the right exit status using Airflow?

0

There are 0 best solutions below