debug a bash script to run remote ssh commands concurrently (data loss when piping to ssh)

Question

I want start remote commands via ssh and simply show the output.

My Question ts NOT how to do it. There are a lot of good answers already.

My question is: why my attempt does not work and how to possibly debug it. So this is mainly a question about the bash syntax.

The motivation was: If running several remote commands in parallel, the output is possibly garbled. So my idea was to run all request thru a big pipe. Each stage executes a remote command and pipes its output to the next stage(s). Each stage finally uses cat to append the output from the previous stage.

To get a comprehensible example the remote command is just an echo and the remote host is always localhost.

#!/bin/bash

Message () {
    if [ $# -gt 0 ]
    then
        local Arg1=$1
        shift

        # execute other commands in parallel
        Message "$@" | (
            # a very simple remote command
            ssh localhost "echo $Arg1"

            # append the output of previous commands
            cat
        )
    fi
}

Message The quick brown fox jumps over the lazy dog

Running this script returns only a subset of the arguments. Using bash -x shows that it was executed but some output was not pushed into to the pipe.

However: replacing the ssh localhost "echo $Arg1" with a simple echo $Arg1 works very well.

So what went wrong with the ssh command? How does it break the pipe? Some wired exit code?

Answer 1

This is a typical ssh "issue". The "issue" is that ssh would also read from stdin so output from Message "$@" is not all sent to cat. To fix it you can use

ssh -n localhost "echo $Arg1"
# or
ssh localhost "echo $Arg1" < /dev/null

---

Simple example to see the ssh behavior:

$ echo hello world | { ssh 127.0.0.1 true; cat; }
$ echo hello world | { ssh -n 127.0.0.1 true; cat; }
hello world
$ echo hello world | { ssh 127.0.0.1 true < /dev/null; cat; }
hello world

Answer 2

Suggestion: why not use a tempfile for each command instead?

Message() {
    tmpfiles=()

    for arg in "$@"; do
       tmp="$(mktemp)"
       tmpfiles+=("$tmp")
       ssh localhost "echo $arg" > "$tmp" &
    done

    wait
    cat "${tmpfiles[@]}"
    rm "${tmpfiles[@]}"
}