cat and curl large JSON value from gitlab ci

Question

There are many questions on how to post JSON via curl. And I am able to send one word as json but not the whole text.

My requirement is to comment a merge request when one job fail in our GitLab CI.

# From .gitlab-ci.yaml
  after_script:
    - report=$(find services/ -name '*_latest_failure.md')
    - comment="Job failed.\n\n$(cat "$report")"
    - if [[ "${CI_JOB_STATUS}" == "failed" ]] ; then bash -c "source .gitlab-ci/gitlab.sh; gitlab-comment_merge_request "$comment"" ; fi

# From .gitlab-ci/gitlab.sh
function gitlab-comment_merge_request() {
  url="https://$CI_SERVER_HOST/api/v4/projects/$CI_MERGE_REQUEST_PROJECT_ID/merge_requests/$CI_MERGE_REQUEST_IID/notes"
  comment="$1"
  curl -v --location -X POST "$url" \
    --header "PRIVATE-TOKEN: $GITLAB_CI_TOKEN" \
    --header "Content-Type: application/json" \
    --data '{"body": "'"$comment"'"}'
}

Unfortunately, the comment that is created contains only the first word, Job.

I have tried many techniques to build and send the JSON payload. But all give the same result.

So I thought it could be a mistake when creating the $comment variable in my .gitlab-ci.yaml. However, echo $comment in the same file give the right content.

Now, I'm lost. Do you know why the $comment variable that should contain a large text (a markdown text with title, subtitles, and lists) is not sent ?

Thanks

Answer 1

A simplified version of this would look like:

nl='
'
comment="Job failed.${nl}${nl}$(find services/ -name '*_latest_failure.md' -exec cat -- {} +)"
if [ "${CI_JOB_STATUS}" = "failed" ] ; then
  . .gitlab-ci/gitlab.sh
  gitlab_comment_merge_request "$comment"
fi

Note:

• Because you were quoting the output from find before passing it to cat, cat was looking for a single file with all the filenames concatenated together as its name. Don't do that. Using find ... -exec cat -- {} + correctly handles names with spaces without creating new bugs.
• We've taken out bash-only syntax: Changed == to =, [[ to [.
• We took out the use of bash -c and the exec boundary between parent and child processes it implied. If we hadn't done this we would need to export comment.
• We changed the source keyword to its POSIX-compliant synonym, .
• We took the dash out of the function name (this does require the file being sourced to change).

A POSIX-compliant version of your function might look like:

# note we had to rename this: dashes are not required to work in function names in sh
gitlab_comment_merge_request() {
  url="https://$CI_SERVER_HOST/api/v4/projects/$CI_MERGE_REQUEST_PROJECT_ID/merge_requests/$CI_MERGE_REQUEST_IID/notes"
  body=$(jq -n --var comment "$1" '{"body":$comment}') 
  curl -v --location -X POST "$url" \
    --header "PRIVATE-TOKEN: $GITLAB_CI_TOKEN" \
    --header "Content-Type: application/json" \
    --data "$body"
}

If you don't have jq installed, it's straightforward to write a shell wrapper for Python's json module for the same purpose.

Answer 2

It is possible to minimize all the logic and JSON formatting into a single shell script:

gitlab-ci.yaml:

  after_script:
    - bash failed-reports.sh

failed-reports.sh:

#!/usr/bin/env sh

# If no failed job do nothing, exit
[ "$CI_JOB_STATUS" = failed ] || exit

# Pass latest_failure files into arguments
set -- ./services/*_latest_failure.md

# If argument 1 is pattern there is no latest_failure to process
[ "$1" = './services/*_latest_failure.md' ] && exit

# Dummy url for demo purpose
url='https://example.com/'

# Prepare report as a stream
{
    printf '# Job failed.\n\n'
    # Iterate each latest_failure from arguments
    for latest_failure; do
        # Skip file if not readabe or non existent
        [ -r "$latest_failure" ] || continue
        # Document name in Markdown bold with blank line
        printf '## Report `%s`:\n\n' "${latest_failure##*/}"
        # Stream latest_failure markdown
        cat "$latest_failure"
        echo # add a newline after streaming report markdown
    done
} |
# Stream to jq to process report as raw input string into JSON
jq \
    --slurp \
    --raw-input \
    --compact-output \
    '{"body": .}' |
# Stream JSON data to curl so that it can handle very large payloads
curl \
    --url "$url" \
    --location \
    --request POST \
    --header "PRIVATE-TOKEN: $GITLAB_CI_TOKEN" \
    --header "Content-Type: application/json" \
    --data @- # This takes data from the STDIN stream

In case the JSON data payload is very large, it is a safer option to stream it to curl with --data @- rather than passing it as an argument which turns out to be rather limited by the system's maximum allowed argument length.