Any way to know error details in Ansible / AWX even when we have a no_log: true for the task?

Question

We have an Ansible automation in production which talks to a remote API. When we check the logs of that task in AWX, it reveals a lot of sensitive data like the API key, hence we have removed the logging by adding no_log: true for that task.

Now, we need to know the cause of the error when the task fails, but the no_log setting prevents us from seeing any valuable logs. Is there a way to save the error JSON logs in a file on the backend server (only accessible to admins), so that when that remote API task fails, we can troubleshoot using the logs?
Since the AWX instance is accessible to non-admins as well, we cannot rollback no_log to false, to avoid revealing sensitive data.

We normally have to disable the no_log setting temporarily, and re-run the job, but this is not ideal.

Answer 1

Yes, you could use a block and rescue the error of the API, log its registered output, then fail with a convenient message for the end user.

Please mind: you have to delegate the copy task creating the file if you want the error to be logged in the controller (the AWX machine, in your case) and not on the targeted node(s).

Here is an example of such a block:

- block:
    - uri:
        url: "http://username:{{ secret_password | urlencode }}@example.org"
      no_log: true
      register: uri_output

  rescue:
    - set_fact:
        log: "/var/log/ansible-failure-{{ '%Y%m%d%H%M%S' |  strftime }}.log"

    - copy:
        content: "{{ uri_output | to_nice_json }}"
        dest: "{{ log }}"
      delegate_to: localhost

    - fail:
        msg: |-
          The exact error has been redacted due to secret information,
          but have been logged in `{{ log }}`.
          If you do not have sufficient rights, please ask an administrator.

Which, on success, would yield:

TASK [uri] ***************************************************************
ok: [ansible-node]

But in case of error:

TASK [uri] ***************************************************************
fatal: [ansible-node]: FAILED! => changed=false 
  censored: 'the output has been hidden due to the fact that ''no_log: true'' was specified for this result'

TASK [set_fact] **********************************************************
ok: [ansible-node]

TASK [copy] **************************************************************
changed: [ansible-node -> localhost]

TASK [fail] **************************************************************
fatal: [ansible-node]: FAILED! => changed=false 
  msg: |-
    The exact error has been redacted due to secret information,
    but have been logged in `/var/log/ansible-failure-20230908100134.log`.
    If you do not have sufficient rights, please ask an administrator.

Along with the file /var/log/ansible-failure-20230908100134.log, containing:

{
    "changed": false,
    "elapsed": 0,
    "failed": true,
    "msg": "Status code was -1 and not [200]: Request failed: <urlopen error [Errno -2] Name or service not known>",
    "redirected": false,
    "status": -1,
    "url": "http://username:shh%20I%20am%20a%[email protected]"
}