unable to upload a specific file using ansible uri module

Question

I'm able to upload simple text files to Jrog artifactory using ansible. However, the same code fails when I try to upload a file of different type.

Below are the type of files.

[root@mylocalhost tmp]# file file1.txt
file1.txt: ASCII text, with CRLF line terminators
[root@mylocalhost tmp]# file toolkit_2
toolkit_2: ASCII text, with very long lines
[root@mylocalhost tmp]# file /tmp/zip/25.zip
/tmp/zip/25.zip: Zip archive data, at least v2.0 to extract
[root@mylocalhost tmp]# file /tmp/zip/25.tar.gz
/tmp/zip/25.tar.gz: gzip compressed data, from Unix, last modified: Fri Feb  9 03:16:32 2024

Ansible playbook:

- name: List file on Runner Host
  raw: "chmod 777 /tmp/{{ build_number | trim }}.zip  && chmod 777 /tmp/{{ build_number | trim }}.tar.gz && chmod 777 /tmp/toolkit_2 && chmod 777 /tmp/file1.txt && ls -ltr /tmp/{{ build_number | trim }}.zip && ls -ltr /tmp/{{ build_number | trim }}.tar.gz && ls -ltr /tmp/toolkit_2 && ls -ltr /tmp/file1.txt"
  register: runnerfiles
  delegate_to: localhost

- debug:
    msg: "{{ runnerfiles }}"



- name: Upload All to Artifactory
  uri:
    url: "{{ jfrog_artifact_url }}{{ item | trim | basename }}"
    method: PUT
    user: "{{ JFrogUser }}"
    password: "{{ JFrogUserPassword  | replace('~', '=') }}"
    body: "{{ lookup('file', item ) }}"
    status_code: 201  # Adjust based on the expected status code
    timeout: 900
    return_content: yes
    body_format: "raw"
  loop:
    - "/tmp/file1.txt"
    - "/tmp/{{ build_number | trim }}.zip"
    - "/tmp/{{ build_number | trim }}.tar.gz"

Output:

TASK [wifideploy : List file on Runner Host] ***********************************
changed: [remhost -> localhost]

TASK [wifideploy : debug] ******************************************************
ok: [localhost] => {
    "msg": {
        "changed": true,
        "failed": false,
        "rc": 0,
        "stderr": "",
        "stderr_lines": [],
        "stdout": "-rwxrwxrwx 1 root root 21230 Feb  9 11:48 /tmp/25.zip\\n-rwxrwxrwx 1 root root 18958 Feb  9 11:48 /tmp/25.tar.gz\\n-rwxrwxrwx 1 root root 55872 Feb  9 11:48 /tmp/toolkit_2\\n-rwxrwxrwx 1 root root 11 Feb  9 11:48 /tmp/file1.txt\\n",
        "stdout_lines": [
            "-rwxrwxrwx 1 root root 21230 Feb  9 11:48 /tmp/25.zip",
            "-rwxrwxrwx 1 root root 18958 Feb  9 11:48 /tmp/25.tar.gz",
            "-rwxrwxrwx 1 root root 55872 Feb  9 11:48 /tmp/toolkit_2",
            "-rwxrwxrwx 1 root root 11 Feb  9 11:48 /tmp/file1.txt"
        ]
    }
}
ok: [remhost -> localhost] => (item=/tmp/file1.txt)
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: UnicodeEncodeError: 'utf-8' codec can't encode character '\udcf3' in position 10: surrogates not allowed
failed: [remhost -> localhost] (item=/tmp/25.zip) => {"ansible_loop_var": "item", "changed": false, "item": "/tmp/25.zip", "module_stderr": "Traceback (most recent call last):\n  File \"/root/.ansible/tmp/ansible-tmp-1707473267.4003098-161-46575362453121/AnsiballZ_uri.py\", line 107, in <module>\n    _ansiballz_main()\n  File \"/root/.ansible/tmp/ansible-tmp-1707473267.4003098-161-46575362453121/AnsiballZ_uri.py\", line 99, in _ansiballz_main\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"/root/.ansible/tmp/ansible-tmp-1707473267.4003098-161-46575362453121/AnsiballZ_uri.py\", line 47, in invoke_module\n    runpy.run_module(mod_name='ansible.modules.uri', init_globals=dict(_module_fqn='ansible.modules.uri', _modlib_path=modlib_path),\n  File \"/usr/lib64/python3.8/r…
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: UnicodeEncodeError: 'utf-8' codec can't encode character '\udc8b' in position 1: surrogates not allowed
failed: [remhost -> localhost] (item=/tmp/25.tar.gz) => {"ansible_loop_var": "item", "changed": false, "item": "/tmp/25.tar.gz", "module_stderr": "Traceback (most recent call last):\n  File \"/root/.ansible/tmp/ansible-tmp-1707473267.6308842-161-37207543543750/AnsiballZ_uri.py\", line 107, in <module>\n    _ansiballz_main()\n  File \"/root/.ansible/tmp/ansible-tmp-1707473267.6308842-161-37207543543750/AnsiballZ_uri.py\", line 99, in _ansiballz_main\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"/root/.ansible/tmp/ansible-tmp-1707473267.6308842-161-37207543543750/AnsiballZ_uri.py\", line 47, in invoke_module\n    runpy.run_module(mod_name='ansible.modules.uri', init_globals=dict(_module_fqn='ansible.modules.uri', _modlib_path=modlib_path),\n  File \"/usr/lib64/python…
[WARNING]: Module did not set no_log for password

As you can see from the output /tmp/file1.txt got uploaded successfully.

I also, tried to upload using src attribute instead of body but that too gets me error.

- name: Upload All to Artifactory
  uri:
    url: "{{ jfrog_artifact_url }}{{ item | basename }}"
    method: PUT
    user: "{{ JFrogUser }}"
    password: "{{ JFrogUserPassword  | replace('~', '=') }}"
    src: "{{ item }}"
    remote_src: true
    status_code: 201  # Adjust based on the expected status code
    timeout: 900
    return_content: yes
  loop:
    - "/tmp/{{ build_number | trim }}.zip"
    - "/tmp/{{ build_number | trim }}.tar.gz"

Output:

TASK [wifideploy : Upload toolkit_2 to Artifactory] ***
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: TypeError: can't concat str to bytes
fatal: [remhost -> localhost]: FAILED! => {"changed": false, "content": "", "elapsed": 0, "msg": "Status code was -1 and not [201]: An unknown error occurred: can't concat str to bytes", "redirected": false, "status": -1, "url": "https://jfrog.mybank.com/artifactory/raw/mybank/144/toolkit_2"}

Answer 1

For a recent version of Ansible, See Alexander Pletnev's answer.

For older versions of Ansible:

The body parameter of the Ansible's uri module handles expects a string, but when uploading binary files like a .zip or .tar.gz, you are essentially trying to load binary data into a string, which leads to encoding issues as seen in the error messages.

For binary files, it is essential to make sure the data is not being interpreted or modified, which is where the src parameter should come in.
However, the uri module does not directly support binary file uploads using a src parameter like you might find in other modules (e.g., copy).

That would leave, as a workaround, using the command or shell module to execute a curl command, which is capable of handling binary data uploads effectively.

- name: Upload Binary File to Artifactory using curl
  command: >
    curl -u "{{ JFrogUser }}:{{ JFrogUserPassword | replace('~', '=') }}"
    -X PUT "{{ jfrog_artifact_url }}{{ item | basename }}"
    -T "{{ item }}"
    --fail
    --silent
    --show-error
  loop:
    - "/tmp/{{ build_number | trim }}.zip"
    - "/tmp/{{ build_number | trim }}.tar.gz"
  register: curl_upload_result
  delegate_to: localhost
  ignore_errors: yes

- debug:
    var: curl_upload_result

That uses curl to perform the file upload, leveraging its ability to handle binary data without the encoding issues encountered with the uri module.
The --fail, --silent, and --show-error flags are used to control curl's output and error handling, making it easier to manage within Ansible.

---

As we are using OLAM docker runner, it does not have a curl utility. Any other solution would help.

Assuming that an OLAM (Oracle Linux Automation Manager) Docker runner would include Python, you can try and use Python's requests library, which might need to be installed if not already available.
Or, you can refine the Python one-liner to remain purely based on Python's standard library to avoid external dependencies:

- name: Upload Binary File to Artifactory using Python's http.client
  ansible.builtin.shell: |
    python3 -c "import http.client, base64, mimetypes; conn = http.client.HTTPSConnection('{{ jfrog_artifact_url.split('://')[1].split('/')[0] }}'); headers = {'Authorization': 'Basic ' + base64.b64encode('{{ JFrogUser }}:{{ JFrogUserPassword | replace('~', '=') }}'.encode()).decode(), 'Content-Type': mimetypes.MimeTypes().guess_type('{{ item }}')[0] or 'application/octet-stream'}; with open('{{ item }}', 'rb') as f: body = f.read(); conn.request('PUT', '/{{ jfrog_artifact_url.split('://')[1].split('/', 1)[1] }}/{{ item | basename }}', body, headers); response = conn.getresponse(); print(response.status, response.reason)"
  loop:
    - "/tmp/{{ build_number | trim }}.zip"
    - "/tmp/{{ build_number | trim }}.tar.gz"
  delegate_to: localhost
  ignore_errors: yes
  register: python_upload_result

- debug:
    var: python_upload_result

Again, this assumes the Python environment is properly configured within your OLAM Docker runner to execute these commands.
That should be compatible with environments where utilities like curl are not available, but where Python is installed.

Answer 2

The only thing you were missing is body_format: form-urlencoded. There is no need to use any external tools like curl or Python scripts.

As of version 2.15 (and it actually should be since 2.7 when src parameter and body_format: form-urlencoded were added) the uri module works perfectly fine with binary file uploads:

- name: Upload All to Artifactory
  uri:
    url: "{{ jfrog_artifact_url }}{{ item | basename }}"
    method: PUT
    user: "{{ JFrogUser }}"
    password: "{{ JFrogUserPassword  | replace('~', '=') }}"
    body_format: form-urlencoded
    src: "{{ item }}"
    remote_src: true
    status_code: 201  # Adjust based on the expected status code
    timeout: 900
    return_content: yes
  loop:
    - "/tmp/{{ build_number | trim }}.zip"
    - "/tmp/{{ build_number | trim }}.tar.gz"

Also, you can use body_format: form-multipart as an example in uri module documentation shows. I wasn't able, however, open the uploaded zip archive after that. But probably I just used an incorrect mime_type. But anyway, the working solution with form-urlencoded is even simpler as you don't have to maintain the body structure and MIME types.