Unable to archive mutiple files in a single tar using ansible

Question

I wish to compress the comma separated files in variable {{ sourcefiles }} into HELLO.tar on the target source_host and bring it to Ansible controller using fetch module and unarchive (while retaining their absolute paths <- this is optional).

Unfortunately, the tar file has just the last file in the list of files (with absolute path) provided i.e /tmp/4192.txt.

- name: Archive the files

  archive:
    path: "{{ item }}"
    dest: "~/HELLO.tar"
    format: tar
  with_items: "{{ sourcefiles.split(',') }}"

- name: Fetch directories from remote hosts
  fetch:
    src: "~/HELLO.tar"
    dest: "{{ playbook_dir }}/tmpfiles/{{ Latest_Build_Number_rec }}/"
    flat: yes
    fail_on_missing: no 

ansible run:

ansible-playbook passvar.yml -e "Latest_Build_Number_rec=200" -e "sourcefiles=/tmp/419.txt,/tmp/4191.txt,/tmp/4192.txt,/tmp/moht,/tmp/4192.txt" -vv

Output:

[wladmin@ansiblehost 200]$ tar -tvf HELLO.tar
-rw-r--r-- wladmin/aces      0 2023-11-08 04:02 tmp/4192.txt

As you can see, I only get the last file in the loop and not all the files.

Output:

TASK [debug] ******************************************************************************************************************

task path: /home/wladmin/passvar.yml:56

Wednesday 08 November 2023  05:32:19 -0600 (0:00:00.214)       0:00:01.089 ****

ok: [remotesourcehost] => (item=/tmp/419.txt) => {
    "msg": "/tmp/419.txt"
}

ok: [remotesourcehost] => (item=/tmp/4191.txt) => {
    "msg": "/tmp/4191.txt"
}

ok: [remotesourcehost] => (item=/tmp/4192.txt) => {
    "msg": "/tmp/4192.txt"
}

ok: [remotesourcehost] => (item=/tmp/moht) => {
    "msg": "/tmp/moht"
}

ok: [remotesourcehost] => (item=/tmp/4192.txt) => {
    "msg": "/tmp/4192.txt"
}

 

TASK [Archive the files] ******************************************************************************************************

task path: /home/wladmin/passvar.yml:61

Wednesday 08 November 2023  05:32:19 -0600 (0:00:00.046)       0:00:01.136 ****

ok: [remotesourcehost] => (item=/tmp/419.txt) => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python"}, "ansible_loop_var": "item", "archived": ["/tmp/419.txt"], "arcroot": "/tmp/", "changed": false, "dest": "/home/wladmin/HELLO.tar", "expanded_exclude_paths": [], "expanded_paths": ["/tmp/419.txt"], "gid": 64395, "group": "aces", "item": "/tmp/419.txt", "missing": [], "mode": "0644", "owner": "wladmin", "size": 10240, "state": "file", "uid": 600000008, "warnings": ["Platform linux on host remotesourcehost is using the discovered Python interpreter at /usr/bin/python, but future installation of another Python interpreter could change this. See https://docs.ansible.com/ansible/2.8/reference_appendices/interpreter_discovery.html for more information."]}

ok: [remotesourcehost] => (item=/tmp/4191.txt) => {"ansible_loop_var": "item", "archived": ["/tmp/4191.txt"], "arcroot": "/tmp/", "changed": false, "dest": "/home/wladmin/HELLO.tar", "expanded_exclude_paths": [], "expanded_paths": ["/tmp/4191.txt"], "gid": 64395, "group": "aces", "item": "/tmp/4191.txt", "missing": [], "mode": "0644", "owner": "wladmin", "size": 10240, "state": "file", "uid": 600000008}

ok: [remotesourcehost] => (item=/tmp/4192.txt) => {"ansible_loop_var": "item", "archived": ["/tmp/4192.txt"], "arcroot": "/tmp/", "changed": false, "dest": "/home/wladmin/HELLO.tar", "expanded_exclude_paths": [], "expanded_paths": ["/tmp/4192.txt"], "gid": 64395, "group": "aces", "item": "/tmp/4192.txt", "missing": [], "mode": "0644", "owner": "wladmin", "size": 10240, "state": "file", "uid": 600000008}

ok: [remotesourcehost] => (item=/tmp/moht) => {"ansible_loop_var": "item", "archived": [], "arcroot": "/tmp/", "changed": false, "dest": "/home/wladmin/HELLO.tar", "expanded_exclude_paths": [], "expanded_paths": ["/tmp/moht"], "gid": 64395, "group": "aces", "item": "/tmp/moht", "missing": [], "mode": "0644", "owner": "wladmin", "size": 10240, "state": "file", "uid": 600000008}

ok: [remotesourcehost] => (item=/tmp/4192.txt) => {"ansible_loop_var": "item", "archived": ["/tmp/4192.txt"], "arcroot": "/tmp/", "changed": false, "dest": "/home/wladmin/HELLO.tar", "expanded_exclude_paths": [], "expanded_paths": ["/tmp/4192.txt"], "gid": 64395, "group": "aces", "item": "/tmp/4192.txt", "missing": [], "mode": "0644", "owner": "wladmin", "size": 10240, "state": "file", "uid": 600000008}

TASK [Fetch directories from remote hosts] ************************************************************************************

task path: /home/wladmin/passvar.yml:75

Wednesday 08 November 2023  05:32:21 -0600 (0:00:01.914)       0:00:03.051 ****

changed: [remotesourcehost] => {"changed": true, "checksum": "c44a7a35626075a29c6af395c096faada0fd1758", "dest": "/home/wladmin/tmpfiles/200/HELLO.tar", "md5sum": "72e76072ca2022b4d912b5108858fd53", "remote_checksum": "c44a7a35626075a29c6af395c096faada0fd1758", "remote_md5sum": null}

The purpose is to recreate the same file-folder structure on a different host.

One solution could be to copy all the files to a single directory and then compress that directory but I wish to archive while retaining their absolute paths.

Answer 1

You're current approach can be fixed (see @U880D's comment) but there is a much more efficient way to achieve your goal using ansible.posix.synchronize module. Simply get the list of files directly from the target to the container without having to go through an archive. The only prerequisite is to have rsync available on both target and controller.

Here's a full blown example using a docker container as target.

Test project structure:

$ tree
.
├── Dockerfile
├── filelist.txt
├── inventory.yml
├── playbook.yml
└── sync_dir

1 directory, 4 files

This is the Dockerfile for testing:

FROM python:alpine

WORKDIR /work

RUN \
    apk update && apk add rsync && \
    for i in `seq 1 9`; do \
    touch demofile$i.txt; \
    done

CMD tail -f /dev/null

We can build the image and launch the test container:

$ docker build -t testans:local .
$ docker run -d --rm --name testans_container testans:local

We can check we have some test files ready in the default container working directory:

$ docker exec testans_container pwd
/work

$ docker exec testans_container ls -l
total 9
-rw-r--r--    1 root     root             0 Nov  8 17:51 demofile1.txt
-rw-r--r--    1 root     root             0 Nov  8 17:51 demofile2.txt
-rw-r--r--    1 root     root             0 Nov  8 17:51 demofile3.txt
-rw-r--r--    1 root     root             0 Nov  8 17:51 demofile4.txt
-rw-r--r--    1 root     root             0 Nov  8 17:51 demofile5.txt
-rw-r--r--    1 root     root             0 Nov  8 17:51 demofile6.txt
-rw-r--r--    1 root     root             0 Nov  8 17:51 demofile7.txt
-rw-r--r--    1 root     root             0 Nov  8 17:51 demofile8.txt
-rw-r--r--    1 root     root             0 Nov  8 17:51 demofile9.txt

The inventory.yml will target that docker test container:

---
all:
  hosts:
    testans_container:
      ansible_connection: docker
      ansible_python_interpreter: /usr/local/bin/python

We list in filelist.txt the files we want to transfer:

demofile1.txt
demofile4.txt
demofile9.txt

We now create the playbook.yml which will put all this together:

---
- name: partial synchronize demo
  hosts: testans_container
  gather_facts: false

  vars:
    remote_dir: /work
    local_dir: sync_dir
    file_list: filelist.txt

  tasks:
    - name: Copy selected file listed in filelist.txt from target to controller
      ansible.posix.synchronize:
        mode: pull
        src: "{{ remote_dir }}"
        dest: "{{ local_dir }}"
        rsync_opts:
          - "--files-from={{ file_list }}"

Let's check our destination directory is empty before we try:

$ ls -l sync_dir/
total 0

Now we run the playbook:

$ ansible-playbook -i inventory.yml playbook.yml 

PLAY [partial synchronize demo] ******************************************************************************************************************************************************************************************************

TASK [Copy selected file listed in filelist.txt from target to controller] ***********************************************************************************************************************************************************
changed: [testans_container]

PLAY RECAP ***************************************************************************************************************************************************************************************************************************
testans_container          : ok=1    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

And we can see that required files have arrived to destination:

$ ls -l sync_dir/
total 3
-rw-r--r-- 1 user users 0 nov  8 18:51 demofile1.txt
-rw-r--r-- 1 user users 0 nov  8 18:51 demofile4.txt
-rw-r--r-- 1 user users 0 nov  8 18:51 demofile9.txt

Note: the time shift for the files between the container and the controller is normal as the default timezone in the container is UTC while my local machine is configured with CET (i.e. UTC+1)

I will let you run the playbook again to check that it is fully idempotent and play with the remote files to check that any change is transferred back to the controller as requested. Once finished just cleanup the test container:

$ docker stop testans_container