Ansible: Missing sudo password after setting nopasswd - Cache or something?

Question

I have a problem with some of my playbooks. I have two playbooks that are running after another. The first one is setting some requirements, like installing python3 (with a raw command) and sudo. After that I create a sudo group and add my user to it. After these steps I generate a random password, that is not safed anywhere for late use, for security reasons. I also set up a private key and copy the public part to the authorized_keys on the target system. After all is set up, I start the second playbook. This one uses the private key to login via ssh. And here is the problem, for a short while, ansible is responding with fatal: [HOSTNAME]: FAILED! => {"msg": "Missing sudo password"} for every targetsystem. If I wait for a while, it runs as expected.

How can I get rid of this little time, that I have to wait for? Is there a cache I need to kill? A service I need to start?

Maybe you guys could point me in the right direction.. even with google search, I only find hints about setting up sudo or something like that.

# This sets the sudo permissions
  - name: Allow 'sudo' group to have passwordless sudo
    lineinfile:
      dest: /etc/sudoers
      state: present
      regexp: '^%sudo'
      line: '%sudo ALL=(ALL) NOPASSWD: ALL'
      validate: '/usr/sbin/visudo -cf %s'

# The following playbook uses this vars
ansible_host_key_checking: false
ansible_python_interpreter: /usr/bin/python3
ansible_connection: ssh
ansible_user: install
ansible_private_key_file: "{{ hostvars['HOSTNAME']['ssh_keyfile']['stat']['path'] }}"

# This is the first step, that is failing with Missing sudo password
- name: Install Zabbix repository on all Zabbix-components servers
  hosts:
    - zbxserver
    - zbxproxy
    - webserver
    - dbserver
  gather_facts: true
  vars_files:
    - vars/general.yaml
    - vars/login.yaml
    - vars/zabbix.yaml
  tasks:
    - name: Install Zabbix repository
      ansible.builtin.raw: |
        apt install curl -y
        curl -O https://repo.zabbix.com/zabbix/6.4/debian/pool/main/z/zabbix-release/zabbix-release_6.4-1+debian12_all.deb
        dpkg -i zabbix-release_6.4-1+debian12_all.deb
        apt remove curl -y
      become: true
      when: do_requirements == "no"
    - name: Update the package cache
      ansible.builtin.apt:
        update_cache: true
      become: true
      when: do_requirements == "no"

I tried to kill the /tmp/ansible I aso deleted all /tmp files. I also checked for staled ssh sessions and I also restarted the ssh service on the target systems after setting all up.