Airflow MWAA - ALL tasks stuck as "queued" depending on used by KubernetesPodOperator image

Question

I have a strange issue with Airflow (MWAA) and even don't understand, how this can be explained

Our setup:

• Aiflow 2.0.2, managed by AWS (Managed Workflows for Apache Airflow / MWAA)
• the most of the DAGs tasks - are KubernetesPodOperator tasks (using different Dockerfile images for DBT / Python scripts / other tools)

Everything works like a charm for a long time.

But yesterday I needed to add a Python script with querying of MS SQL Server datasource (what never been used before).

I know, that using of PYODBC & FreeTDS inside Python Dockerfile for connection to MS SQL Server is a bit tricky and had successful experince already in the past.

For that I adjusted our Python image Dockerfile following:

# create a layer from python:3.7.4
FROM python:3.7.4

# Create and set the working directory
WORKDIR /usr/src/app/

# Causes all output to stdout to be flushed immediately
ENV PYTHONUNBUFFERED 1
ENV APT_KEY_DONT_WARN_ON_DANGEROUS_USAGE=DontWarn





# THIS SETUP PART IS REQUIRED FOR PYODBC & FREETDS connection to SQL Server
# update apt-get
RUN apt-get update -qq
RUN apt-get install -y apt-utils
# RUN apt-get install -y --no-install-recommends curl gcc g++ gnupg unixodbc-dev apt-transport-https debconf-utils build-essential 
RUN apt-get -y install gcc
# install FreeTDS and dependencies
RUN apt-get update \
 && apt-get install unixodbc -y \
 && apt-get install unixodbc-dev -y \
 && apt-get install freetds-dev -y \
 && apt-get install freetds-bin -y \
 && apt-get install tdsodbc -y \
 && apt-get install --reinstall build-essential -y
# populate "ocbcinst.ini" as this is where ODBC driver config sits
RUN echo "[FreeTDS]\n\
Description = FreeTDS Driver\n\
Driver = /usr/lib/x86_64-linux-gnu/odbc/libtdsodbc.so\n\
Setup = /usr/lib/x86_64-linux-gnu/odbc/libtdsS.so" >> /etc/odbcinst.ini

# install requirements
RUN python -m pip install --upgrade pip

COPY requirements_docker_python.txt .
RUN pip install --upgrade --no-cache -r requirements_docker_python.txt

# copy all python scripts into root directory
COPY . /

# specify work directory as root
WORKDIR /

ENTRYPOINT ["python"]

and added into requirements_docker_python.txt a new library: pyodbc==4.0.35

But after that I met an unexpected behaviour: ALL the newly triggered DAGs' tasks stuck in queued state.

Then I rolled back my Dockefile modifications - and execution of the tasks has been restored

I tried to add again the new changes into Dockerfile - and again all the tasks stuck in queued state! At this time in the Airflow scheduler log group I see continiously repeated sequence of events:

• Airflow Connection: aws_conn_id=aws_default
• No credentials retrieved from Connection
• Creating session with aws_access_key_id=None region_name=eu-central-1
• role_arn is None

After all the checks I decided to roll back my Dockerfile's changes again - and as result the normal execution of the DAGs is again restored (here is the screenshot of my last MR's changes):

BUT THESE CHANGES AFFECT JUST DOCKER IMAGE WE TRIGGER FROM THE TASKS!

So, I don't understand how it can affect the MWAA instance itself

Any suggestions how my Dockerfile changes can be related with the general Airflow tasks stuck?