Flower, No such file or directory: '/multiproc_dir/gauge_all_8054.db'

Question

I am trying to run flower in docker container using the command

celery -A <my_task_folder> flower

and I get the following error:

[E 230619 06:35:06 ioloop:758] Exception in callback functools.partial(<bound method EventsState.event of <State: events=4 tasks=0>>, {'hostname': 'celery@2667b382dde2', 'utcoffset': 0, 'pid': 6802, 'clock': 185, 'freq': 2.0, 'active': 4, 'processed': 2722, 'loadavg': [1.28, 0.74, 0.41], 'sw_ident': 'py-celery', 'sw_ver': '5.2.7', 'sw_sys': 'Linux', 'timestamp': 1687156506.5616019, 'type': 'worker-heartbeat', 'local_received': 1687156506.5646658}) Traceback (most recent call last): File "/usr/local/lib/python3.10/site-packages/tornado/ioloop.py", line 738, in _run_callback ret = callback() File "/usr/local/lib/python3.10/site-packages/flower/events.py", line 91, in event self.metrics.worker_online.labels(worker_name).set(1) File "/usr/local/lib/python3.10/site-packages/prometheus_client/metrics.py", line 190, in labels self._metrics[labelvalues] = self.class( File "/usr/local/lib/python3.10/site-packages/prometheus_client/metrics.py", line 365, in init super().init( File "/usr/local/lib/python3.10/site-packages/prometheus_client/metrics.py", line 138, in init self._metric_init() File "/usr/local/lib/python3.10/site-packages/prometheus_client/metrics.py", line 378, in _metric_init self._value = values.ValueClass( File "/usr/local/lib/python3.10/site-packages/prometheus_client/values.py", line 68, in init self.__reset() File "/usr/local/lib/python3.10/site-packages/prometheus_client/values.py", line 82, in __reset files[file_prefix] = MmapedDict(filename) File "/usr/local/lib/python3.10/site-packages/prometheus_client/mmap_dict.py", line 62, in init self._f = open(filename, 'rb' if read_mode else 'a+b') FileNotFoundError: [Errno 2] No such file or directory: '/multiproc_dir/gauge_all_8054.db'

The versions of the relevant packages are as follows:

redis = "4.3.1"
celery = "5.2.7"
flower = "1.0.0"
prometheus-fastapi-instrumentator = "5.8.2"
prometheus-client = "0.17.0"

and dockerfile for creating the image is given by:

FROM python:3.10

WORKDIR /app/

# Install Poetry
RUN curl -sSL https://install.python-poetry.org | POETRY_HOME=/opt/poetry python - --version 1.4.2 && cd /usr/local/bin && ln -s /opt/poetry/bin/poetry && poetry config virtualenvs.create false

# Copy poetry.lock* in case it doesn't exist in the repo
COPY pyproject.toml poetry.lock* /app/

# Allow installing dev dependencies to run tests
ARG INSTALL_DEV=false
RUN bash -c "if [ $INSTALL_DEV == 'true' ] ; then poetry install --no-root ; else poetry install --no-root --no-dev ; fi"

ENV prometheus_multiproc_dir=/multiproc_dir

COPY . /app

EXPOSE 5000

CMD ["gunicorn", "--workers", "4", "--worker-class", "uvicorn.workers.UvicornWorker", "--bind", "0.0.0.0:5000", "src.main:app"]

Answer 1

This has nothing to do with Flower/Celery, but rather prometheus. As the error message is saying, /multiproc_dir does not exist. Simply creating the folder in your Dockerfile should solve the problem:

ENV prometheus_multiproc_dir=/multiproc_dir
RUN mkdir -p $prometheus_multiproc_dir