Kafka and Django management command

181 Views Asked by At

Hello everyone hope I find you well!

So I'm trying to setup a new project with Django and a Kafka consumer. I'm using confluent-python package to create my consumer. So I'm facing a very odd situation. When I start my project through my docker entrypoint I run a django management command with the following code:

import simplejson
from confluent_kafka import Consumer
from django.core.management import BaseCommand
from loguru import logger
from apps.<my_service> import KafkaHandler


class Command(BaseCommand):
    help = "Start Reporting Kafka Consumer"

    def handle(self, *args, **kwargs):
        logger.debug("Starting consumer")
        consumer = Consumer(
            {"bootstrap.servers": "kafka:19091", "group.id": "reporting"}
        )
        consumer.subscribe(["test_producer"])
        kafka_handler = KafkaHandler()

        try:
            while True:
                logger.debug("Consuming")
                msg = consumer.poll(1.0)

                if msg is None:
                    continue
                if msg.error():
                    logger.debug(f"Consumer error: {msg.error()}")
                    continue

                event_message = msg.value().decode()
                logger.debug(f"Received message: {event_message}")
                json_message = simplejson.loads(event_message)
                kafka_handler.handle(json_message)

        finally:
            consumer.close()

After I start my project through command docker-compose up I see that the log "Start consumer" is printed two times and that I have two consumer connections instead of only one so I suspect that this code is being run twice which is very odd. Can someone help me with this situation? Bellow I share my docker-entrypoint

!/bin/bash

set -e

if [[ "${PROJECT_RUN_MODE}" == "production" ]]; then
    exec <run production mode>
elif [[ "${PROJECT_RUN_MODE}" == "develop" ]]; then
    if [[ $# == 0 ]]; then
        echo "Starting dev server..."
        exec python "${PROJECT_DIR}/manage.py" start_consumer
    else
        exec "$@"
    fi
else
    echo "Unknown run mode (${PROJECT_RUN_MODE}). Aborting..."
    exit 1
fi

I tried run the code as a python script with django setup and also as a management command and the problem still persisted.

UPDATE: I also tried to run the command with just one log and it still executes twice bellow I share my Dockerfile and compose.yml

Dockerfile

FROM python:3.11.1-slim as compile-image

# production / develop
ARG DEPLOYMENT_MODE="production"
ARG BASE_DIR="/opt/project"
ARG PROJECT_DIR="${PROJECT_DIR}/project_dir"
ARG DJANGO_SETTINGS_MODULE="settings.settings"
ARG DELETE_STATIC=0

ENV BASE_DIR ${BASE_DIR}
ENV PROJECT_DIR ${PROJECT_DIR}
ENV PROJECT_PORT ${PROJECT_PORT}
ENV PHOENIX_RUN_MODE ${DEPLOYMENT_MODE}
ENV DJANGO_SETTINGS_MODULE ${DJANGO_SETTINGS_MODULE}
ENV PYTHONUNBUFFERED 1
ENV DELETE_STATIC ${DELETE_STATIC}
ENV PATH="/opt/venv/bin:$PATH"

WORKDIR "${PROJECT_DIR}"
## Add code
COPY . "${PROJECT_DIR}"

RUN apt-get update && \
    apt-get -y install g++ libjpeg-dev zlib1g-dev libffi-dev default-mysql-client default-libmysqlclient-dev musl-dev curl && \
    curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | bash -s -- -y && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/* && \
    python -m venv /opt/venv && \
    pip install --upgrade pip setuptools wheel && \
    # install requirements
    pip install -r "settings/requirements/${DEPLOYMENT_MODE}.txt"


FROM python:3.11.1-slim as run-image

# production / develop

ARG DEPLOYMENT_MODE="production"
ARG BASE_DIR="/opt/project"
ARG PROJECT_DIR="${BASE_DIR}/project_dir"
ARG PHOENIX_PORT="80"
ARG DJANGO_SETTINGS_MODULE="settings.settings"
ARG STOPSIGNAL="SIGTERM"

ENV BASE_DIR ${BASE_DIR}
ENV PROJECT_DIR ${PROJECT_DIR}
ENV PROJECT_PORT ${PROJECT_PORT}
ENV PHOENIX_RUN_MODE ${DEPLOYMENT_MODE}
ENV DJANGO_SETTINGS_MODULE ${DJANGO_SETTINGS_MODULE}
ENV PYTHONUNBUFFERED 1
ENV PATH="/opt/venv/bin:$PATH"

COPY --from=compile-image /opt/venv /opt/venv
RUN apt-get update && \
    apt-get -y install supervisor g++ gettext libffi-dev default-mysql-client default-libmysqlclient-dev musl-dev && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/* && \
    mkdir -p /var/log/supervisor

WORKDIR "${PROJECT_DIR}"
## Add code
COPY . "${PROJECT_DIR}"


# Collect static files directly on the image if the build arg is enabled
RUN chmod +x "${PROJECT_DIR}/docker/docker-entrypoint.sh"  && \
    cp "${PROJECT_DIR}/docker/supervisord.conf" /etc/supervisor/conf.d/supervisord.conf

# Define entry point
ENTRYPOINT ["${PROJECT_DIR}/docker/docker-entrypoint.sh"]

# Expose server port and declare signal for gracefully stopping the container
EXPOSE ${PROJECT_PORT}
STOPSIGNAL ${STOPSIGNAL}

docker-compose.yml

version: "3"

services:
  phoenix:
    entrypoint: ["bash", "docker/docker-entrypoint.sh"]
    build:
      context: ..
      dockerfile: docker/Dockerfile
      args:
        DEPLOYMENT_MODE: develop
    stop_signal: SIGINT
    environment:
    <envs>
    ports:
    - <ports>
0

There are 0 best solutions below