running django tests --parallel and splitting log files per test-runner worker

41 Views Asked by At

I'm using manage.py test --parallel to run my tests and want to create a separate log file for each test runner.

Currently, all the test runner worker write to the same log file, so I get a single log file with contents "striped" like this:

[ForkPoolWorker-2] ...
[ForkPoolWorker-1] ...
[ForkPoolWorker-1] ...
[ForkPoolWorker-3] ...
[ForkPoolWorker-2] ...
[ForkPoolWorker-4] ...
[ForkPoolWorker-1] ...

I create a custom configure_logging() method like this:

import logging.config
import os


def configure_logging() -> None:
    """Custom logging configuration with process-id named files"""

    process_id = os.getpid()
    log_file_path = f"runner-{process_id}.log"

    LOGGING = {
        "version": 1,
        "disable_existing_loggers": False,
        "handlers": {
            "file": {
                "level": "DEBUG",
                "class": "logging.FileHandler",
                "filename": log_file_path,
            },
        },
        "loggers": {
            "django.log": {
                "handlers": ["file"],
                "level": "DEBUG",
                "propagate": True,
            },
        },
    }

    logging.config.dictConfig(LOGGING)

I'm using a custom test-runner to wire things up:

from django.test.runner import DiscoverRunner

from myapp.tests.logging_config import configure_logging


class ParallelTestRunner(DiscoverRunner):
    """Parallel test runner with separate log files for each worker"""

    def setup_test_environment(self, **kwargs):
        """Configure the test environment with our custom log setup"""

        super().setup_test_environment(**kwargs)
        # Configure logging with a unique file per test process
        configure_logging()

I wire that up in settings.py like this:

TEST_RUNNER = "myapp.tests.runners.ParallelTestRunner"

However, when I look for the log files, it appears it's only creating a single log file like runner-16.log, so I think the name-generation for the log files it happening in the testrunner main process, before the worker split.

Any ideas on how I can wire this up so that the log files get created per testrunner worker? I'd like to see log files like:

runner-11.log
runner-12.log
runner-13.log
runner-14.log
runner-15.log
runner-16.log

Thanks!

0

There are 0 best solutions below