Task overlap in Django-Q

Question

I have a task that I want to run every minute so the data is as fresh as possible. However depending on the size of the update it can take longer than one minute to finish. Django-Q creates new task and queues it every minute though so there is some overlap synchronizing the same data pretty much. Is it possible to not schedule the task that is already in progress?

Answer 1

I ended up creating decorator that locks the task execution and on new task run just returns immediately if the lock is not available. Timeout is 1 hour (enough in my case).

from functools import wraps

from django.core.cache import cache
from redis.exceptions import LockNotOwnedError


def django_q_task_lock(func):
    """
    Decorator for django q tasks for preventing overlap in parallel task runs
    """

    @wraps(func)
    def wrapped_task(*args, **kwargs):
        task_lock = cache.lock(f"django_q-{func.__name__}", timeout=60 * 60)
        if task_lock.acquire(blocking=False):
            try:
                func(*args, **kwargs)
            except Exception as e:
                try:
                    task_lock.release()
                except LockNotOwnedError:
                    pass
                raise e
            try:
                task_lock.release()
            except LockNotOwnedError:
                pass

    return wrapped_task

@django_q_task_lock
def potentialy_long_running_task():
    ...
    # task logic
    ...