Some task are not processing using Django-Q

Issue

I have a django Q cluster running with this configuration:

Q_CLUSTER = {
    'name': 'pretty_name',
    'workers': 1,
    'recycle': 500,
    'timeout': 500,
    'queue_limit': 5,
    'cpu_affinity': 1,
    'label': 'Django Q',
    'save_limit': 0,
    'ack_failures': True,
    'max_attempts': 1,
    'attempt_count': 1,
    'redis': {
        'host': CHANNEL_REDIS_HOST,
        'port': CHANNEL_REDIS_PORT,
        'db': 5,
    }

}

On this cluster I have a scheduled task supposed to run every 15 minutes.
Sometimes it works fine and this is what I can see on my worker logs:

[Q] INFO Enqueued 1
[Q] INFO Process-1 created a task from schedule [2]
[Q] INFO Process-1:1 processing [oranges-georgia-snake-social]
[ My Personal Custom Task Log]
[Q] INFO Processed [oranges-georgia-snake-social]

But other times the task does not start, this is what I get on my log:

[Q] INFO Enqueued 1
[Q] INFO Process-1 created a task from schedule [2]

And then nothing for the next 15 minutes.

Any idea where this might come from ?

Solution

So this was my prod environment and it appears that my dev environment was using the same redis db and even though no task existed on my dev environment it seems that this was the cause of the issue.

The solution was to change the redis db between my dev and prod environment !

Answered By – Kimor

This Answer collected from stackoverflow, is licensed under cc by-sa 2.5 , cc by-sa 3.0 and cc by-sa 4.0

Leave a Reply

(*) Required, Your email will not be published