AMQP Messaging, Task, and Queue handling in Python using Celery. ()

Nike | 31 May 2012 16:59

tasks aren't executed under some conditions

Hi all!

First of all our setup: celery runs on a separate ec2 64-bit instance
under supervisor and uses eventlet + redis as a backend. We have one
periodic task with 2 mins interval that may produce a lot of regular
tasks. After some time the tasks stop to execute and only this is in
logs

[2012-05-31 14:40:37,438: INFO/Beat] Scheduler: Sending due task
xxx.tasks.periodic_updates
[2012-05-31 14:42:37,525: INFO/Beat] Scheduler: Sending due task
xxx.tasks.periodic_updates
[2012-05-31 14:44:37,625: INFO/Beat] Scheduler: Sending due task
xxx.tasks.periodic_updates
[2012-05-31 14:46:37,715: INFO/Beat] Scheduler: Sending due task
xxx.tasks.periodic_updates
[2012-05-31 14:48:37,818: INFO/Beat] Scheduler: Sending due task
xxx.tasks.periodic_updates
[2012-05-31 14:50:37,854: INFO/Beat] Scheduler: Sending due task
xxx.tasks.periodic_updates
[2012-05-31 14:52:37,958: INFO/Beat] Scheduler: Sending due task
xxx.tasks.periodic_updates

maybe related but I don't have many proofs - I suspect it somehow
related to a number of exceptions regular tasks throw. I noticed that
after 4 raised exceptions (i.e. 4 regular tasks completed with an
error) celery switches to this "dead" mode

looking into redis I see that its own queue grows by 1 for every
periodic task (every 2 mins)

(Continue reading)

headers

Ask Solem | 3 Jun 2012 23:03

Re: tasks aren't executed under some conditions


On 31 May 2012, at 15:59, Nike wrote:

> Hi all!
> 
> First of all our setup: celery runs on a separate ec2 64-bit instance
> under supervisor and uses eventlet + redis as a backend. We have one
> periodic task with 2 mins interval that may produce a lot of regular
> tasks. After some time the tasks stop to execute and only this is in
> logs
> 
> 
> [2012-05-31 14:40:37,438: INFO/Beat] Scheduler: Sending due task
> xxx.tasks.periodic_updates
> [2012-05-31 14:42:37,525: INFO/Beat] Scheduler: Sending due task
> xxx.tasks.periodic_updates
> [2012-05-31 14:44:37,625: INFO/Beat] Scheduler: Sending due task
> xxx.tasks.periodic_updates
> [2012-05-31 14:46:37,715: INFO/Beat] Scheduler: Sending due task
> xxx.tasks.periodic_updates
> [2012-05-31 14:48:37,818: INFO/Beat] Scheduler: Sending due task
> xxx.tasks.periodic_updates
> [2012-05-31 14:50:37,854: INFO/Beat] Scheduler: Sending due task
> xxx.tasks.periodic_updates
> [2012-05-31 14:52:37,958: INFO/Beat] Scheduler: Sending due task
> xxx.tasks.periodic_updates
> 
> maybe related but I don't have many proofs - I suspect it somehow
> related to a number of exceptions regular tasks throw. I noticed that
> after 4 raised exceptions (i.e. 4 regular tasks completed with an

(Continue reading)

headers

nuklea | 8 Nov 2012 09:56

Re: tasks aren't executed under some conditions

Have same problem.

четверг, 31 мая 2012 г., 22:59:49 UTC+8 пользователь Nike написал:

Hi all!

First of all our setup: celery runs on a separate ec2 64-bit instance
under supervisor and uses eventlet + redis as a backend. We have one
periodic task with 2 mins interval that may produce a lot of regular
tasks. After some time the tasks stop to execute and only this is in
logs

[2012-05-31 14:40:37,438: INFO/Beat] Scheduler: Sending due task
xxx.tasks.periodic_updates
[2012-05-31 14:42:37,525: INFO/Beat] Scheduler: Sending due task
xxx.tasks.periodic_updates
[2012-05-31 14:44:37,625: INFO/Beat] Scheduler: Sending due task
xxx.tasks.periodic_updates
[2012-05-31 14:46:37,715: INFO/Beat] Scheduler: Sending due task
xxx.tasks.periodic_updates
[2012-05-31 14:48:37,818: INFO/Beat] Scheduler: Sending due task
xxx.tasks.periodic_updates
[2012-05-31 14:50:37,854: INFO/Beat] Scheduler: Sending due task
xxx.tasks.periodic_updates
[2012-05-31 14:52:37,958: INFO/Beat] Scheduler: Sending due task
xxx.tasks.periodic_updates

maybe related but I don't have many proofs - I suspect it somehow
related to a number of exceptions regular tasks throw. I noticed that
after 4 raised exceptions (i.e. 4 regular tasks completed with an
error) celery switches to this "dead" mode

looking into redis I see that its own queue grows by 1 for every
periodic task (every 2 mins)

redis 127.0.0.1:6377> llen celery
(integer) 80

here is how it's run in config
/manage.py celeryd -l INFO -c 1000 -P eventlet -B --time-limit=50

if I restart celery it executes all tasks from its queue and keeps
executing them for some time.

any ideas what happens here?

Thanks

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/celery-users/-/wpjEZNLkDFsJ.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To unsubscribe from this group, send email to celery-users+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/celery-users?hl=en.

headers

Ask Solem | 9 Nov 2012 14:23

Re: tasks aren't executed under some conditions


On 8 Nov 2012, at 08:56, nuklea <nuklea@...> wrote:

> Have same problem.
> 
> четверг, 31 мая 2012 г., 22:59:49 UTC+8 пользователь Nike написал:
> Hi all! 
> 
> First of all our setup: celery runs on a separate ec2 64-bit instance 
> under supervisor and uses eventlet + redis as a backend. We have one 
> periodic task with 2 mins interval that may produce a lot of regular 
> tasks. After some time the tasks stop to execute and only this is in 
> logs 
> 
> 
> [2012-05-31 14:40:37,438: INFO/Beat] Scheduler: Sending due task 
> xxx.tasks.periodic_updates 
> [2012-05-31 14:42:37,525: INFO/Beat] Scheduler: Sending due task 
> xxx.tasks.periodic_updates 
> [2012-05-31 14:44:37,625: INFO/Beat] Scheduler: Sending due task 
> xxx.tasks.periodic_updates 
> [2012-05-31 14:46:37,715: INFO/Beat] Scheduler: Sending due task 
> xxx.tasks.periodic_updates 
> [2012-05-31 14:48:37,818: INFO/Beat] Scheduler: Sending due task 
> xxx.tasks.periodic_updates 
> [2012-05-31 14:50:37,854: INFO/Beat] Scheduler: Sending due task 
> xxx.tasks.periodic_updates 
> [2012-05-31 14:52:37,958: INFO/Beat] Scheduler: Sending due task 
> xxx.tasks.periodic_updates 
> 
> maybe related but I don't have many proofs - I suspect it somehow 
> related to a number of exceptions regular tasks throw. I noticed that 
> after 4 raised exceptions (i.e. 4 regular tasks completed with an 
> error) celery switches to this "dead" mode 
> 

Do you have a log of the exceptions?

Could you start the worker with --loglevel=DEBUG?

What is the output of the commands:

    $ celery report
    $ celery inspect active
    $ celery inspect reserved
    $ celery inspect scheduled
    $ celery inspect stats

If you use a custom app then you must add that to the commands:

    $ celery -A app ...

or if you use django-celery then you must add manage.py:

    $ python manage.py celery ...  

-- 
Ask Solem
twitter.com/asksol | +44 (0)7713357179

--

-- 
You received this message because you are subscribed to the Google Groups "celery-users" group.
To post to this group, send email to celery-users@...
To unsubscribe from this group, send email to celery-users+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/celery-users?hl=en.