Ian Boardman | 18 Oct 20:36 2012
Picon

celeryd start command silently exits with no error

Running Ubuntu Lucid, Django 1.4, Celery 2.4.6, Python 2.6.  The init.d script for our celery daemons worked up until a recent reboot.  I am no expert in this package, but I have now poked around with the code and the documentation for several hours, so I should be able to make good use of any suggestions. Below is what happens when I run the startup script:

$ sudo bash -x /etc/init.d/celeryd start
+ DEFAULT_PID_FILE=/var/run/celeryd <at> %n.pid
+ DEFAULT_LOG_FILE=/var/log/celeryd <at> %n.log
+ DEFAULT_LOG_LEVEL=INFO
+ DEFAULT_NODES=celery
+ DEFAULT_CELERYD='-m celery.bin.celeryd_detach'
+ CELERY_DEFAULTS=/etc/default/celeryd
+ test -f /etc/default/celeryd
+ . /etc/default/celeryd
++ CELERYD_NODES=evitap-asrmaster
++ CELERYD_CHDIR=/var/django/sites/lm/
++ CELERYD_MULTI='/var/django/sites/lm//manage.py celeryd_multi'
++ CELERYCTL='/var/django/sites/lm//manage.py celeryctl'
++ CELERYD_OPTS=--concurrency=8
++ CELERY_CONFIG_MODULE=pipeline.utils.celeryconfig
++ CELERYD_LOG_FILE=/var/log/celery/%n.log
++ CELERYD_PID_FILE=/var/run/celery/%n.pid
++ CELERYD_USER=celery
++ CELERYD_GROUP=celery
++ export DJANGO_SETTINGS_MODULE=settings
++ DJANGO_SETTINGS_MODULE=settings
+ '[' -f /etc/default/celeryd ']'
+ . /etc/default/celeryd
++ CELERYD_NODES=__host_name__
++ CELERYD_CHDIR=/var/django/sites/lm/
++ CELERYD_MULTI='/var/django/sites/lm//manage.py celeryd_multi'
++ CELERYCTL='/var/django/sites/lm//manage.py celeryctl'
++ CELERYD_OPTS=--concurrency=8
++ CELERY_CONFIG_MODULE=pipeline.utils.celeryconfig
++ CELERYD_LOG_FILE=/var/log/celery/%n.log
++ CELERYD_PID_FILE=/var/run/celery/%n.pid
++ CELERYD_USER=celery
++ CELERYD_GROUP=celery
++ export DJANGO_SETTINGS_MODULE=settings
++ DJANGO_SETTINGS_MODULE=settings
+ CELERYD_PID_FILE=/var/run/celery/%n.pid
+ CELERYD_LOG_FILE=/var/log/celery/%n.log
+ CELERYD_LOG_LEVEL=INFO
+ CELERYD_MULTI='/var/django/sites/lm//manage.py celeryd_multi'
+ CELERYD='-m celery.bin.celeryd_detach'
+ CELERYCTL='/var/django/sites/lm//manage.py celeryctl'
+ CELERYD_NODES=evitap-asrmaster
+ export CELERY_LOADER
+ '[' -n '' ']'
++ dirname /var/log/celery/%n.log
+ CELERYD_LOG_DIR=/var/log/celery
++ dirname /var/run/celery/%n.pid
+ CELERYD_PID_DIR=/var/run/celery
+ '[' '!' -d /var/log/celery ']'
+ '[' '!' -d /var/run/celery ']'
+ '[' -n celery ']'
+ DAEMON_OPTS=' --uid=celery'
+ chown celery /var/log/celery /var/run/celery
+ '[' -n celery ']'
+ DAEMON_OPTS=' --uid=celery --gid=celery'
+ chgrp celery /var/log/celery /var/run/celery
+ '[' -n /var/django/sites/lm/ ']'
+ DAEMON_OPTS=' --uid=celery --gid=celery --workdir="/var/django/sites/lm/"'
+ export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/X11R6/bin:/usr/sbin:/sbin
+ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/X11R6/bin:/usr/sbin:/sbin
+ case "$1" in
+ check_dev_null
+ '[' '!' -c /dev/null ']'
+ start_workers
+ /var/django/sites/lm//manage.py celeryd_multi start __host_name__ --uid=celery --gid=celery '--workdir="/var/django/sites/lm/"' --pidfile=/var/run/celery/%n.pid --logfile=/var/log/celery/%n.log --loglevel=INFO '--cmd=-m celery.bin.celeryd_detach' --concurrency=8
[1;36mceleryd-multi v2.4.6 [0m
> Starting nodes...
> __host_name__.__host_name__: [1;32mOK [0m
+ exit 0

After this, I find nothing running.  The log file under /var/run/celery isn't even touched.  There are no errors in our syslog. 
Please advise!

--Ian Boardman


 

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/celery-users/-/WMIQtLKXNvcJ.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To unsubscribe from this group, send email to celery-users+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/celery-users?hl=en.
Ian Boardman | 18 Oct 21:22 2012
Picon

Re: celeryd start command silently exits with no error

In my post, I meant to say /var/log/celery is not touched, but there is 
also is nothing under /var/run/celery. I checked that /var/log/celery and 
/var/run/celery are both writeable by celery.

James Reynolds | 18 Oct 23:06 2012
Picon

Re: celeryd start command silently exits with no error

Try SUing to celery and manually starting celery, example if django:

python manage.py celeryd

I will be that user/group celery doesn't have access to files that are needed to run the app. In my case, in was various log / lock files in /var/log/(application)

On Thursday, October 18, 2012 3:22:18 PM UTC-4, Ian Boardman wrote:

In my post, I meant to say /var/log/celery is not touched, but there is also is nothing under /var/run/celery. I checked that /var/log/celery and /var/run/celery are both writeable by celery.
 

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/celery-users/-/BsVw4MoV0KEJ.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To unsubscribe from this group, send email to celery-users+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/celery-users?hl=en.
Ian Boardman | 18 Oct 23:41 2012
Picon

Re: celeryd start command silently exits with no error

James,

  I tried it and see no error related to permissions.  But thanks for the suggestion.
-- Ian

On Thursday, October 18, 2012 5:06:56 PM UTC-4, James Reynolds wrote:
Try SUing to celery and manually starting celery, example if django:

python manage.py celeryd

I will be that user/group celery doesn't have access to files that are needed to run the app. In my case, in was various log / lock files in /var/log/(application)

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/celery-users/-/lj85vxE5eMYJ.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To unsubscribe from this group, send email to celery-users+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/celery-users?hl=en.
Matt Long | 19 Oct 03:05 2012

Re: Re: celeryd start command silently exits with no error

Does running "python manage.py celeryd" on the command line in the appropriate directory start up successfully? That'll help determine if the issue is isolated to your daemonization or not.

Aside from that, I'm comparing your init.d script to my own, and the main difference is that your --cmd parameter is:

-m celery.bin.celeryd_detach

while mine is:

/usr/bin/python /path/to/django/project/manage.py celeryd

I would try changing your --cmd to invoking celeryd via the manage.py script to see if that helps.

I would also try temporarily making it run as root rather than celery to rule out permission issues or not.

Hope that helps!

On Thu, Oct 18, 2012 at 2:41 PM, Ian Boardman <isb0459-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
James,
  I tried it and see no error related to permissions.  But thanks for the suggestion.
-- Ian


On Thursday, October 18, 2012 5:06:56 PM UTC-4, James Reynolds wrote:
Try SUing to celery and manually starting celery, example if django:

python manage.py celeryd

I will be that user/group celery doesn't have access to files that are needed to run the app. In my case, in was various log / lock files in /var/log/(application)

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/celery-users/-/lj85vxE5eMYJ.

To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To unsubscribe from this group, send email to celery-users+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/celery-users?hl=en.



--
Matt Long
Cofounder and Lead Developer

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To unsubscribe from this group, send email to celery-users+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/celery-users?hl=en.
Ian Boardman | 19 Oct 19:19 2012
Picon

Re: Re: celeryd start command silently exits with no error

Hi.  So I ran this as user celery with some of the usual options for the logs, and I got an unexpected error about SOUTH_DATABASE_ADAPTERS.  As I said, I had little knowledge of django, and had nothing to do with the installation I was trying to debug.  So hunting down the what, where and how down "South," I finally followed the directions and put in an explicit configuration in the django site settings, even though up to a week ago, celeryd started up completely fine w/o any such configuration.  

  The error does not appear when one runs the startup from the script in /etc/init.d because the ultimate command execution (i.e. the one that I ran by hand, as Matt suggested, which exec's celeryd) occurs under a process which has had its stderr closed. Specifically, when manage.py imports the module south, __init__.py is run and hits the error on the missing configuration, but the error message disappears into /dev/null. You won't see the error when this command is being run after several intermediate calls in a detached (and muted) child off the parent shell.
  Thanks to all for helping.
--Ian


On Thursday, October 18, 2012 9:05:37 PM UTC-4, Matt Long wrote:
Does running "python manage.py celeryd" on the command line in the appropriate directory start up successfully? That'll help determine if the issue is isolated to your daemonization or not.

Aside from that, I'm comparing your init.d script to my own, and the main difference is that your --cmd parameter is:

-m celery.bin.celeryd_detach

while mine is:

/usr/bin/python /path/to/django/project/manage.py celeryd

I would try changing your --cmd to invoking celeryd via the manage.py script to see if that helps.

I would also try temporarily making it run as root rather than celery to rule out permission issues or not.

Hope that helps!

On Thu, Oct 18, 2012 at 2:41 PM, Ian Boardman <isb...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
James,
  I tried it and see no error related to permissions.  But thanks for the suggestion.
-- Ian


On Thursday, October 18, 2012 5:06:56 PM UTC-4, James Reynolds wrote:
Try SUing to celery and manually starting celery, example if django:

python manage.py celeryd

I will be that user/group celery doesn't have access to files that are needed to run the app. In my case, in was various log / lock files in /var/log/(application)

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/celery-users/-/lj85vxE5eMYJ.

To post to this group, send email to celery...-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To unsubscribe from this group, send email to celery-users... <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/celery-users?hl=en.



--
Matt Long
Cofounder and Lead Developer

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/celery-users/-/KEff7_M7xMMJ.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To unsubscribe from this group, send email to celery-users+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/celery-users?hl=en.

Gmane