Django Running Celery with Supervisor Celery + RabbitMQ with Supervisor
Example
Celery requires a broker to handle message-passing. We use RabbitMQ because it’s easy to setup and it is well supported.
Install rabbitmq using the following command
sudo apt-get install rabbitmq-server
Once the installation is complete, create user, add a virtual host and set permissions.
sudo rabbitmqctl add_user myuser mypassword
sudo rabbitmqctl add_vhost myvhost
sudo rabbitmqctl set_user_tags myuser mytag
sudo rabbitmqctl set_permissions -p myvhost myuser ".*" ".*" ".*"
To start the server:
sudo rabbitmq-server
We can install celery with pip:
pip install celery
In your Django settings.py file, your broker URL would then look something like
BROKER_URL = 'amqp://myuser:mypassword@localhost:5672/myvhost'
Now start the celery worker
celery -A your_app worker -l info
This command start a Celery worker to run any tasks defined in your django app.
Supervisor is a Python program that allows you to control and keep running any unix processes. It can also restart crashed processes. We use it to make sure Celery workers are always running.
First, Install supervisor
sudo apt-get install supervisor
Create your_proj.conf file in your supervisor conf.d (/etc/supervisor/conf.d/your_proj.conf):
[program:your_proj_celery]
command=/home/your_user/your_proj/.venv/bin/celery --app=your_proj.celery:app worker -l info
directory=/home/your_user/your_proj
numprocs=1
stdout_logfile=/home/your_user/your_proj/logs/celery-worker.log
stderr_logfile=/home/your_user/your_proj/logs/low-worker.log
autostart=true
autorestart=true
startsecs=10
Once our configuration file is created and saved, we can inform Supervisor of our new program through the supervisorctl command. First we tell Supervisor to look for any new or changed program configurations in the /etc/supervisor/conf.d directory with:
sudo supervisorctl reread
Followed by telling it to enact any changes with:
sudo supervisorctl update
Once our programs are running, there will undoubtedly be a time when we want to stop, restart, or see their status.
sudo supervisorctl status
For restart your celery instance:
sudo supervisorctl restart your_proj_celery
