Django Celery + RabbitMQ with Supervisor


Example

Celery requires a broker to handle message-passing. We use RabbitMQ because it’s easy to setup and it is well supported.

Install rabbitmq using the following command

sudo apt-get install rabbitmq-server

Once the installation is complete, create user, add a virtual host and set permissions.

sudo rabbitmqctl add_user myuser mypassword
sudo rabbitmqctl add_vhost myvhost
sudo rabbitmqctl set_user_tags myuser mytag
sudo rabbitmqctl set_permissions -p myvhost myuser ".*" ".*" ".*"

To start the server:

sudo rabbitmq-server

We can install celery with pip:

pip install celery

In your Django settings.py file, your broker URL would then look something like

BROKER_URL = 'amqp://myuser:mypassword@localhost:5672/myvhost'

Now start the celery worker

celery -A your_app worker -l info

This command start a Celery worker to run any tasks defined in your django app.

Supervisor is a Python program that allows you to control and keep running any unix processes. It can also restart crashed processes. We use it to make sure Celery workers are always running.

First, Install supervisor

sudo apt-get install supervisor

Create your_proj.conf file in your supervisor conf.d (/etc/supervisor/conf.d/your_proj.conf):

[program:your_proj_celery]
command=/home/your_user/your_proj/.venv/bin/celery --app=your_proj.celery:app worker -l info
directory=/home/your_user/your_proj
numprocs=1
stdout_logfile=/home/your_user/your_proj/logs/celery-worker.log
stderr_logfile=/home/your_user/your_proj/logs/low-worker.log
autostart=true
autorestart=true
startsecs=10

Once our configuration file is created and saved, we can inform Supervisor of our new program through the supervisorctl command. First we tell Supervisor to look for any new or changed program configurations in the /etc/supervisor/conf.d directory with:

sudo supervisorctl reread

Followed by telling it to enact any changes with:

sudo supervisorctl update

Once our programs are running, there will undoubtedly be a time when we want to stop, restart, or see their status.

sudo supervisorctl status

For restart your celery instance:

sudo supervisorctl restart your_proj_celery