Blog Archive, February 2021

In Python world Celery is a popular tool for running background tasks. It includes Celery beat that allows to run periodic tasks on some schedule. At my current work I participated in developing a system for running multiple periodic tasks. Since our technology stack is based on Python, Celery beat was a natural choice. One of the requirements to the system was granular control over individual tasks, that is, the ability to start/stop/restart each task individually. The straightforward solution was to run multiple Celery beat/worker pairs for each task, but after some googling it turned out that running multiple Celery beat instances seemed to be impossible. At least everybody said so, e.g. check this Stack Overflow discussion. If you try to do so you'll get duplicate tasks in your workers because each beat sends tasks to each worker. (...)