About this integration
With the dagster-celery
integration you can execute and monitor Dagster jobs as Celery tasks.
Installation
pip install dagster-celery
Example
# Materialize your assets with Celery
# Read the docs on Executors to learn more: https://docs.dagster.io/deployment/executors
from dagster_celery import celery_executor
from dagster import define_asset_job
executor = celery_executor.configured({
'broker': 'pyamqp://guest@localhost//', # The URL of the Celery broker
'backend': 'rpc://', # The URL of the Celery results backend
'include': ['my_module'], # Modules every worker should import
})
celery_enabled_job = define_asset_job("celery_enabled_job", executor_def=executor)
About Celery
Celery is a simple, flexible, and reliable distributed system to process vast amounts of messages, while providing operations with the tools required to maintain such a system. It’s a task queue with focus on real-time processing, while also supporting task scheduling. The Celery Executor makes it easier for developers to scale up applications by distributing the tasks on multiple machines.