Dagster + Kubernetes
Launch runs as Kubernetes Jobs. Use a Helm chart to deploy Dagster on a K8s cluster.
About this integration
This library contains utilities for running Dagster with Kubernetes. This includes a Python API allowing Dagit to launch runs as Kubernetes Jobs, as well as a Helm chart you can use as the basis for a Dagster deployment on a Kubernetes cluster.
Installation
pip install dagster-k8s
Example
from dagster import job
from dagster_k8s import k8s_job_op
first_op = k8s_job_op.configured(
{
"image": "busybox",
"command": ["/bin/sh", "-c"],
"args": ["echo HELLO"],
"kubeconfig_file": "path/to/kubeconfig",
},
name="first_op",
)
second_op = k8s_job_op.configured(
{
"image": "busybox",
"command": ["/bin/sh", "-c"],
"args": ["echo GOODBYE"],
"kubeconfig_file": "path/to/kubeconfig",
},
name="second_op",
)
@job
def full_job():
second_op(first_op())
About Kubernetes
Kubernetes is an open-source container orchestration system for automating software deployment, scaling, and management. Google originally designed Kubernetes, but the Cloud Native Computing Foundation now maintains the project.