Loading…
This event has ended. Visit the official site or create your own event on Sched.
Hosted by D2iQ and Container Solutions!  Register here: cloud-native-summit.com.

Looking for a specific timezone? Click one of the links below:


Back To Schedule
Wednesday, April 1 • 3:00pm - 3:45pm
Running Airflow Reliably With Kubernetes

Sign up or log in to save this to your schedule, view media, leave feedback and see who's attending!

Stream here: ​https://gateway.on24.com/wcc/eh/2010041/lp/2234394/astronomer-running-airflow-reliably-with-kubernetes

Apache Airflow is a Python-based task orchestrator that has seen widespread adoption among startups and enterprises alike to author, schedule, and monitor data workflows.

By deploying the Airflow stack via Helm on Kubernetes, fresh environments can be easily spun up or down, scaling to near 0 when no jobs are running. As companies scale up their Airflow usage, they need more control, and observability over their stack as it becomes more ingrained into their culture and more important to the business.

This talk will go through the technical challenges of supporting thousands of airflow deployments, how to monitor them, reliably push DAG updates, and how to build all the supporting infrastructure of a rock-solid Airflow system in a cloud native environment using open source software.

Speakers
avatar for Viraj Parekh

Viraj Parekh

Head of Field Engineering, Astronomer
One of the first members of the Astronomer team, Viraj helps with everything from data analytics to Airflow user training to all-things Kube. As head of field engineering, he's the bridge between Astronomer's core engineering teams and the growing market with a focus on adoption within... Read More →


Wednesday April 1, 2020 3:00pm - 3:45pm CEST
Track 3