Better Airflow with Metaflow
8 points by vtuulos 2 years ago | 3 comments- chse_cake 2 years agoI have used Airflow as a product for over a year and I can say for sure that Airflow sucks for orchestrating ML.
The ability to pass state in Airflow absolutely sucks. It's interesting that Metaflow is trying to compile into Airflow but I am skeptical about it's scalability on Airflow especially with DAGs that have a larger task footprint.
- savin-goyal 2 years agoMetaflow passes state using an object store (s3, azure blob store, etc.) even within Airflow - short circuiting Airflow's xcom machinery. But agreed, Airflow presents many more scalability challenges - this integration addresses a few in-place as well as preserves the ability to swap out Airflow with a more scalable workflow orchestrator if needed.
- vtuulos 2 years agoTotally! One intended use case of this integration is that it allows you to move easily to a more scalable orchestrator if you hit limits of Airflow
- savin-goyal 2 years ago