Better Airflow with Metaflow

8 points by vtuulos 2 years ago | 3 comments
  • chse_cake 2 years ago
    I have used Airflow as a product for over a year and I can say for sure that Airflow sucks for orchestrating ML.

    The ability to pass state in Airflow absolutely sucks. It's interesting that Metaflow is trying to compile into Airflow but I am skeptical about it's scalability on Airflow especially with DAGs that have a larger task footprint.

    • savin-goyal 2 years ago
      Metaflow passes state using an object store (s3, azure blob store, etc.) even within Airflow - short circuiting Airflow's xcom machinery. But agreed, Airflow presents many more scalability challenges - this integration addresses a few in-place as well as preserves the ability to swap out Airflow with a more scalable workflow orchestrator if needed.
      • vtuulos 2 years ago
        Totally! One intended use case of this integration is that it allows you to move easily to a more scalable orchestrator if you hit limits of Airflow