Workflow orchestration#
Vineyard seamlessly integrates with the workflow orchestration engines, e.g., Apache Airflow and Kedro, enabling users to effortlessly incorporate Vineyard into their workflows for enhanced performance.
Moreover, the Airflow integration empowers users to work with large Python objects
featuring complex data types (e.g., pandas.DataFrame) at minimal cost, while
eliminating the need for cumbersome pickle.dump/loads operations.
Airflow uses vineyard as the XCom backend to efficiently handle complex data in Python.
The Kedro integration enables users to easily share large data objects across nodes in a pipeline and eliminates the high cost of (de)serialization and I/O compared with alternatives like AWS S3 or Minio, without the need to modify the pipeline code intrusively, and provides seamless user experience when scaling pipelines to Kubernetes.
Kedro uses vineyard as a DataSet implementation for efficient intermediate data sharing.