Integrations#

Vineyard is designed for serving as the immediate data sharing engine and has been integrated with various big-data computing engines. Namely the machine learning frameworks as well as the distributed data processing engine Dask.

Executing machine learning workflows on top of vineyard.

Using vineyard as the data source / sink of dask computations.

Vineyard has integrated to the workflow orchestrating engines (apache airflow) to helps use adopt vineyard into their own workflows for the performance gains. Moreover, the airflow integration allows user operating on large Python objects of complex data types (e.g., pandas.DataFrame) at low cost and avoid the burden of pickle.dump/loads.

Airflow uses vineyard as the XCom backend to efficiently handle complex data in Python.