Vineyard: an In-Memory Data Manager for Data-Intensive Analytics

An innovative cloud-native in-memory immutable data manager that offers out-of-the-box high-level abstractions and zero-copy in-memory sharing for distributed data in various big data tasks.

pip install vineyard Successfully installed vineyard python client = vineyard.connect() object_id = client.put('Hello, vineyard!') client.get(object_id) 'Hello, vineyard!'

Efficient data sharing

Vineyard shares immutable data across different systems using shared memory without extra overheads, eliminating the overhead of serialization and deserialization.

Learn More

Out-of-the-box data abstraction

Vineyard defines a metadata-payload separated data model to capture the payload commonalities and method commonalities between sharable objects.

Learn More

Pluggable I/O routines

Vineyard provides common manipulation routines for immutable data as drivers, which extend the capabilities of data structures by registering appropriate drivers.

Learn More

Data orchestration on Kubernetes

Vineyard provides efficient distributed data sharing in cloud-native environments by embracing cloud-native big data processing.

Learn More

Use Cases

Object store for all

Putting and getting arbitrary objects using Vineyard, in a zero-copy way!

Learn More

Data sharing in Kedro pipelines

Sharing intermediate data between tasks in Kedro pipelines.

Learn More

Data processing for machine learning

Sharing large objects between different systems in data preprocessing pipelines in machine learning applications.

Learn More

Data sharing on Kubernetes

Coordinating the flow of objects and jobs on Kubernetes with the data-aware scheduler plugin.

Learn More