C++ API Reference

ObjectID and Object

using vineyard::ObjectID = uint64_t

ObjectID is an opaque type for vineyard’s object id. The object ID is generated by vineyard server, the underlying type of ObjectID is a 64-bit unsigned integer.

class vineyard::Object : public vineyard::ObjectBase, public std::enable_shared_from_this<Object>

Object is the core concept in vineyard. Object can be a scalar, a tuple, a vector, a tensor, or even a distrubted graph in vineyard. Objects are stored in vineyard, and can be shared to process that connects to the same vineyard cluster.

Every object in vineyard has a unique identifier ObjectID that can be passed back and forth in the computation pipeline. An object is composed by a matadata, and a set of blobs.

Object in vineyard is by-design to be hierarchical, and can have other Object as members. For example, a tensor object may has a vector as its payload, and a distributed dataframe has many dataframe objects as its chunks, and a dataframe is composed by an aray of tensors as columns.

Subclassed by vineyard::Registered< Array< Entry > >, vineyard::Registered< Array< T > >, vineyard::Registered< Array< vid_t > >, vineyard::Registered< ArrowFragmentGroup >, vineyard::Registered< ArrowVertexMap< arrow::util::string_view, VID_T > >, vineyard::Registered< ArrowVertexMap< OID_T, VID_T > >, vineyard::Registered< BaseBinaryArray< ArrayType > >, vineyard::Registered< BaseListArray< ArrayType > >, vineyard::Registered< Blob >, vineyard::Registered< BooleanArray >, vineyard::Registered< ByteStream >, vineyard::Registered< DataFrame >, vineyard::Registered< DataframeStream >, vineyard::Registered< FixedSizeBinaryArray >, vineyard::Registered< GlobalDataFrame >, vineyard::Registered< GlobalTensor >, vineyard::Registered< Hashmap< K, V, std::hash< K >, std::equal_to< K > > >, vineyard::Registered< NullArray >, vineyard::Registered< NumericArray< T > >, vineyard::Registered< Pair >, vineyard::Registered< ParallelStream >, vineyard::Registered< RecordBatch >, vineyard::Registered< Scalar< T > >, vineyard::Registered< SchemaProxy >, vineyard::Registered< Table >, vineyard::Registered< Tuple >, vineyard::ArrowFragmentBase, vineyard::ITensor, vineyard::Registered< T >

Public Functions

virtual ~Object()
const ObjectID id() const

The object id of this object.

const ObjectMeta &meta() const

The metadata of this object.

const size_t nbytes() const

The nbytes of this object, can be treat as the memory usage of this object.

virtual void Construct(const ObjectMeta &meta)

Construct an object from metadata. The metadata meta should comes from client’s GetMetaData method.

The implementation of Construct method is usually boilerplate. Vineyard provides a code generator to help developers code there own data structures and can be shared via vineyard.

Parameters

meta – The metadata that be used to constrct the object.

inline virtual void PostConstruct(const ObjectMeta &meta)

PostConstruct is called at the end of Construct to perform user-specific constructions that the code generator cannot handle.

Parameters

meta – The metadata that be used to construct the object.

inline virtual Status Build(Client &client) final

Object is also a kind of ObjectBase, and can be used as a member to construct new objects. The Object type also has a Build method but it does nothing, though.

virtual std::shared_ptr<Object> _Seal(Client &client) final

Object is also a kind of ObjectBase, and can be used as a member to construct new objects. The Object type also has a _Seal method but it does nothing, though.

Status Persist(ClientBase &client) const

Persist the object to make it visible for clients that connected to other vineyardd instances in the cluster.

Parameters

client – The client that to be used to perform the Persist request.

const bool IsLocal() const

The verb “local” means it is a local object to the client. A local object’s blob can be accessed by the client in a zero-copy fashion. Otherwise only the metadata is accessible.

Returns

True iff the object is a local object.

const bool IsPersist() const

The verb “persist” means it is visible for client that connected to other vineyardd instances in the cluster. After Persist the object becomes persistent.

Returns

True iff the object is a persistent object.

const bool IsGlobal() const

The verb “global” means it is a global object and only refers some local objects.

Returns

True iff the object is a global object.

Protected Functions

inline Object()

Protected Attributes

ObjectID id_
mutable ObjectMeta meta_

Friends

friend class ClientBase
friend class Client
friend class RPCClient
friend class ObjectMeta
class vineyard::ObjectBuilder : public vineyard::ObjectBase

Subclassed by vineyard::ArrowFragmentGroupBuilder, vineyard::ArrowVertexMapBuilder< arrow::util::string_view, VID_T >, vineyard::BlobWriter, vineyard::BooleanArrayBaseBuilder, vineyard::ByteStreamBaseBuilder, vineyard::DataFrameBaseBuilder, vineyard::DataframeStreamBaseBuilder, vineyard::FixedSizeBinaryArrayBaseBuilder, vineyard::GlobalDataFrameBaseBuilder, vineyard::GlobalTensorBaseBuilder, vineyard::NullArrayBaseBuilder, vineyard::PairBaseBuilder, vineyard::ParallelStreamBuilder, vineyard::RecordBatchBaseBuilder, vineyard::SchemaProxyBaseBuilder, vineyard::TableBaseBuilder, vineyard::TupleBaseBuilder

Public Functions

inline virtual ~ObjectBuilder()
virtual Status Build(Client &client) override = 0

Building the object means construct all blobs of this object to vineyard server.

Parameters

client – The vineyard client that been used to create blobs in the connected vineyard server.

virtual std::shared_ptr<Object> Seal(Client &client)
virtual std::shared_ptr<Object> _Seal(Client &client) override = 0

Sealing the object means construct the metadata for the object and create metadata in vineyard server.

Parameters

client – The vineyard client that been used to create metadata in the connected vineyard server.

inline bool sealed() const

Protected Functions

inline void set_sealed(bool const sealed = true)
class vineyard::ObjectBase

ObjectBase is the most base class for vineyard’s Object and ObjectBuilder.

An ObjectBase instance is a builable value that it’s Build method put blobs into vineyard server, and it’s _Seal method is responsible for creating metadata in vineyard server for the object.

Subclassed by vineyard::Object, vineyard::ObjectBuilder

Public Functions

virtual Status Build(Client &client) = 0

Building the object means construct all blobs of this object to vineyard server.

Parameters

client – The vineyard client that been used to create blobs in the connected vineyard server.

virtual std::shared_ptr<Object> _Seal(Client &client) = 0

Sealing the object means construct the metadata for the object and create metadata in vineyard server.

Parameters

client – The vineyard client that been used to create metadata in the connected vineyard server.

Metadata of objects

class vineyard::ObjectMeta

ObjectMeta is the type for metadata of an Object. The ObjectMeta can be treat as a dict-like type. If the the metadata if the metadata obtained from vineyard, the metadata is readonly. Otherwise key-value attributes or object members could be associated with the metadata to construct a new vineyard object.

Public Types

using const_iterator = nlohmann::detail::iteration_proxy_value<json::const_iterator>

Public Functions

ObjectMeta()
inline ~ObjectMeta()
void SetClient(ClientBase *client)

Associate the client with the metadata.

ClientBase *GetClient() const

Get the associate client with the metadata.

void SetId(const ObjectID &id)

Set the object ID for the metadata.

const ObjectID GetId() const

Get the corresponding object ID of the metadata.

const Signature GetSignature() const

Get the corresponding object signature of the metadata.

void ResetSignature()

Reset the signatures in the metadata (for duplicating objects).

void SetGlobal(bool global = true)

Mark the vineyard object as a global object.

const bool IsGlobal() const

Check whether the vineyard object is a global object.

void SetTypeName(const std::string &type_name)

Set the typename of the metadata. The typename will be used to resolve in the ObjectFactory to create new object instances when get objects from vineyard server.

std::string const &GetTypeName() const

Get the typename of the metadata.

void SetNBytes(const size_t nbytes)

Set the nbytes attribute for the metadata, basically it indicates the memory usage of the object.

const size_t GetNBytes() const

Get the nbytes attribute of the object. Note that the nbytes attribute doesn’t always reflect the TOTAL monopolistic space usage of the bulk store, since two objects may share some blobs.

const InstanceID GetInstanceId() const

Get the instance ID of vineyard server where the metadata is created on.

const bool IsLocal() const

Whether the object meta is the metadata of a local object.

const bool Haskey(std::string const &key) const

Whether specific key exists in this metadata.

void ResetKey(std::string const &key)

Reset the given key in the metadata.

void AddKeyValue(const std::string &key, const std::string &value)

Add a string value entry to the metadata.

Parameters
  • key – The name of metadata entry.

  • value – The value of the metadata entry.

template<typename T>
inline void AddKeyValue(const std::string &key, T const &value)

Add a generic value entry to the metadata.

Parameters
  • T – The type of metadata’s value.

  • key – The name of metadata entry.

  • value – The value of the metadata entry.

template<typename T>
inline void AddKeyValue(const std::string &key, std::set<T> const &values)

Add a generic set value entry to the metadata.

Parameters
  • T – The type of metadata’s value.

  • key – The name of metadata entry, it will be first convert to JSON array by nlohmann::json.

  • value – The value of the metadata entry.

template<typename T>
inline void AddKeyValue(const std::string &key, std::vector<T> const &values)

Add a generic vector value entry to the metadata.

Parameters
  • T – The type of metadata’s value.

  • key – The name of metadata entry, it will be first convert to JSON array by nlohmann::json.

  • value – The value of the metadata entry.

template<typename Value>
inline void AddKeyValue(const std::string &key, std::map<std::string, Value> const &values)

Add a associated map value entry to the metadata.

Parameters
  • Value – The type of metadata’s value.

  • key – The name of metadata entry, it will be first convert to string.

  • value – The value of the metadata entry.

template<typename Value>
inline void AddKeyValue(const std::string &key, std::map<json, Value> const &values)

Add a associated map value entry to the metadata.

Parameters
  • Value – The type of metadata’s value.

  • key – The name of metadata entry, it will be first convert to string.

  • value – The value of the metadata entry.

template<typename Value>
inline void AddKeyValue(const std::string &key, std::unordered_map<std::string, Value> const &values)

Add a associated map value entry to the metadata.

Parameters
  • Value – The type of metadata’s value.

  • key – The name of metadata entry, it will be first convert to string.

  • value – The value of the metadata entry.

template<typename Value>
inline void AddKeyValue(const std::string &key, std::unordered_map<json, Value> const &values)

Add a associated map value entry to the metadata.

Parameters
  • Value – The type of metadata’s value.

  • key – The name of metadata entry, it will be first convert to string.

  • value – The value of the metadata entry.

void AddKeyValue(const std::string &key, json const &values)

Add a nlohmann::json value entry to the metadata.

Parameters
  • T – The type of metadata’s value.

  • key – The name of metadata entry, it will be first convert to string by nlohmann::json.

  • value – The value of the metadata entry.

inline const std::string GetKeyValue(const std::string &key) const

Get string metadata value.

Parameters

key – The key of metadata.

template<typename T>
inline const T GetKeyValue(const std::string &key) const

Get generic metadata value.

Parameters
  • T – The type of metadata value.

  • key – The key of metadata.

template<typename T>
inline void GetKeyValue(const std::string &key, T &value) const

Get generic metadata value, with automatically type deduction.

Parameters
  • T – The type of metadata value.

  • key – The key of metadata.

  • value – The result will be stored in value, the generic result is pass by reference to help type deduction.

template<typename T>
inline void GetKeyValue(const std::string &key, std::set<T> &values) const

Get generic set metadata value, with automatically type deduction.

Parameters
  • T – The type of metadata value.

  • key – The key of metadata.

  • value – The result will be stored in value, the generic result is pass by reference to help type deduction.

template<typename T>
inline void GetKeyValue(const std::string &key, std::vector<T> &values) const

Get generic vector metadata value, with automatically type deduction.

Parameters
  • T – The type of metadata value.

  • key – The key of metadata.

  • value – The result will be stored in value, the generic result is pass by reference to help type deduction.

template<typename Value>
inline void GetKeyValue(const std::string &key, std::map<std::string, Value> &values) const

Get associated map metadata value, with automatically type deduction.

Parameters
  • Value – The type of metadata value.

  • key – The key of metadata.

  • value – The result will be stored in value, the generic result is pass by reference to help type deduction.

template<typename Value>
inline void GetKeyValue(const std::string &key, std::map<json, Value> &values) const

Get associated map metadata value, with automatically type deduction.

Parameters
  • Value – The type of metadata value.

  • key – The key of metadata.

  • value – The result will be stored in value, the generic result is pass by reference to help type deduction.

template<typename Value>
inline void GetKeyValue(const std::string &key, std::unordered_map<std::string, Value> &values) const

Get associated map metadata value, with automatically type deduction.

Parameters
  • Value – The type of metadata value.

  • key – The key of metadata.

  • value – The result will be stored in value, the generic result is pass by reference to help type deduction.

template<typename Value>
inline void GetKeyValue(const std::string &key, std::unordered_map<json, Value> &values) const

Get associated map metadata value, with automatically type deduction.

Parameters
  • Value – The type of metadata value.

  • key – The key of metadata.

  • value – The result will be stored in value, the generic result is pass by reference to help type deduction.

void GetKeyValue(const std::string &key, json &value) const

Get json metadata value.

Parameters
  • key – The key of metadata.

  • value – The result will be stored in value.

void AddMember(const std::string &name, const ObjectMeta &member)

Add member to ObjectMeta.

Parameters
  • name – The name of member object.

  • member – The metadata of member object to be added.

void AddMember(const std::string &name, const Object &member)

Add member to ObjectMeta.

Parameters
  • name – The name of member object.

  • member – The member object to be added.

void AddMember(const std::string &name, const Object *member)

Add member to ObjectMeta.

Parameters
  • name – The name of member object.

  • member – The member object to be added.

void AddMember(const std::string &name, const std::shared_ptr<Object> &member)

Add member to ObjectMeta.

Parameters
  • name – The name of member object.

  • member – The member object to be added.

void AddMember(const std::string &name, const ObjectID member_id)

Add member to ObjectMeta.

Parameters
  • name – The name of member object.

  • member – The object ID of member object to be added.

std::shared_ptr<Object> GetMember(const std::string &name) const

Get member value from vineyard.

Parameters

name – The name of member object.

Returns

member The member object.

ObjectMeta GetMemberMeta(const std::string &name) const

Get member’s ObjectMeta value.

Parameters
  • name – The name of member object.

  • member – The metadata of member object. will be stored in value.

Status GetBuffer(const ObjectID blob_id, std::shared_ptr<arrow::Buffer> &buffer) const

Get buffer member (directed or indirected) from the metadata. The metadata should has already been initialized.

void SetBuffer(const ObjectID &id, const std::shared_ptr<arrow::Buffer> &buffer)
void Reset()

Reset the metadata as an initialized one.

void PrintMeta() const
const bool incomplete() const
const json &MetaData() const
json &MutMetaData()
void SetMetaData(ClientBase *client, const json &meta)
inline const_iterator begin() const
inline const_iterator end() const
const std::shared_ptr<BufferSet> &GetBufferSet() const
template<>
const json GetKeyValue(const std::string &key) const
template<>
const json GetKeyValue(const std::string &key) const

Connect to vineyard

class vineyard::ClientBase

ClientBase is the base class for vineyard IPC and RPC client.

ClientBase implements common communication stuffs, and leave the IPC and RPC specific functionalities to Client and RPCClient.

Vineyard’s Client and RPCClient is non-copyable.

Subclassed by vineyard::Client, vineyard::RPCClient

Public Functions

ClientBase()
inline virtual ~ClientBase()
ClientBase(const ClientBase&) = delete
ClientBase(ClientBase&&) = delete
ClientBase &operator=(const ClientBase&) = delete
ClientBase &operator=(ClientBase&&) = delete
Status GetData(const ObjectID id, json &tree, const bool sync_remote = false, const bool wait = false)

Get object metadata from vineyard using given object ID.

Parameters
  • id – The ID of the requested object.

  • tree – The returned metadata tree of the requested object.

  • sync_remote – Whether to trigger an immediate remote metadata synchronization before get specific metadata. Default is false.

  • wait – The request could be blocked util the object with given id has been created on vineyard by other clients. Default is false.

Returns

Status that indicates whether the get action succeeds.

Status GetData(const std::vector<ObjectID> &ids, std::vector<json> &trees, const bool sync_remote = false, const bool wait = false)

Get multiple object metadatas from vineyard using given object IDs.

Parameters
  • ids – The IDs of the requested objects

  • trees – The returned metadata trees of the requested objects

  • sync_remote – Whether to trigger an immediate remote metadata synchronization before get specific metadata. Default is false.

  • wait – The request could be blocked util the object with given id has been created on vineyard by other clients. Default is false.

Returns

Status that indicates whether the get action has succeeded.

Status CreateData(const json &tree, ObjectID &id, Signature &signature, InstanceID &instance_id)

Create the metadata in the vineyard server.

Parameters
  • tree – The metadata that will be created in vineyard.

  • id – The returned object ID of the created data.

  • instance_id – The vineyard instance ID where this object is created. at.

Returns

Status that indicates whether the create action has succeeded.

Status CreateMetaData(ObjectMeta &meta_data, ObjectID &id)

Create the metadata in the vineyard server, after created, the resulted object id in the meta_data will be filled.

Parameters
  • meta_data – The metadata that will be created in vineyard.

  • id – The returned object ID of the created metadata.

Returns

Status that indicates whether the create action has succeeded.

virtual Status GetMetaData(const ObjectID id, ObjectMeta &meta_data, const bool sync_remote = false) = 0

Get the meta-data of the requested object.

Parameters
  • id – The ID of the requested object

  • meta_data – The returned metadata of the requested object

  • sync_remote – Whether trigger remote sync

Returns

Status that indicates whether the get action has succeeded.

Status SyncMetaData()

Sync remote metadata from etcd to the connected vineyardd.

Returns

Status that indicates whether the sync action has succeeded.

Status DelData(const ObjectID id, const bool force = false, const bool deep = true)

Delete metadata in vineyard. When the object is a used by other object, it will be deleted only when the force parameter is specified.

Parameters
  • id – The ID to delete.

  • force – Whether to delete the object forcely. Forcely delete an object means the object and objects which use this object will be delete. Default is false.

  • deep – Whether to delete the member of this object. Default is false. Note that when deleting object which has direct blob members, the processing on those blobs yields a “deep” behavior.

Returns

Status that indicates whether the delete action has succeeded.

Status DelData(const std::vector<ObjectID> &ids, const bool force = false, const bool deep = true)

Delete multiple metadatas in vineyard.

Parameters
  • ids – The IDs to delete.

  • force – Whether to delete the object forcely. Forcely delete an object means the object and objects which use this object will be delete. Default is false.

  • deep – Whether to delete the member of this object. Default is false. Note that when deleting objects which have direct blob members, the processing on those blobs yields a “deep” behavior.

Returns

Status that indicates whether the delete action has succeeded.

Status ListData(std::string const &pattern, bool const regex, size_t const limit, std::unordered_map<ObjectID, json> &meta_trees)

List objectmetas in vineyard, using the given typename patterns.

Parameters
  • pattern – The pattern string that will be used to matched against objects’ typename.

  • regex – Whether the pattern is a regular expression pattern. Default is false. When regex is false, the pattern will be treated as a glob pattern.

  • limit – The number limit for how many objects will be returned at most.

  • meta_trees – An map that contains the returned object metadatas.

Returns

Status that indicates whether the list action has succeeded.

Status Persist(const ObjectID id)

Persist the given object to etcd to make it visible to clients that been connected to vineyard servers in the cluster.

Parameters

id – The object id of object that will be persisted.

Returns

Status that indicates whether the persist action has succeeded.

Status IfPersist(const ObjectID id, bool &persist)

Check if the given object has been persist to etcd.

Parameters
  • id – The object id to check.

  • persist – The result variable will be stored in persist as return value. The value true means the object is visible by other vineyard servers.

Returns

Status that indicates whether the check has succeeded.

Status Exists(const ObjectID id, bool &exists)

Check if the given object exists in vineyard server.

Parameters
  • id – The object id to check.

  • exists – The result variable will be stored in exists as return value. The value true means the object exists.

Returns

Status that indicates whether the check has succeeded.

Status ShallowCopy(const ObjectID id, ObjectID &target_id)

Make a shallow copy on the given object. A “shallow copy” means the result object has the same type with the source object and they shares all member objects.

Parameters
  • id – The object id to shallow copy.

  • target_id – The result object id will be stored in target_id as return value.

Returns

Status that indicates whether the shallow copy has succeeded.

Status PutName(const ObjectID id, std::string const &name)

Vineyard support associating a user-specific name with an object. PutName registers a name entry in vineyard server. An object can be assoiciated with more than one names.

Parameters
  • id – The ID of the object.

  • name – The user-specific name that will be associated with the given object.

Returns

Status that indicates whether the request has succeeded.

Status GetName(const std::string &name, ObjectID &id, const bool wait = false)

Retrieve the object ID by assoicated name.

Parameters
  • name – The name of the requested object.

  • id – The returned object ID.

  • wait – If wait is specified, the request will be blocked util the given name has been registered on vineyard by other clients.

Returns

Status that indicates whether the query has succeeded.

Status DropName(const std::string &name)

Deregister a name entry. The assoicated object will be kept and won’t be deleted.

Parameters

name – The name that will be deregistered.

Returns

Status that indicates whether the query has succeeded.

Status MigrateObject(const ObjectID object_id, ObjectID &result_id, bool is_stream = false)

Migrate remote object to local.

Parameters
  • object_id – The existing object that will be migrated to current vineyardd.

  • result_id – Record the result object id.

  • is_stream – Indicates whether the migrated object is a stream

Returns

Status that indicates if the migration success.

Status MigrateStream(const ObjectID object_id, ObjectID &result_id)

Migrate remote stream to local.

Parameters
  • object_id – The existing stream that will be migrated to current vineyardd.

  • result_id – Record the result stream id.

Returns

Status that indicates if the migration success.

bool Connected() const

Check if the client still connects to the vineyard server.

Returns

True when the connection is still alive, otherwise false.

void Disconnect()

Disconnect this client.

inline std::string const &IPCSocket()

Get the UNIX domain socket location of the connected vineyardd server.

Returns

Location of the IPC socket.

inline std::string const &RPCEndpoint()

The RPC endpoint of the connected vineyardd server.

Returns

The RPC endpoint.

inline const InstanceID instance_id() const

Get the instance id of the connected vineyard server.

Note that for RPC client the instance id is not available.

Returns

The vineyard server’s instance id.

Status ClusterInfo(std::map<InstanceID, json> &meta)

Retrieve the cluster information of the connected vineyard server.

The cluster information for every instance mainly includes the host address (i.e., ip address).

Returns

Status that indicates whether the query has succeeded.

Status InstanceStatus(std::shared_ptr<struct InstanceStatus> &status)

Return the status of connected vineyard instance.

If success, the status parameter will be reseted as an instance of InstanceStatus.

Parameters

status – The result instance status.

Returns

Status that indicates whether the query has succeeded.

Status Instances(std::vector<InstanceID> &instances)

List all instances in the connected vineyard cluster.

Parameters

A – list of instance IDs will be stored in instances.

Returns

Status that indicates whether the query has succeeded.

inline const std::string &Version() const

Get the version of connected vineyard server.

Returns

Return a version string MAJOR.MINOR.PATCH that follows the semver convention.

Status Debug(const json &debug, json &tree)

Issue a debug request.

Parameters
  • debug – The payload that will be sent to the debug handler.

  • result – The result that returned by the debug handler.

Returns

Status that indicates whether the debug action succeeds.

Protected Functions

Status doWrite(const std::string &message_out)
Status doRead(std::string &message_in)
Status doRead(json &root)
Status migrateObjectImpl(const ObjectID object_id, ObjectID &result_id, bool const local, bool const is_stream, std::string const &peer, std::string const &peer_rpc_endpoint)

Implementation for migrate remote object to local.

Returns

Status that indicates if the migration success.

Protected Attributes

mutable bool connected_
std::string ipc_socket_
std::string rpc_endpoint_
int vineyard_conn_
InstanceID instance_id_
std::string server_version_
std::recursive_mutex client_mutex_
class vineyard::Client : public vineyard::ClientBase

Vineyard’s IPC Client connects to to UNIX domain socket of the vineyard server. Vineyard’s IPC Client talks to vineyard server and manipulate objects in vineyard.

Public Functions

~Client() override
Status Connect()

Connect to vineyard using the UNIX domain socket file specified by the environment variable VINEYARD_IPC_SOCKET.

Returns

Status that indicates whether the connect has succeeded.

Status Connect(const std::string &ipc_socket)

Connect to vineyardd using the given UNIX domain socket ipc_socket.

Parameters

ipc_socket – Location of the UNIX domain socket.

Returns

Status that indicates whether the connect has succeeded.

Status Fork(Client &client)

Create a new client using self UNIX domain socket.

virtual Status GetMetaData(const ObjectID id, ObjectMeta &meta_data, const bool sync_remote = false) override

Obtain metadata from vineyard server.

Parameters
  • id – The object id to get.

  • meta_data – The result metadata will be store in meta_data as return value.

  • sync_remote – Whether to trigger an immediate remote metadata synchronization before get specific metadata. Default is false.

Returns

Status that indicates whether the get action has succeeded.

Status GetMetaData(const std::vector<ObjectID> &id, std::vector<ObjectMeta>&, const bool sync_remote = false)

Obtain multiple metadatas from vineyard server.

Parameters
  • ids – The object ids to get.

  • meta_data – The result metadata will be store in meta_data as return value.

  • sync_remote – Whether to trigger an immediate remote metadata synchronization before get specific metadata. Default is false.

Returns

Status that indicates whether the get action has succeeded.

Status CreateBlob(size_t size, std::unique_ptr<BlobWriter> &blob)

Create a blob in vineyard server. When creating a blob, vineyard server’s bulk allocator will prepare a block of memory of the requested size, the map the memory to client’s process to share the allocated memory.

Parameters
  • size – The size of requested blob.

  • blob – The result mutable blob will be set in blob.

Returns

Status that indicates whether the create action has succeeded.

Status GetBlob(const ObjectID id, Payload &object)

Get a blob from vineyard server. When obtaining blobs from vineyard server, the memory address in the server process will be mmapped to the client’s process to share the memory.

Parameters
  • idObject id for the blob to get.

  • blob – The result immutable blob will be set in blob. Note that blob is special, since it can be get as immutable object before sealing.

Returns

Status that indicates whether the create action has succeeded.

Status GetBlobs(const std::vector<ObjectID> &ids, std::vector<std::shared_ptr<Blob>> &blobs)

Get a set of blobs from vineyard server. See also GetBlob.

Parameters
  • idsObject ids for the blobs to get.

  • blobs – The result immutable blobs will be added to blobs.

Returns

Status that indicates whether the create action has succeeded.

Status CreateStream(const ObjectID &id)

Allocate a stream on vineyard. The metadata of parameter id must has already been created on vineyard.

Parameters

id – The id of metadata that will be used to create stream.

Returns

Status that indicates whether the create action has succeeded.

Status OpenStream(const ObjectID &id, OpenStreamMode mode)

open a stream on vineyard. Failed if the stream is already opened on the given mode.

Parameters
  • id – The id of stream to mark.

  • mode – The mode, OpenStreamMode::read or OpenStreamMode::write.

Returns

Status that indicates whether the open action has succeeded.

Status GetNextStreamChunk(ObjectID const id, size_t const size, std::unique_ptr<arrow::MutableBuffer> &blob)

Allocate a chunk of given size in vineyard for a stream. When the request cannot be statisfied immediately, e.g., vineyard doesn’t have enough memory or the specified has accumulated too many chunks, the request will be blocked until the request been processed.

Parameters
  • id – The id of the stream.

  • size – The size of the chunk to allocate.

  • blob – The allocated mutable buffer will be set in blob.

Returns

Status that indicates whether the allocation has succeeded.

Status PullNextStreamChunk(ObjectID const id, std::unique_ptr<arrow::Buffer> &blob)

Poll a chunk from a stream. When there’s no more chunk available in the stream, i.e., the stream has been stoped, a status code kStreamDrained or kStreamFinish will be returned, otherwise the reader will be blocked until writer creates a new chunk in the stream.

Parameters
  • id – The id of the stream.

  • blob – The immutable chunk generated by the writer of the stream.

Returns

Status that indicates whether the polling has succeeded.

Status StopStream(ObjectID const id, bool failed)

Stop a stream, mark it as finished or aborted.

Parameters
  • id – The id of the stream.

  • failed – Whether the stream is stoped at a successful state. True means the stream has been exited normally, otherwise false.

Returns

Status that indicates whether the request has succeeded.

std::shared_ptr<Object> GetObject(const ObjectID id)

Get an object from vineyard. The ObjectFactory will be used to resolve the constructor of the object.

Parameters

id – The object id to get.

Returns

A std::shared_ptr<Object> that can be safely cast to the underlying concrete object type. When the object doesn’t exists an std::runtime_error exception will be raised.

Status GetObject(const ObjectID id, std::shared_ptr<Object> &object)

Get an object from vineyard. The ObjectFactory will be used to resolve the constructor of the object.

Parameters
  • id – The object id to get.

  • object – The result object will be set in parameter object.

Returns

When errors occur during the request, this method won’t throw exceptions, rather, it results a status to represents the error.

template<typename T>
inline std::shared_ptr<T> GetObject(const ObjectID id)

Get an object from vineyard. The type parameter T will be used to resolve the constructor of the object.

Parameters

id – The object id to get.

Returns

A std::shared_ptr<Object> that can be safely cast to the underlying concrete object type. When the object doesn’t exists an std::runtime_error exception will be raised.

template<typename T>
inline Status GetObject(const ObjectID id, std::shared_ptr<T> &object)

Get an object from vineyard. The type parameter T will be used to resolve the constructor of the object.

This method can be used to get concrete object from vineyard without explicitly dynamic_cast, and the template type parameter can be deduced in many situations:

std::shared_ptr<Array<int>> int_array;
client.GetObject(id, int_array);

Parameters
  • id – The object id to get.

  • object – The result object will be set in parameter object.

Returns

When errors occur during the request, this method won’t throw exceptions, rather, it results a status to represents the error.

std::vector<std::shared_ptr<Object>> GetObjects(const std::vector<ObjectID> &ids)

Get multiple objects from vineayrd.

Parameters

ids – The object IDs to get.

Returns

A list of objects.

std::vector<std::shared_ptr<Object>> ListObjects(std::string const &pattern, const bool regex = false, size_t const limit = 5)

List objects in vineyard, using the given typename patterns.

Parameters
  • pattern – The pattern string that will be used to matched against objects’ typename.

  • regex – Whether the pattern is a regular expression pattern. Default is false. When regex is false, the pattern will be treated as a glob pattern.

  • limit – The number limit for how many objects will be returned at most.

Returns

A vector of objects that listed from vineyard server.

Status AllocatedSize(const ObjectID id, size_t &size)

Get the allocated size for the given object.

Status CreateArena(const size_t size, int &fd, size_t &available_size, uintptr_t &base, uintptr_t &space)
Status ReleaseArena(const int fd, std::vector<size_t> const &offsets, std::vector<size_t> const &sizes)

Public Static Functions

static Client &Default()

Get a default client reference, using the UNIX domain socket file specified by the environment variable VINEYARD_IPC_SOCKET.

Returns

A reference of the default Client instance.

Protected Functions

Status CreateBuffer(const size_t size, ObjectID &id, Payload &payload, std::shared_ptr<arrow::MutableBuffer> &buffer)
Status GetBuffer(const ObjectID id, std::shared_ptr<arrow::Buffer> &buffer)
Status GetBuffers(const std::set<ObjectID> &ids, std::map<ObjectID, std::shared_ptr<arrow::Buffer>> &buffers)
Status GetBufferSizes(const std::set<ObjectID> &ids, std::map<ObjectID, size_t> &sizes)
Status DropBuffer(const ObjectID id, const int fd)

An (unsafe) internal-usage method that drop the buffer, without checking the dependency. To achieve the “DeleteObject” semantic for blobs, use DelData instead.

Note that the target buffer could be both sealed and unsealed.

class vineyard::RPCClient : public vineyard::ClientBase

Public Functions

~RPCClient() override
Status Connect()

Connect to vineyard using the TCP endpoint specified by the environment variable VINEYARD_RPC_ENDPOINT.

Returns

Status that indicates whether the connect has succeeded.

Status Connect(const std::string &rpc_endpoint)

Connect to vineyardd using the given TCP endpoint rpc_endpoint.

Parameters

rpc_endpoint – The TPC endpoint of vineyard server, in the format of host:port.

Returns

Status that indicates whether the connect has succeeded.

Status Connect(const std::string &host, uint32_t port)

Connect to vineyardd using the given TCP host and port.

Parameters
  • host – The host of vineyard server.

  • port – The TCP port of vineayrd server’s RPC service.

Returns

Status that indicates whether the connect has succeeded.

Status Fork(RPCClient &client)

Create a new client using self endpoint.

virtual Status GetMetaData(const ObjectID id, ObjectMeta &meta_data, const bool sync_remote = false) override

Obtain metadata from vineyard server. Note that unlike IPC client, RPC client doesn’t map shared memorys to the client’s process.

Parameters
  • id – The object id to get.

  • meta_data – The result metadata will be store in meta_data as return value.

  • sync_remote – Whether to trigger an immediate remote metadata synchronization before get specific metadata. Default is false.

Returns

Status that indicates whether the get action has succeeded.

Status GetMetaData(const std::vector<ObjectID> &id, std::vector<ObjectMeta> &meta_data, const bool sync_remote = false)

Obtain multiple metadatas from vineyard server.

Parameters
  • ids – The object ids to get.

  • meta_data – The result metadata will be store in meta_data as return value.

  • sync_remote – Whether to trigger an immediate remote metadata synchronization before get specific metadata. Default is false.

Returns

Status that indicates whether the get action has succeeded.

std::shared_ptr<Object> GetObject(const ObjectID id)

Get an object from vineyard. The ObjectFactory will be used to resolve the constructor of the object.

In RPCClient, all blob fields in the result object is unaccessible, access those fields will trigger an std::runtime_error.

Parameters

id – The object id to get.

Returns

A std::shared_ptr<Object> that can be safely cast to the underlying concrete object type. When the object doesn’t exists an std::runtime_error exception will be raised.

Status GetObject(const ObjectID id, std::shared_ptr<Object> &object)

Get an object from vineyard. The ObjectFactory will be used to resolve the constructor of the object.

Parameters
  • id – The object id to get.

  • object – The result object will be set in parameter object.

Returns

When errors occur during the request, this method won’t throw exceptions, rather, it results a status to represents the error.

std::vector<std::shared_ptr<Object>> GetObjects(const std::vector<ObjectID> &ids)

Get multiple objects from vineayrd.

Parameters

ids – The object IDs to get.

Returns

A list of objects.

template<typename T>
inline std::shared_ptr<T> GetObject(const ObjectID id)

Get an object from vineyard. The type parameter T will be used to resolve the constructor of the object.

Parameters

id – The object id to get.

Returns

A std::shared_ptr<Object> that can be safely cast to the underlying concrete object type. When the object doesn’t exists an std::runtime_error exception will be raised.

template<typename T>
inline Status GetObject(const ObjectID id, std::shared_ptr<T> &object)

Get an object from vineyard. The type parameter T will be used to resolve the constructor of the object.

This method can be used to get concrete object from vineyard without explicitly dynamic_cast, and the template type parameter can be deduced in many situations:

std::shared_ptr<Array<int>> int_array;
client.GetObject(id, int_array);

Parameters
  • id – The object id to get.

  • object – The result object will be set in parameter object.

Returns

When errors occur during the request, this method won’t throw exceptions, rather, it results a status to represents the error.

std::vector<std::shared_ptr<Object>> ListObjects(std::string const &pattern, const bool regex = false, size_t const limit = 5)

List objects in vineyard, using the given typename patterns.

Parameters
  • pattern – The pattern string that will be used to matched against objects’ typename.

  • regex – Whether the pattern is a regular expression pattern. Default is false. When regex is false, the pattern will be treated as a glob pattern.

  • limit – The number limit for how many objects will be returned at most.

Returns

A vector of objects that listed from vineyard server.

inline const InstanceID remote_instance_id() const

Get the remote instance id of the connected vineyard server.

Note that for RPC client the instance id is not available, thus we have the “remote instance id” to indicate which server we are connecting to.

Returns

The vineyard server’s instance id.

State of server

struct vineyard::InstanceStatus

Public Functions

explicit InstanceStatus(const json &tree)

Initialize the status value using a json returned from the vineyard server.

Parameters

tree – JSON that returned from the vineyard server.

Public Members

const InstanceID instance_id

The connected instance id.

const std::string deployment

The deployment manner, can be local or distributed.

const size_t memory_usage

The current memory usage in vineyard server, in bytes.

const size_t memory_limit

The memory upper bound of this vineyard server, in bytes.

const size_t deferred_requests

How many requests are deferred in the queue.

const size_t ipc_connections

How many Client connects to this vineyard server.

const size_t rpc_connections

How many RPCClient connects to this vineyard server.

Primitives

class vineyard::Blob : public vineyard::Registered<Blob>

The unit to store data payload in vineyard. When the client gets a blob from vineyard, the vineyard server maps a chunk of memory from its memory space to the client space in a zero-copy fashion.

Public Functions

size_t size() const

Get the size of the blob, i.e., the number of bytes of the data payload in the blob.

Note that the size of blob is the “allocated size” of the blob, and may (usually) not be the same value of the requested size.

Returns

The (allocated) size of the blob.

size_t allocated_size() const

Get the allocated size of the blob, i.e., the number of bytes of the data payload in the blob.

Returns

The allocated size of the blob.

const char *data() const

Get the const data pointer of the data payload in the blob.

Returns

The const data pointer.

const std::shared_ptr<arrow::Buffer> &Buffer() const

Get the arrow buffer of the blob.

Returns

The arrow buffer which holds the data payload of the blob.

virtual void Construct(ObjectMeta const &meta) override

Construct the blob locally for the given object meta.

Parameters

meta – The given object meta.

void Dump() const

Dump the buffer for debugging.

Public Static Functions

static inline std::shared_ptr<Object> Create()
static std::shared_ptr<Blob> MakeEmpty(Client &client)

Create an empty blob in the vineyard server.

Parameters

client – The client connected to the vineyard server.

static std::shared_ptr<Blob> FromBuffer(Client &client, const ObjectID object_id, const size_t size, const uintptr_t pointer)

Create the blob from a buffer from allocator.

Parameters
  • object_id – The object ID of this blob. allocator.

  • pointer – The address of buffer in the client allocator.

  • size – The estimated size of the buffer.

class vineyard::BlobWriter : public vineyard::ObjectBuilder

The writer to write a blob in vineyard. The writer is initialized in the client with a local buffer and its size, and a blob in vineyard will be created when Build is invoked.

Public Functions

ObjectID id() const

Return the object id of this blob builder. Note that before sealing the blob builder the object id cannot be used to get “Blob” objects.

Returns

The ObjectID of the blob writer.

size_t size() const

Get the size of the blob, i.e., the number of bytes of the data payload in the blob.

Returns

The size of the blob.

char *data()

Get the data pointer of the data payload in the blob.

Returns

The data pointer.

const char *data() const

Get the const data pointer of the data payload in the blob.

Returns

The const data pointer.

const std::shared_ptr<arrow::MutableBuffer> &Buffer() const

Get the mutable buffer of the blob.

Returns

The mutable buffer of the blob, which can be modified to update the content in the blob.

virtual Status Build(Client &client) override

Build a blob in vineyard server.

Parameters

client – The client connected to the vineyard server.

Status Abort(Client &client)

Abort the blob builder.

Parameters

client – Release the blob builder object if it is not sealed.

void AddKeyValue(std::string const &key, std::string const &value)

Add key-value metadata for the blob.

Parameters
  • key – The key of the metadata.

  • value – The value of the metadata.

void AddKeyValue(std::string const &key, std::string &&value)

Add key-value metadata for the blob.

Parameters
  • key – The key of the metadata.

  • value – The value of the metadata.

void Dump() const

Dump the buffer for debugging.

Stream

class vineyard::ByteStream : public vineyard::Registered<ByteStream>

The basic stream with each chunk representing a segment of bytes.

Public Functions

inline virtual void Construct(const ObjectMeta &meta) override

Construct an object from metadata. The metadata meta should comes from client’s GetMetaData method.

The implementation of Construct method is usually boilerplate. Vineyard provides a code generator to help developers code there own data structures and can be shared via vineyard.

Parameters

meta – The metadata that be used to constrct the object.

inline Status OpenReader(Client &client, std::unique_ptr<ByteStreamReader> &reader)

Open a reader to consume data from the byte stream.

Parameters
  • client – The client connected to the vineyard server

  • The – unique pointer to the reader

inline Status OpenWriter(Client &client, std::unique_ptr<ByteStreamWriter> &writer)

Open a writer to produce data to the byte stream.

Parameters
  • client – The client connected to the vineyard server

  • The – unique pointer to the writer

inline std::unordered_map<std::string, std::string> GetParams()

Public Static Functions

static inline std::shared_ptr<Object> Create()

Data types

template<typename T>
class vineyard::Array : public vineyard::Registered<Array<T>>

The array type in vineyard.

tparam T

The type for the elements.

Public Functions

inline virtual void Construct(const ObjectMeta &meta) override

Construct an object from metadata. The metadata meta should comes from client’s GetMetaData method.

The implementation of Construct method is usually boilerplate. Vineyard provides a code generator to help developers code there own data structures and can be shared via vineyard.

Parameters

meta – The metadata that be used to constrct the object.

inline const T &operator[](size_t loc) const

Get the element at the given location.

Parameters

loc – The given location to get the element.

inline size_t size() const

Get the size of the array.

Returns

The size.

inline const T *data() const

Get the pointer to the beginning of the data buffer.

Parameters

The – pointer to the data buffer

Public Static Functions

static inline std::shared_ptr<Object> Create()
template<typename T>
class vineyard::ArrayBuilder : public vineyard::ArrayBaseBuilder<T>

ArrayBuilder is designed for constructing arrays that supported by vineyard.

tparam T

The type for the elements.

Public Functions

inline ArrayBuilder(Client &client, size_t size)
inline ArrayBuilder(Client &client, std::vector<T> const &vec)

Initialize the ArrayBuilder from an existing std::vector of type T.

Parameters
  • client – The client connected to the vineyard server.

  • vec – The local std::vector of type T.

inline ArrayBuilder(Client &client, const T *data, size_t size)

Initialize the ArrayBuilder from an existing C array of type T.

Parameters
  • client – The client connected to the vineyard server.

  • data – The pointer to the array.

  • size – The size of the array.

inline const size_t size() const

Get the size of the array, i.e., number of elements in the array.

Returns

The size.

inline T &operator[](size_t idx)

Get the element located in the given index of the array.

Parameters

idx – The give index.

Returns

The element at the given index.

inline T *data() noexcept

Get the data pointer to the array.

Returns

The data pointer.

inline const T *data() const noexcept

Get the const data pointer to the array.

Returns

The const data pointer.

inline Status Build(Client &client) override

Build the array object.

Parameters

client – The client connected to the vineyard server.

template<typename K, typename V, typename H = std::hash<K>, typename E = std::equal_to<K>>
class vineyard::Hashmap : public vineyard::Registered<Hashmap<K, V, std::hash<K>, std::equal_to<K>>>, public std::hash<K>, public std::equal_to<K>

The hash map in vineyard.

tparam K

The type for the key.

tparam V

The type for the value.

tparam std::hash<K>

The hash function for the key.

tparam std::equal_to<K>

The compare function for the key.

Public Types

using T = std::pair<K, V>
using Entry = ska::detailv3::sherwood_v3_entry<T>
using EntryPointer = const Entry*
using Hasher = ska::detailv3::KeyOrValueHasher<K, std::pair<K, V>, H>
using Equal = ska::detailv3::KeyOrValueEquality<K, std::pair<K, V>, E>
using value_type = T
using size_type = size_t
using difference_type = std::ptrdiff_t
using hasher = H
using key_equal = E
using reference = value_type&
using const_reference = const value_type&
using pointer = value_type*
using const_pointer = value_type*
using flat_hash_table_type = ska::detailv3::sherwood_v3_table<T, K, H, Hasher, E, Equal, std::allocator<T>, typename std::allocator_traits<std::allocator<T>>::template rebind_alloc<ska::detailv3::sherwood_v3_entry<T>>>

Public Functions

inline virtual void Construct(const ObjectMeta &meta) override

Construct an object from metadata. The metadata meta should comes from client’s GetMetaData method.

The implementation of Construct method is usually boilerplate. Vineyard provides a code generator to help developers code there own data structures and can be shared via vineyard.

Parameters

meta – The metadata that be used to constrct the object.

inline virtual void PostConstruct(const ObjectMeta &meta) override

Set the hash policy after the construction of the HashMap.

inline iterator begin() const

The beginning iterator.

inline iterator end() const

The ending iterator.

inline iterator find(const K &key)

Find the iterator by key.

inline const iterator find(const K &key) const

Return the const iterator by key.

inline size_t count(const K &key) const

Return the number of occurancies of the key.

inline size_t size() const

Return the size of the HashMap, i.e., the number of elements stored in the HashMap.

inline bool empty() const

Check whether the HashMap is empty.

inline const V &at(const K &key) const

Get the value by key. Here the existance of the key is checked.

Public Static Functions

static inline std::shared_ptr<Object> Create()
struct iterator

The iterator to iterate key-value mappings in the HashMap.

Public Functions

iterator() = default
inline explicit iterator(EntryPointer current)
inline iterator &operator++()
inline iterator operator++(int)
inline const value_type &operator*() const
inline const value_type *operator->() const

Public Members

EntryPointer current = EntryPointer()

Friends

inline friend friend bool operator== (const iterator &lhs, const iterator &rhs)
inline friend friend bool operator!= (const iterator &lhs, const iterator &rhs)
template<typename K, typename V, typename H = std::hash<K>, typename E = std::equal_to<K>>
class vineyard::HashmapBuilder : public vineyard::HashmapBaseBuilder<K, V, std::hash<K>, std::equal_to<K>>

HashmapBuilder is used for constructing hashmaps that supported by vineyard.

tparam K

The type for the key.

tparam V

The type for the value.

tparam std::hash<K>

The hash function for the key.

tparam std::equal_to<K>

The compare function for the key.

Public Functions

inline explicit HashmapBuilder(Client &client)
inline explicit HashmapBuilder(Client &client, ska::flat_hash_map<K, V> &&hashmap)
inline V &operator[](const K &key)

Get the mapping value of the given key.

inline V &operator[](K &&key)

Get the mapping value of the given key.

template<class ...Args>
inline void emplace(Args&&... args)

Emplace key-value pair into the hashmap.

inline V &at(const K &key)

Get the mapping value of the given key.

inline const V &at(const K &key) const

Get the const mapping value of the given key.

inline size_t size() const

Get the size of the hashmap.

inline void reserve(size_t size)

Reserve the size for the hashmap.

inline bool empty() const

Check whether the hashmap is empty.

inline ska::flat_hash_map<K, V, H, E>::iterator begin()

Return the beginning iterator.

inline ska::flat_hash_map<K, V, H, E>::const_iterator begin() const

Return the const beginning iterator.

inline ska::flat_hash_map<K, V, H, E>::const_iterator cbegin() const

Return the const beginning iterator.

inline ska::flat_hash_map<K, V, H, E>::iterator end()

Return the ending iterator.

inline ska::flat_hash_map<K, V, H, E>::const_iterator end() const

Return the const ending iterator.

inline ska::flat_hash_map<K, V, H, E>::const_iterator cend() const

Return the const ending iterator.

inline ska::flat_hash_map<K, V, H, E>::iterator find(const K &key)

Find the value by key.

inline Status Build(Client &client) override

Build the hashmap object.

template<typename T>
class vineyard::Tensor : public vineyard::ITensor, public vineyard::BareRegistered<Tensor<T>>

Public Functions

inline virtual void Construct(const ObjectMeta &meta) override

Construct an object from metadata. The metadata meta should comes from client’s GetMetaData method.

The implementation of Construct method is usually boilerplate. Vineyard provides a code generator to help developers code there own data structures and can be shared via vineyard.

Parameters

meta – The metadata that be used to constrct the object.

inline std::vector<int64_t> strides() const

Get the strides of the tensor.

Returns

The strides of the tensor. The definition of the tensor’s strides can be found in https://pytorch.org/docs/stable/tensor_attributes.html

inline virtual std::vector<int64_t> const &shape() const override

Get the shape of the tensor.

Returns

The shape vector where the ith element represents the size of the ith axis.

inline virtual std::vector<int64_t> const &partition_index() const override

Get the index of this partition in the global tensor.

Returns

The index vector where the ith element represents the index in the ith axis.

inline virtual AnyType value_type() const override

Get the type of tensor’s elements.

Returns

The type of the tensor’s elements.

inline const T *data() const

Get the data pointer to the tensor’s data buffer.

Returns

The data pointer.

inline virtual const std::shared_ptr<arrow::Buffer> buffer() const override

Get the buffer of the tensor.

Returns

The shared pointer to an arrow buffer which holds the data buffer of the tensor.

inline const std::shared_ptr<ArrowTensorT> ArrowTensor()

Return a view of the original tensor so that it can be used as arrow’s Tensor.

Public Static Functions

static inline std::shared_ptr<Object> Create()
template<typename T>
class vineyard::TensorBuilder : public vineyard::ITensorBuilder, public vineyard::TensorBaseBuilder<T>

TensorBuilder is used for building tensors that supported by vineyard.

tparam T

Public Functions

inline TensorBuilder(Client &client, std::vector<int64_t> const &shape)

Initialize the TensorBuilder with the tensor shape.

Parameters
  • client – The client connected to the vineyard server.

  • shape – The shape of the tensor.

inline TensorBuilder(Client &client, std::vector<int64_t> const &shape, std::vector<int64_t> const &partition_index)

Initialize the TensorBuilder for a partition of a GlobalTensor.

Parameters
  • client – The client connected to the vineyard server.

  • shape – The shape of the partition.

  • partition_index – The partition index in the global tensor.

inline std::vector<int64_t> const &shape() const

Get the shape of the tensor.

Returns

The shape vector where the ith element represents the size of the ith axis.

inline std::vector<int64_t> const &partition_index() const

Get the index of this partition in the global tensor.

Returns

The index vector where the ith element represents the index in the ith axis.

inline void set_shape(std::vector<int64_t> const &shape)

Set the shape of the tensor.

Parameters

shape – The vector for the shape, where the ith element represents the size of the shape in the ith axis.

inline void set_partition_index(std::vector<int64_t> const &partition_index)

Set the index in the global tensor.

Parameters

partition_index – The vector of indices, where the ith element represents the index in the ith axis.

inline std::vector<int64_t> strides() const

Get the strides of the tensor.

Returns

The strides of the tensor. The definition of the tensor’s strides can be found in https://pytorch.org/docs/stable/tensor_attributes.html

inline T *data() const

Get the data pointer of the tensor.

inline Status Build(Client &client) override

Build the tensor.

Parameters

client – The client connceted to the vineyard server.

class vineyard::DataFrame : public vineyard::Registered<DataFrame>

Public Functions

inline virtual void Construct(const ObjectMeta &meta) override

Construct an object from metadata. The metadata meta should comes from client’s GetMetaData method.

The implementation of Construct method is usually boilerplate. Vineyard provides a code generator to help developers code there own data structures and can be shared via vineyard.

Parameters

meta – The metadata that be used to constrct the object.

const std::vector<json> &Columns() const

Get the column names.

Returns

The vector of column names.

std::shared_ptr<ITensor> Index() const

Get the index of dataframe.

Returns

The shared pointer to the index tensor.

std::shared_ptr<ITensor> Column(json const &column) const

Get the column of the given column name.

Parameters

column – The given column name.

Returns

The shared pointer to the column tensor.

const std::pair<size_t, size_t> partition_index() const

Get the partition index of the global dataframe.

Returns

The pair of the partition_index on rows and the partition_index on columns.

const std::pair<size_t, size_t> shape() const

Get the shape of the dataframe.

Returns

The pair of the number of rows and the number of columns.

const std::shared_ptr<arrow::RecordBatch> RecordBatchView() const

Get a RecordBatch view for the dataframe.

Public Static Functions

static inline std::shared_ptr<Object> Create()
class vineyard::DataFrameBuilder : public vineyard::DataFrameBaseBuilder

DataFrameBuilder is used for constructing dataframes that supported by vineyard.

Public Functions

inline explicit DataFrameBuilder(Client &client)
const std::pair<size_t, size_t> partition_index() const

Get the partition index of the global dataframe.

Returns

The pair of the partition_index on rows and the partition_index on columns.

void set_partition_index(size_t partition_index_row, size_t partition_index_column)

Set the index in the global dataframe.

Parameters
  • partition_index_row – The row index.

  • partition_index_column – The column index.

void set_row_batch_index(size_t row_batch_index)

Set the row batch index in the global dataframe. Note that the row batch index gives the order of batches on rows.

Parameters

row_batch_index – The row batch index.

void set_index(std::shared_ptr<ITensorBuilder> builder)

Set the index of dataframe by add a index column to dataframe.

Parameters

builder – The index tensor builder.

std::shared_ptr<ITensorBuilder> Column(json const &column) const

Get the column of the given column name.

Parameters

column – The given column name.

Returns

The shared pointer to the column tensor.

void AddColumn(json const &column, std::shared_ptr<ITensorBuilder> builder)

Add a column to the dataframe by registering a tensor builder to the column name. When the dataframe is built, the tensor builder will be employed to build the column.

Parameters
  • column – The name of the column.

  • builder – The tensor builder for the column.

void DropColumn(json const &column)

Drop the column with the given column name.

Parameters

column – The name of column to be dropped.

virtual Status Build(Client &client) override

Build the dataframe object.

Parameters

client – The client connected to the vineyard server.

class vineyard::Pair : public vineyard::Registered<Pair>

Public Functions

inline virtual void Construct(const ObjectMeta &meta) override

Construct an object from metadata. The metadata meta should comes from client’s GetMetaData method.

The implementation of Construct method is usually boilerplate. Vineyard provides a code generator to help developers code there own data structures and can be shared via vineyard.

Parameters

meta – The metadata that be used to constrct the object.

inline const std::shared_ptr<Object> First() const

Get the first element of the pair.

Returns

The shared pointer to the first object.

inline const std::shared_ptr<Object> Second() const

Get the second element of the pair.

Returns

The shared pointer to the second object.

inline const iterator begin() const

Get the beginning iterator.

Returns

The beginning iterrator.

inline const iterator end() const

Get the ending iterator.

Returns

The ending iterator.

Public Static Functions

static inline std::shared_ptr<Object> Create()
class iterator : public std::iterator<std::bidirectional_iterator_tag, std::shared_ptr<Object>, size_t, const std::shared_ptr<Object>*, std::shared_ptr<Object>>

The iterator for the pair object to iterate from the first to the last element.

Public Functions

inline explicit iterator(Pair const *pair, size_t index = 0)
inline iterator &operator++()
inline bool operator==(iterator other) const
inline bool operator!=(iterator other) const
inline reference operator*() const
class vineyard::PairBuilder : public vineyard::PairBaseBuilder

PairBuilder is designed for building pairs.

Public Functions

inline explicit PairBuilder(Client &client)
inline std::shared_ptr<ObjectBuilder> First()

Get the builder for the first element.

Returns

The builder for the first element.

inline std::shared_ptr<ObjectBuilder> Second()

Get the builder for the second element.

Returns

The builder for the second element.

inline void SetFirst(std::shared_ptr<ObjectBuilder> const &first)

Set the builder for the first element. When building the pair, the builder will be invoked to build the first element.

Parameters

first – The builder for the first object.

inline void SetFirst(std::shared_ptr<Object> const &first)

Set the builder for the first element. When building the pair, the builder will be invoked to build the first element.

Parameters

first – The value for the first object.

inline void SetSecond(std::shared_ptr<ObjectBuilder> const &second)

Set the builder for the second element. When building the pair, the builder will be invoked to build the second element.

Parameters

second – The builder for the second object.

inline void SetSecond(std::shared_ptr<Object> const &second)

Set the builder for the second element. When building the pair, the builder will be invoked to build the second element.

Parameters

second – The value for the second object.

class vineyard::Tuple : public vineyard::Registered<Tuple>

The tuple type in vineyard.

Public Functions

inline virtual void Construct(const ObjectMeta &meta) override

Construct an object from metadata. The metadata meta should comes from client’s GetMetaData method.

The implementation of Construct method is usually boilerplate. Vineyard provides a code generator to help developers code there own data structures and can be shared via vineyard.

Parameters

meta – The metadata that be used to constrct the object.

inline const size_t Size() const

Get the size of the tuple, i.e., the number of elements it contains.

Returns

The size of the tuple.

inline const std::shared_ptr<Object> At(size_t index) const

Get the value at the given index.

Parameters

index – The given index to get the value.

inline const iterator begin() const

Get the beginning iterator.

Returns

The beginning iterrator.

inline const iterator end() const

Get the ending iterator.

Returns

The ending iterator.

Public Static Functions

static inline std::shared_ptr<Object> Create()
class iterator : public std::iterator<std::bidirectional_iterator_tag, std::shared_ptr<Object>, size_t, const std::shared_ptr<Object>*, std::shared_ptr<Object>>

The iterator for the tuple object to iterate from the first to the last element.

Public Functions

inline explicit iterator(Tuple const *tuple, size_t index = 0)
inline iterator &operator++()
inline bool operator==(iterator other) const
inline bool operator!=(iterator other) const
inline reference operator*() const
class vineyard::TupleBuilder : public vineyard::TupleBaseBuilder

TupleBuilder is designed for generating tuples.

Public Functions

inline explicit TupleBuilder(Client &client)
inline explicit TupleBuilder(Client &client, size_t const size)

Initialize the TupleBuilder with a given size.

Parameters
  • client – The client connected to the vineyard server.

  • size – The size of the tuple to build.

inline const size_t Size() const

Get the size of the tuple, i.e., the number of elements it contains.

Returns

The size of the tuple.

inline void SetSize(size_t size)

Set the size for the tuple. Note that the size of a tuple can be set only once.

Parameters

size – The size for the tuple.

inline std::shared_ptr<ObjectBuilder> At(size_t index)

Get the builder at the given index. Here the index is bound-checked.

Parameters

index – The given index.

Returns

The builder at the given index.

inline void SetValue(size_t idx, std::shared_ptr<ObjectBuilder> const &value)

Set the builder for the value at the given index. When building the tuple, the builder will be invoked to build the value.

Parameters
  • idx – The index of the value.

  • value – The builder for the value for the given index.

inline void SetValue(size_t idx, std::shared_ptr<Object> const &value)

Set the builder for the value at the given index. When building the tuple, the builder will be invoked to build the value.

Parameters
  • idx – The index of the value.

  • value – The value for the given index.

template<typename T>
class vineyard::Scalar : public vineyard::Registered<Scalar<T>>

The scalar type in vineyard. Note that the value of the scalar is writing into the meta tree as a string directly, instead of storing the value in the shared memory with a blob.

tparam T

The type for the value.

Public Functions

inline virtual void Construct(const ObjectMeta &meta) override

Construct an object from metadata. The metadata meta should comes from client’s GetMetaData method.

The implementation of Construct method is usually boilerplate. Vineyard provides a code generator to help developers code there own data structures and can be shared via vineyard.

Parameters

meta – The metadata that be used to constrct the object.

inline const T Value() const

Get the value of the scalar.

Returns

The value of the scalar.

inline AnyType Type() const

Get the type of the scalar.

Returns

The type of the scalar.

Public Static Functions

static inline std::shared_ptr<Object> Create()
template<typename T>
class vineyard::ScalarBuilder : public vineyard::ScalarBaseBuilder<T>

ScalarBuilder is used for building scalars that supported by vineyard.

tparam T

The type for the scalar.

Public Functions

inline explicit ScalarBuilder(Client &client)
inline explicit ScalarBuilder(Client &client, T const &value)

Initialize the scalar with the value.

Parameters
  • client – The client connected to the vineyard server.

  • value – The value for the scalar.

inline void SetValue(T const &value)

Set the value of the scalar.

Parameters

value – The value for the scalar.

Distributed data types

class vineyard::GlobalTensor : public vineyard::Registered<GlobalTensor>, private vineyard::GlobalObject

GlobalTensor is a holder for a set of tensor chunks that are distributed over many vineyard nodes.

Public Functions

virtual void Construct(const ObjectMeta &meta) override

Construct an object from metadata. The metadata meta should comes from client’s GetMetaData method.

The implementation of Construct method is usually boilerplate. Vineyard provides a code generator to help developers code there own data structures and can be shared via vineyard.

Parameters

meta – The metadata that be used to constrct the object.

std::vector<int64_t> const &shape() const
std::vector<int64_t> const &partition_shape() const
const std::vector<std::shared_ptr<ITensor>> &LocalPartitions(Client &client) const

Get the local partitions of the vineyard instance that is connected from the client.

Parameters

client – The client connected to a vineyard instance.

Returns

The vector of pointers to the local partitions.

const std::vector<std::shared_ptr<ITensor>> &LocalPartitions(const InstanceID instance_id) const

Get the local partitions stored in the given vineyard instance.

Parameters

instance_id – The given ID of the vineyard instance.

Returns

The vector of pointers to the local partitions.

Public Static Functions

static inline std::shared_ptr<Object> Create()
class vineyard::GlobalTensorBuilder : public vineyard::GlobalTensorBaseBuilder

GlobalTensorBuilder is designed for building global tensors.

Public Functions

inline explicit GlobalTensorBuilder(Client &client)
std::vector<int64_t> const &partition_shape() const

Get the partition shape of the global tensor. Here the ith element represents how many partitions are made on the ith axis.

void set_partition_shape(std::vector<int64_t> const &partition_shape)

Set the partition shape of the global tensor. Here the ith element represents how many partitions are made on the ith axis.

std::vector<int64_t> const &shape() const

Get the entire shape of the global tensor.

void set_shape(std::vector<int64_t> const &shape)

Set the entire shape of the global tensor.

void AddPartition(const ObjectID partition_id)

Add a partition in the vineyard instance to the global tensor.

Parameters
  • instance_id – The ID of the vineyard instance.

  • partition_id – The ObjectID of the partition to added.

void AddPartitions(const std::vector<ObjectID> &partition_ids)

Add a group of partitions in the vineyard instance to the global tensor.

Parameters
  • instance_id – The ID of the vineyard instance.

  • partition_id – The vector of ObjectIDs for the group of partitions to add.

virtual std::shared_ptr<Object> _Seal(Client &client) override

Seal the meta data of the global tensor. When creating a global tensor, clients from different machines that are connected to different vineyard instances will sync the partition info to seal the meta data for the global tensor.

Parameters

client – The client connected to the vineyard server.

virtual Status Build(Client &client) override

Build the global tensor.

Parameters

client – The client connected to the vineyard server.

class vineyard::GlobalDataFrame : public vineyard::Registered<GlobalDataFrame>, private vineyard::GlobalObject

GlobalDataFrame is a DataFrame that refers a set of dataframe chunks in many vineyardd nodes.

Public Functions

virtual void Construct(const ObjectMeta &meta) override

Construct an object from metadata. The metadata meta should comes from client’s GetMetaData method.

The implementation of Construct method is usually boilerplate. Vineyard provides a code generator to help developers code there own data structures and can be shared via vineyard.

Parameters

meta – The metadata that be used to constrct the object.

const std::pair<size_t, size_t> partition_shape() const

Set the partition shape of the global dataframe.

Parameters
  • partition_shape_row – The number of partitions on rows.

  • partition_shape_column – The number of partitions on columns.

const std::vector<std::shared_ptr<DataFrame>> &LocalPartitions(Client &client) const

Get the local partitions of the vineyard instance that is connected from the client.

Parameters

client – The client connected to a vineyard instance.

Returns

The vector of pointers to the local partitions.

const std::vector<std::shared_ptr<DataFrame>> &LocalPartitions(const InstanceID instance_id) const

Get the local partitions stored in the given vineyard instance.

Parameters

instance_id – The given ID of the vineyard instance.

Returns

The vector of pointers to the local partitions.

Public Static Functions

static inline std::shared_ptr<Object> Create()
class vineyard::GlobalDataFrameBuilder : public vineyard::GlobalDataFrameBaseBuilder

GlobalDataFrameBuilder is designed for building global dataframes.

Public Functions

inline explicit GlobalDataFrameBuilder(Client &client)
~GlobalDataFrameBuilder() = default
const std::pair<size_t, size_t> partition_shape() const

Get the partition shape of the global dataframe.

Returns

The pair of <number_of_partitions_on_rows, number_of_partitions_on_columns>.

void set_partition_shape(size_t partition_shape_row, size_t partition_shape_column)

Set the partition shape of the global dataframe.

Parameters
  • partition_shape_row – The number of partitions on rows.

  • partition_shape_column – The number of partitions on columns.

void AddPartition(ObjectID const partition_id)

Add a partition in the vineyard instance to the global dataframe.

Parameters
  • instance_id – The ID of the vineyard instance.

  • partition_id – The ObjectID of the partition to added.

void AddPartitions(const std::vector<ObjectID> &partition_ids)

Add a group of partitions in the vineyard instance to the global dataframe.

Parameters
  • instance_id – The ID of the vineyard instance.

  • partition_id – The vector of ObjectIDs for the group of partitions to added.

virtual std::shared_ptr<Object> _Seal(Client &client) override

Seal the meta data of the global dataframe. When creating a global dataframe, clients from different machines that are connected to different vineyard instances will sync the partition info to seal the meta data for the global dataframe.

Parameters

client – The client connected to the vineyard server.

virtual Status Build(Client &client) override

Build the global dataframe object.

Parameters

client – The client connected to the vineyard server.