C++ API Reference¶

Objects¶

using vineyard::ObjectID = uint64_t¶: ObjectID is an opaque type for vineyard’s object id. The object ID is generated by vineyard server, the underlying type of ObjectID is a 64-bit unsigned integer.

class Object : public vineyard::ObjectBase, public std::enable_shared_from_this<Object>¶

Object is the core concept in vineyard. Object can be a scalar, a tuple, a vector, a tensor, or even a distributed graph in vineyard. Objects are stored in vineyard, and can be shared to process that connects to the same vineyard cluster.

Every object in vineyard has a unique identifier ObjectID that can be passed back and forth in the computation pipeline. An object is composed by a metadata, and a set of blobs.

Object in vineyard is by-design to be hierarchical, and can have other Object as members. For example, a tensor object may has a vector as its payload, and a distributed dataframe has many dataframe objects as its chunks, and a dataframe is composed by an array of tensors as columns.

Subclassed by vineyard::Registered< Array< Entry > >, vineyard::Registered< Array< T > >, vineyard::Registered< ArrowFragmentGroup >, vineyard::Registered< BaseBinaryArray< ArrayType > >, vineyard::Registered< BaseListArray< ArrayType > >, vineyard::Registered< Blob >, vineyard::Registered< BooleanArray >, vineyard::Registered< DataFrame >, vineyard::Registered< FixedSizeBinaryArray >, vineyard::Registered< FixedSizeListArray >, vineyard::Registered< Hashmap< K, V, prime_number_hash_wy< K >, std::equal_to< K > > >, vineyard::Registered< HDataFrame< T > >, vineyard::Registered< KVCache >, vineyard::Registered< KVCacheBlock >, vineyard::Registered< NullArray >, vineyard::Registered< NumericArray< T > >, vineyard::Registered< ParallelStream >, vineyard::Registered< PerfectHashmap< K, V > >, vineyard::Registered< RecordBatch >, vineyard::Registered< RefcntMapObject >, vineyard::Registered< RemoteBlob >, vineyard::Registered< Scalar< T > >, vineyard::Registered< SchemaProxy >, vineyard::Registered< Sequence >, vineyard::Registered< VineyardFile >, vineyard::ArrowFragmentBase, vineyard::ITensor, vineyard::KVTensor, vineyard::Registered< T >

Public Functions

virtual ~Object()¶

const ObjectID id() const¶: The object id of this object.

const ObjectMeta &meta() const¶: The metadata of this object.

const size_t nbytes() const¶: The nbytes of this object, can be treated as the memory usage of this object.

virtual void Construct(const ObjectMeta &meta)¶

Construct an object from metadata. The metadata meta should come from client’s GetMetaData method.

The implementation of Construct method is usually boilerplate. Vineyard provides a code generator to help developers code their own data structures and can be shared via vineyard.

Parameters:: meta – The metadata that be used to construct the object.

inline virtual void PostConstruct(const ObjectMeta &meta)¶

PostConstruct is called at the end of Construct to perform user-specific constructions that the code generator cannot handle.

Parameters:: meta – The metadata that be used to construct the object.

inline virtual Status Build(Client &client) final¶: Object is also a kind of ObjectBase, and can be used as a member to construct new objects. The Object type also has a Build method but it does nothing, though.

virtual std::shared_ptr<Object> _Seal(Client &client) final¶: Object is also a kind of ObjectBase, and can be used as a member to construct new objects. The Object type also has a _Seal method but it does nothing, though.

Status Persist(ClientBase &client) const¶

Persist the object to make it visible for clients that connected to other vineyardd instances in the cluster.

Parameters:: client – The client that to be used to perform the Persist request.

const bool IsLocal() const¶

The verb “local” means it is a local object to the client. A local object’s blob can be accessed by the client in a zero-copy fashion. Otherwise only the metadata is accessible.

Returns:: True iff the object is a local object.

const bool IsPersist() const¶

The verb “persist” means it is visible for client that connected to other vineyardd instances in the cluster. After Persist the object becomes persistent.

Returns:: True iff the object is a persistent object.

const bool IsGlobal() const¶

The verb “global” means it is a global object and only refers some local objects.

Returns:: True iff the object is a global object.

Protected Functions

inline Object()¶

Protected Attributes

ObjectID id_¶

mutable ObjectMeta meta_¶

Friends

friend class ClientBase

friend class Client

friend class PlasmaClient

friend class RPCClient

friend class ObjectMeta

friend class ObjectFactory

class ObjectBuilder : public vineyard::ObjectBase ¶

Subclassed by vineyard::CollectionBuilder< DataFrame >, vineyard::CollectionBuilder< ITensor >, vineyard::CollectionBuilder< RecordBatch >, vineyard::ArrowFragmentGroupBuilder, vineyard::BlobWriter, vineyard::BooleanArrayBaseBuilder, vineyard::CollectionBuilder< T >, vineyard::DataFrameBaseBuilder, vineyard::FixedSizeBinaryArrayBaseBuilder, vineyard::FixedSizeListArrayBaseBuilder, vineyard::KVCacheBlockBuilder, vineyard::KVCacheBuilder, vineyard::KVTensorBuilder, vineyard::NullArrayBaseBuilder, vineyard::ParallelStreamBaseBuilder, vineyard::RecordBatchBaseBuilder, vineyard::RefcntMapObjectBuilder, vineyard::SchemaProxyBaseBuilder, vineyard::SequenceBaseBuilder, vineyard::TensorBaseBuilder< std::string >

Public Functions

inline virtual ~ObjectBuilder()¶

virtual Status Build(Client &client) override = 0¶

Building the object means construct all blobs of this object to vineyard server.

Parameters:: client – The vineyard client that been used to create blobs in the connected vineyard server.

virtual std::shared_ptr<Object> Seal(Client &client)¶

virtual Status Seal(Client &client, std::shared_ptr<Object> &object)¶

virtual std::shared_ptr<Object> _Seal(Client &client) override¶

Sealing the object means construct the metadata for the object and create metadata in vineyard server.

Parameters:: client – The vineyard client that been used to create metadata in the connected vineyard server.

virtual Status _Seal(Client &client, std::shared_ptr<Object> &object)¶

inline bool sealed() const¶

Protected Functions

inline void set_sealed(bool const sealed = true)¶

class ObjectBase¶

ObjectBase is the most base class for vineyard’s Object and ObjectBuilder.

An ObjectBase instance is a build-able value that it’s Build method put blobs * into vineyard server, and it’s _Seal method is responsible for creating metadata in vineyard server for the object.

Subclassed by vineyard::Object, vineyard::ObjectBuilder

Public Functions

virtual Status Build(Client &client) = 0¶

Building the object means construct all blobs of this object to vineyard server.

Parameters:: client – The vineyard client that been used to create blobs in the connected vineyard server.

virtual std::shared_ptr<Object> _Seal(Client &client) = 0¶

Sealing the object means construct the metadata for the object and create metadata in vineyard server.

Parameters:: client – The vineyard client that been used to create metadata in the connected vineyard server.

Metadata¶

class ObjectMeta¶

ObjectMeta is the type for metadata of an Object. The ObjectMeta can be treated as a dict-like type. If the metadata obtained from vineyard, the metadata is readonly. Otherwise key-value attributes or object members could be associated with the metadata to construct a new vineyard object.

Public Types

using const_iterator = nlohmann::detail::iteration_proxy_value<json::const_iterator>¶

Public Functions

ObjectMeta()¶

~ObjectMeta()¶

ObjectMeta(const ObjectMeta&)¶

ObjectMeta &operator=(ObjectMeta const &other)¶

void SetClient(ClientBase *client)¶: Associate the client with the metadata.

ClientBase *GetClient() const¶: Get the associate client with the metadata.

void SetId(const ObjectID &id)¶: Set the object ID for the metadata.

const ObjectID GetId() const¶: Get the corresponding object ID of the metadata.

const Signature GetSignature() const¶: Get the corresponding object signature of the metadata.

void ResetSignature()¶: Reset the signatures in the metadata (for duplicating objects).

void SetGlobal(bool global = true)¶: Mark the vineyard object as a global object.

const bool IsGlobal() const¶: Check whether the vineyard object is a global object.

void SetTypeName(const std::string &type_name)¶: Set the typename of the metadata. The typename will be used to resolve in the ObjectFactory to create new object instances when get objects from vineyard server.

std::string const &GetTypeName() const¶: Get the typename of the metadata.

void SetNBytes(const size_t nbytes)¶: Set the nbytes attribute for the metadata, basically it indicates the memory usage of the object.

const size_t GetNBytes() const¶: Get the nbytes attribute of the object. Note that the nbytes attribute doesn’t always reflect the TOTAL monopolistic space usage of the bulk store, since two objects may share some blobs.

const InstanceID GetInstanceId() const¶: Get the instance ID of vineyard server where the metadata is created on.

const bool IsLocal() const¶: Whether the object meta is the metadata of a local object.

void ForceLocal() const¶: Mark the metadata as a “local” metadata to make sure the construct process proceed.

const bool Haskey(std::string const &key) const¶: Whether specific key exists in this metadata.

const bool HasKey(std::string const &key) const¶: Whether specific key exists in this metadata.

void ResetKey(std::string const &key)¶: Reset the given key in the metadata.

void AddKeyValue(const std::string &key, const std::string &value)¶

Add a string value entry to the metadata.

Parameters:

key – The name of metadata entry.
value – The value of the metadata entry.

template<typename T> inline void AddKeyValue(const std::string &key, T const &value)¶

Add a generic value entry to the metadata.

Parameters:

T – The type of metadata’s value.
key – The name of metadata entry.
value – The value of the metadata entry.

template<typename T> inline void AddKeyValue(const std::string &key, std::set<T> const &values)¶

Add a generic set value entry to the metadata.

Parameters:

T – The type of metadata’s value.
key – The name of metadata entry, it will be first convert to JSON array by nlohmann::json.
value – The value of the metadata entry.

template<typename T> inline void AddKeyValue(const std::string &key, std::vector<T> const &values)¶

Add a generic vector value entry to the metadata.

Parameters:

T – The type of metadata’s value.
key – The name of metadata entry, it will be first convert to JSON array by nlohmann::json.
value – The value of the metadata entry.

template<typename T> inline void AddKeyValue(const std::string &key, Tuple<T> const &values)¶

Add a generic vector value entry to the metadata.

Parameters:

T – The type of metadata’s value.
key – The name of metadata entry, it will be first convert to JSON array by nlohmann::json.
value – The value of the metadata entry.

template<typename Value> inline void AddKeyValue(const std::string &key, std::map<std::string, Value> const &values)¶

Add a associated map value entry to the metadata.

Parameters:

Value – The type of metadata’s value.
key – The name of metadata entry, it will be first convert to string.
value – The value of the metadata entry.

template<typename Value> inline void AddKeyValue(const std::string &key, Map<std::string, Value> const &values)¶

Add a associated map value entry to the metadata.

Parameters:

Value – The type of metadata’s value.
key – The name of metadata entry, it will be first convert to string.
value – The value of the metadata entry.

template<typename Value> inline void AddKeyValue(const std::string &key, std::map<json, Value> const &values)¶

Add a associated map value entry to the metadata.

Parameters:

Value – The type of metadata’s value.
key – The name of metadata entry, it will be first convert to string.
value – The value of the metadata entry.

template<typename Value> inline void AddKeyValue(const std::string &key, Map<json, Value> const &values)¶

Add a associated map value entry to the metadata.

Parameters:

Value – The type of metadata’s value.
key – The name of metadata entry, it will be first convert to string.
value – The value of the metadata entry.

template<typename Value> inline void AddKeyValue(const std::string &key, std::unordered_map<std::string, Value> const &values)¶

Add a associated map value entry to the metadata.

Parameters:

Value – The type of metadata’s value.
key – The name of metadata entry, it will be first convert to string.
value – The value of the metadata entry.

template<typename Value> inline void AddKeyValue(const std::string &key, UnorderedMap<std::string, Value> const &values)¶

Add a associated map value entry to the metadata.

Parameters:

Value – The type of metadata’s value.
key – The name of metadata entry, it will be first convert to string.
value – The value of the metadata entry.

template<typename Value> inline void AddKeyValue(const std::string &key, std::unordered_map<json, Value> const &values)¶

Add a associated map value entry to the metadata.

Parameters:

Value – The type of metadata’s value.
key – The name of metadata entry, it will be first convert to string.
value – The value of the metadata entry.

template<typename Value> inline void AddKeyValue(const std::string &key, UnorderedMap<json, Value> const &values)¶

Add a associated map value entry to the metadata.

Parameters:

Value – The type of metadata’s value.
key – The name of metadata entry, it will be first convert to string.
value – The value of the metadata entry.

void AddKeyValue(const std::string &key, json const &values)¶

Add a nlohmann::json value entry to the metadata.

Parameters:

T – The type of metadata’s value.
key – The name of metadata entry, it will be first convert to string by nlohmann::json.
value – The value of the metadata entry.

inline const std::string GetKeyValue(const std::string &key) const¶

Get string metadata value.

Parameters:: key – The key of metadata.

template<typename T> inline const T GetKeyValue(const std::string &key) const¶

Get generic metadata value.

Parameters:

T – The type of metadata value.
key – The key of metadata.

template<typename T> inline void GetKeyValue(const std::string &key, T &value) const¶

Get generic metadata value, with automatically type deduction.

Parameters:

T – The type of metadata value.
key – The key of metadata.
value – The result will be stored in value, the generic result is pass by reference to help type deduction.

template<typename T> inline void GetKeyValue(const std::string &key, std::set<T> &values) const¶

Get generic set metadata value, with automatically type deduction.

Parameters:

T – The type of metadata value.
key – The key of metadata.
value – The result will be stored in value, the generic result is pass by reference to help type deduction.

template<typename T> inline void GetKeyValue(const std::string &key, std::vector<T> &values) const¶

Get generic vector metadata value, with automatically type deduction.

Parameters:

T – The type of metadata value.
key – The key of metadata.
value – The result will be stored in value, the generic result is pass by reference to help type deduction.

template<typename T> inline void GetKeyValue(const std::string &key, Tuple<T> &values) const¶

Get generic vector metadata value, with automatically type deduction.

Parameters:

T – The type of metadata value.
key – The key of metadata.
value – The result will be stored in value, the generic result is pass by reference to help type deduction.

template<typename Value> inline void GetKeyValue(const std::string &key, std::map<std::string, Value> &values) const¶

Get associated map metadata value, with automatically type deduction.

Parameters:

Value – The type of metadata value.
key – The key of metadata.
value – The result will be stored in value, the generic result is pass by reference to help type deduction.

template<typename Value> inline void GetKeyValue(const std::string &key, Map<std::string, Value> &values) const¶

Get associated map metadata value, with automatically type deduction.

Parameters:

Value – The type of metadata value.
key – The key of metadata.
value – The result will be stored in value, the generic result is pass by reference to help type deduction.

template<typename Value> inline void GetKeyValue(const std::string &key, std::map<json, Value> &values) const¶

Get associated map metadata value, with automatically type deduction.

Parameters:

Value – The type of metadata value.
key – The key of metadata.
value – The result will be stored in value, the generic result is pass by reference to help type deduction.

template<typename Value> inline void GetKeyValue(const std::string &key, Map<json, Value> &values) const¶

Get associated map metadata value, with automatically type deduction.

Parameters:

Value – The type of metadata value.
key – The key of metadata.
value – The result will be stored in value, the generic result is pass by reference to help type deduction.

template<typename Value> inline void GetKeyValue(const std::string &key, std::unordered_map<std::string, Value> &values) const¶

Get associated map metadata value, with automatically type deduction.

Parameters:

Value – The type of metadata value.
key – The key of metadata.
value – The result will be stored in value, the generic result is pass by reference to help type deduction.

template<typename Value> inline void GetKeyValue(const std::string &key, UnorderedMap<std::string, Value> &values) const¶

Get associated map metadata value, with automatically type deduction.

Parameters:

Value – The type of metadata value.
key – The key of metadata.
value – The result will be stored in value, the generic result is pass by reference to help type deduction.

template<typename Value> inline void GetKeyValue(const std::string &key, std::unordered_map<json, Value> &values) const¶

Get associated map metadata value, with automatically type deduction.

Parameters:

Value – The type of metadata value.
key – The key of metadata.
value – The result will be stored in value, the generic result is pass by reference to help type deduction.

template<typename Value> inline void GetKeyValue(const std::string &key, UnorderedMap<json, Value> &values) const¶

Get associated map metadata value, with automatically type deduction.

Parameters:

Value – The type of metadata value.
key – The key of metadata.
value – The result will be stored in value, the generic result is pass by reference to help type deduction.

void GetKeyValue(const std::string &key, json &value) const¶

Get json metadata value.

Parameters:

key – The key of metadata.
value – The result will be stored in value.

void AddRemoteBlob(const RemoteBlob &blob)¶

Add remote blob’s buffer to ObjectMeta.

Parameters:: blob – The remote blob to be added.

void AddRemoteBlob(ObjectID id, const std::shared_ptr<Buffer> &buffer)¶

Add remote blob’s buffer to ObjectMeta.

Parameters:

id – The object ID of remote blob.
buffer – The buffer of remote blob.

void AddMember(const std::string &name, const ObjectMeta &member)¶

Add member to ObjectMeta.

Parameters:

name – The name of member object.
member – The metadata of member object to be added.

void AddMember(const std::string &name, const Object &member)¶

Add member to ObjectMeta.

Parameters:

name – The name of member object.
member – The member object to be added.

void AddMember(const std::string &name, const Object *member)¶

Add member to ObjectMeta.

Parameters:

name – The name of member object.
member – The member object to be added.

void AddMember(const std::string &name, const std::shared_ptr<Object> &member)¶

Add member to ObjectMeta.

Parameters:

name – The name of member object.
member – The member object to be added.

void AddMember(const std::string &name, const ObjectID member_id)¶

Add member to ObjectMeta.

Parameters:

name – The name of member object.
member – The object ID of member object to be added.

std::shared_ptr<Object> GetMember(const std::string &name) const¶

Get member value from vineyard.

Parameters:: name – The name of member object.
Returns:: member The member object.

Status GetMember(const std::string &name, std::shared_ptr<Object> &object) const¶

Get member value from vineyard.

Parameters:: name – The name of member object.
Returns:: member The member object.

template<typename T> inline std::shared_ptr<T> GetMember(const std::string &name) const¶

Get member value from vineyard.

Parameters:: name – The name of member object.
Returns:: member The member object.

template<typename T> inline Status GetMember(const std::string &name, std::shared_ptr<T> &object) const¶

Get member value from vineyard.

Parameters:: name – The name of member object.
Returns:: member The member object.

ObjectMeta GetMemberMeta(const std::string &name) const¶

Get member’s ObjectMeta value.

Parameters:: name – The name of member object.
Returns:: member The metadata of member object. will be stored in value.

Status GetMemberMeta(const std::string &name, ObjectMeta &meta) const¶

Get member’s ObjectMeta value.

Parameters:

name – The name of member object.
meta – The metadata of member object.

Returns:

Whether the member metadata has been found.

Status GetBuffer(const ObjectID blob_id, std::shared_ptr<Buffer> &buffer) const¶: Get buffer member (directed or indirected) from the metadata. The metadata should has already been initialized.

void SetBuffer(const ObjectID &id, const std::shared_ptr<Buffer> &buffer)¶

void Reset()¶: Reset the metadata as an initialized one.

size_t MemoryUsage() const¶: Compute the memory usage of this object.

size_t MemoryUsage(json &usages, const bool pretty = true) const¶: Compute the memory usage of this object, in a json tree format.

uint64_t Timestamp() const¶: Get the associate timestamp (in milliseconds) inside the metadata, 0 means unknown.

json Labels() const¶

const std::string Label(const std::string &key) const¶

std::string ToString() const¶

void PrintMeta() const¶

const bool incomplete() const¶

const json &MetaData() const¶

json &MutMetaData()¶

void SetMetaData(ClientBase *client, const json &meta)¶

inline const_iterator begin() const¶

inline const_iterator end() const¶

const std::shared_ptr<BufferSet> &GetBufferSet() const¶

template<> const json GetKeyValue(const std::string &key) const¶

template<> const json GetKeyValue(const std::string &key) const

Public Static Functions

static std::unique_ptr<ObjectMeta> Unsafe(std::string meta, size_t nobjects, ObjectID *objects, uintptr_t *pointers, size_t *sizes)¶: Construct object metadata from unsafe sources.

static std::unique_ptr<ObjectMeta> Unsafe(json meta, size_t nobjects, ObjectID *objects, uintptr_t *pointers, size_t *sizes)¶: Construct object metadata from unsafe sources.

Vineyard Clients¶

class ClientBase¶

ClientBase is the base class for vineyard IPC and RPC client.

ClientBase implements common communication stuffs, and leave the IPC and RPC specific functionalities to Client and RPCClient.

Vineyard’s Client and RPCClient is non-copyable.

Subclassed by vineyard::BasicIPCClient, vineyard::RPCClient

Public Functions

ClientBase()¶

inline virtual ~ClientBase()¶

inline virtual Status Release(ObjectID const &id)¶

ClientBase(const ClientBase&) = delete¶

ClientBase(ClientBase&&) = delete¶

ClientBase &operator=(const ClientBase&) = delete¶

ClientBase &operator=(ClientBase&&) = delete¶

Status GetData(const ObjectID id, json &tree, const bool sync_remote = false, const bool wait = false)¶

Get object metadata from vineyard using given object ID.

Parameters:

id – The ID of the requested object.
tree – The returned metadata tree of the requested object.
sync_remote – Whether to trigger an immediate remote metadata synchronization before get specific metadata. Default is false.
wait – The request could be blocked util the object with given id has been created on vineyard by other clients. Default is false.

Returns:

Status that indicates whether the get action succeeds.

Status GetData(const std::vector<ObjectID> &ids, std::vector<json> &trees, const bool sync_remote = false, const bool wait = false)¶

Get multiple object metadatas from vineyard using given object IDs.

Parameters:

ids – The IDs of the requested objects
trees – The returned metadata trees of the requested objects
sync_remote – Whether to trigger an immediate remote metadata synchronization before get specific metadata. Default is false.
wait – The request could be blocked util the object with given id has been created on vineyard by other clients. Default is false.

Returns:

Status that indicates whether the get action has succeeded.

Status CreateData(const json &tree, ObjectID &id, Signature &signature, InstanceID &instance_id)¶

Create the metadata in the vineyard server.

Parameters:

tree – The metadata that will be created in vineyard.
id – The returned object ID of the created data.
instance_id – The vineyard instance ID where this object is created. at.

Returns:

Status that indicates whether the create action has succeeded.

Status CreateData(const std::vector<json> &trees, std::vector<ObjectID> &ids, std::vector<Signature> &signatures, std::vector<InstanceID> &instance_ids)¶

Status CreateMetaData(ObjectMeta &meta_data, ObjectID &id)¶

Create the metadata in the vineyard server, after created, the resulted object id in the meta_data will be filled.

Parameters:

meta_data – The metadata that will be created in vineyard.
id – The returned object ID of the created metadata.

Returns:

Status that indicates whether the create action has succeeded.

Status CreateMetaData(std::vector<ObjectMeta> &meta_datas, std::vector<ObjectID> &ids)¶

Status CreateMetaData(ObjectMeta &meta_data, InstanceID const &instance_id, ObjectID &id)¶

Create the metadata in the vineyard server with specified instance id, after created, the resulted object id in the meta_data will be filled.

The specified instance id is required that the metadata can be created using the RPC client on the specified instance as a placeholder.

Parameters:

meta_data – The metadata that will be created in vineyard.
id – The returned object ID of the created metadata.

Returns:

Status that indicates whether the create action has succeeded.

Status CreateMetaData(std::vector<ObjectMeta> &meta_datas, InstanceID const &instance_id, std::vector<ObjectID> &ids)¶

virtual Status GetMetaData(const ObjectID id, ObjectMeta &meta_data, const bool sync_remote = false) = 0¶

Get the meta-data of the requested object.

Parameters:

id – The ID of the requested object
meta_data – The returned metadata of the requested object
sync_remote – Whether trigger remote sync

Returns:

Status that indicates whether the get action has succeeded.

Status SyncMetaData()¶

Sync remote metadata from etcd to the connected vineyardd.

Returns:: Status that indicates whether the sync action has succeeded.

Status DelData(const ObjectID id, const bool force = false, const bool deep = true)¶

Delete metadata in vineyard. When the object is a used by other object, it will be deleted only when the force parameter is specified.

Parameters:

id – The ID to delete.
force – Whether to delete the object forcely. Forcely delete an object means the object and objects which use this object will be delete. Default is false.
deep – Whether to delete the member of this object. Default is true. Note that when deleting object which has direct blob members, the processing on those blobs yields a “deep” behavior.
memory_trim – Whether to trim the memory pool inside the shared memory allocator to return the unused physical memory back to the OS.

Returns:

Status that indicates whether the delete action has succeeded.

Status DelData(const ObjectID id, const bool force, const bool deep, const bool memory_trim)¶

Status DelData(const std::vector<ObjectID> &ids, const bool force = false, const bool deep = true)¶

Delete multiple metadatas in vineyard.

Parameters:

ids – The IDs to delete.
force – Whether to delete the object forcely. Forcely delete an object means the object and objects which use this object will be delete. Default is false.
deep – Whether to delete the member of this object. Default is true. Note that when deleting objects which have direct blob members, the processing on those blobs yields a “deep” behavior.
memory_trim – Whether to trim the memory pool inside the shared memory allocator to return the unused physical memory back to the OS.

Returns:

Status that indicates whether the delete action has succeeded.

Status DelData(const std::vector<ObjectID> &ids, const bool force, const bool deep, const bool memory_trim)¶

Status ListData(std::string const &pattern, bool const regex, size_t const limit, std::unordered_map<ObjectID, json> &meta_trees)¶

List objectmetas in vineyard, using the given typename patterns.

Parameters:

pattern – The pattern string that will be used to matched against objects’ typename.
regex – Whether the pattern is a regular expression pattern. Default is false. When regex is false, the pattern will be treated as a glob pattern.
limit – The number limit for how many objects will be returned at most.
meta_trees – An map that contains the returned object metadatas.

Returns:

Status that indicates whether the list action has succeeded.

Status ListNames(std::string const &pattern, bool const regex, size_t const limit, std::map<std::string, ObjectID> &names)¶

List names in vineyard, using the given name patterns.

Parameters:

pattern – The pattern string that will be used to matched against objects’ name.
regex – Whether the pattern is a regular expression pattern. Default is false. When regex is false, the pattern will be treated as a glob pattern.
limit – The number limit for how many objects will be returned at most.
names – An map that contains the returned names and corresponding object ids.

Returns:

Status that indicates whether the list action has succeeded.

Status CreateStream(const ObjectID &id)¶

Allocate a stream on vineyard. The metadata of parameter id must has already been created on vineyard.

Parameters:: id – The id of metadata that will be used to create stream.
Returns:: Status that indicates whether the create action has succeeded.

Status OpenStream(const ObjectID &id, StreamOpenMode mode)¶

open a stream on vineyard. Failed if the stream is already opened on the given mode.

Parameters:

id – The id of stream to mark.
mode – The mode, StreamOpenMode::read or StreamOpenMode::write.

Returns:

Status that indicates whether the open action has succeeded.

Status PushNextStreamChunk(ObjectID const id, ObjectID const chunk)¶

Push a chunk from a stream. When there’s no more chunk available in the stream, i.e., the stream has been stopped, a status code kStreamDrained or kStreamFinish will be returned, otherwise the reader will be blocked until writer creates a new chunk in the stream.

Parameters:

id – The id of the stream.
blob – The immutable chunk generated by the writer of the stream.

Returns:

Status that indicates whether the polling has succeeded.

Status PullNextStreamChunk(ObjectID const id, ObjectID &chunk)¶

Pull a chunk from a stream. When there’s no more chunk available in the stream, i.e., the stream has been stopped, a status code kStreamDrained or kStreamFinish will be returned, otherwise the reader will be blocked until writer creates a new chunk in the stream.

Parameters:

id – The id of the stream.
chunk – The immutable chunk generated by the writer of the stream.

Returns:

Status that indicates whether the polling has succeeded.

Status PullNextStreamChunk(ObjectID const id, ObjectMeta &chunk)¶

Parameters:

id – The id of the stream.
chunk – The immutable chunk generated by the writer of the stream.

Returns:

Status that indicates whether the polling has succeeded.

Status PullNextStreamChunk(ObjectID const id, std::shared_ptr<Object> &chunk)¶

Parameters:

id – The id of the stream.
chunk – The immutable chunk generated by the writer of the stream.

Returns:

Status that indicates whether the polling has succeeded.

Status StopStream(ObjectID const id, bool failed)¶

Stop a stream, mark it as finished or aborted.

Parameters:

id – The id of the stream.
failed – Whether the stream is stopped at a successful state. True means the stream has been exited normally, otherwise false.

Returns:

Status that indicates whether the request has succeeded.

Status DropStream(ObjectID const id)¶

Close a stream, mark it as aborted if it is not finished yet.

Parameters:: id – The id of the stream.
Returns:: Status that indicates whether the request has succeeded.

Status Persist(const ObjectID id)¶

Persist the given object to etcd to make it visible to clients that been connected to vineyard servers in the cluster.

Parameters:: id – The object id of object that will be persisted.
Returns:: Status that indicates whether the persist action has succeeded.

Status IfPersist(const ObjectID id, bool &persist)¶

Check if the given object has been persist to etcd.

Parameters:

id – The object id to check.
persist – The result variable will be stored in persist as return value. The value true means the object is visible by other vineyard servers.

Returns:

Status that indicates whether the check has succeeded.

Status Exists(const ObjectID id, bool &exists)¶

Check if the given object exists in vineyard server.

Parameters:

id – The object id to check.
exists – The result variable will be stored in exists as return value. The value true means the object exists.

Returns:

Status that indicates whether the check has succeeded.

Status ShallowCopy(const ObjectID id, ObjectID &target_id)¶

Make a shallow copy on the given object. A “shallow copy” means the result object has the same type with the source object and they shares all member objects.

Parameters:

id – The object id to shallow copy.
target_id – The result object id will be stored in target_id as return value.

Returns:

Status that indicates whether the shallow copy has succeeded.

Status ShallowCopy(const ObjectID id, json const &extra_metadata, ObjectID &target_id)¶

Make a shallow copy on the given object. A “shallow copy” means the result object has the same type with the source object and they share all member objects.

Parameters:

id – The object id to shallow copy.
extra_metadata – Feed extra metadata when shallow copying.
target_id – The result object id will be stored in target_id as return value.

Returns:

Status that indicates whether the shallow copy has succeeded.

Status PutName(const ObjectID id, std::string const &name)¶

Vineyard support associating a user-specific name with an object. PutName registers a name entry in vineyard server. An object can be assoiciated with more than one names.

Parameters:

id – The ID of the object.
name – The user-specific name that will be associated with the given object.

Returns:

Status that indicates whether the request has succeeded.

Status GetName(const std::string &name, ObjectID &id, const bool wait = false)¶

Retrieve the object ID by associated name.

Parameters:

name – The name of the requested object.
id – The returned object ID.
wait – If wait is specified, the request will be blocked util the given name has been registered on vineyard by other clients.

Returns:

Status that indicates whether the query has succeeded.

Status DropName(const std::string &name)¶

Deregister a name entry. The associated object will be kept and won’t be deleted.

Parameters:: name – The name that will be deregistered.
Returns:: Status that indicates whether the query has succeeded.

Status MigrateObject(const ObjectID object_id, ObjectID &result_id)¶

Migrate remote object to connected instance.

Parameters:

object_id – The existing object that will be migrated to current vineyardd.
result_id – Record the result object id.

Returns:

Status that indicates if the migration success.

Status Clear()¶: Clear all objects that are visible to current instances in the cluster.

Status MemoryTrim(bool &trimmed)¶: Trim the memory pool inside the shared memory allocator to return the unused physical memory back to the OS kernel, like the malloc_trim API from glibc.

Status Label(const ObjectID object, std::string const &key, std::string const &value)¶

Associate given labels to an existing object.

Parameters:: object – Object to be labeled.

Status Label(const ObjectID object, std::map<std::string, std::string> const &labels)¶

Associate given labels to an existing object.

Parameters:: object – Object to be labeled.

Status Evict(std::vector<ObjectID> const &objects)¶

Evict objects from the vineyardd server.

Parameters:: objects – Objects to be evicted.

Status Load(std::vector<ObjectID> const &objects, const bool pin = false)¶

Load objects to ensure they resident in vineyardd server’s memory, with an optional arguments to pin these objects to prevent from being spilled.

Parameters:: objects – Objects to be reloaded, and possibly pinned.

Status Unpin(std::vector<ObjectID> const &objects)¶

Unpin objects from the vineyardd server’s memory.

Parameters:: objects – Objects to be unpinned.

bool Connected() const¶

Check if the client still connects to the vineyard server.

Returns:: True when the connection is still alive, otherwise false.

virtual void Disconnect()¶: Disconnect this client.

Status Open(std::string const &ipc_socket)¶

Create a new anonymous session in vineyardd and connect to it .

Parameters:: ipc_socket – Location of the UNIX domain socket.
Returns:: Status that indicates whether the connection of has succeeded.

void CloseSession()¶: Close the session that the client is connecting to.

inline std::string const &IPCSocket()¶

Get the UNIX domain socket location of the connected vineyardd server.

Returns:: Location of the IPC socket.

inline std::string const &RPCEndpoint()¶

The RPC endpoint of the connected vineyardd server.

Returns:: The RPC endpoint.

inline virtual bool IsIPC() const¶

Check if the client is an IPC client.

Returns:: True if the client is an IPC client, otherwise false.

inline virtual bool IsRPC() const¶

Check if the client is a RPC client.

Returns:: True if the client is a RPC client, otherwise false.

inline const InstanceID instance_id() const¶

Get the instance id of the connected vineyard server.

Note that for RPC client the instance id is not available.

Returns:: The vineyard server’s instance id.

inline virtual const InstanceID remote_instance_id() const¶

Get the remote instance id of the connected vineyard server.

Note that for RPC client the instance id is not available, thus we have the “remote instance id” to indicate which server we are connecting to.

Returns:: The vineyard server’s instance id.

inline const SessionID session_id() const¶

Get the session id of the connected vineyard server.

Note that for RPC client the instance id is not available.

Returns:: The vineyard server’s session id.

Status ClusterInfo(std::map<InstanceID, json> &meta)¶

Retrieve the cluster information of the connected vineyard server.

The cluster information for every instance mainly includes the host address (i.e., ip address).

Returns:: Status that indicates whether the query has succeeded.

Status InstanceStatus(std::shared_ptr<struct InstanceStatus> &status)¶

Return the status of connected vineyard instance.

If success, the status parameter will be reset as an instance of InstanceStatus.

Parameters:: status – The result instance status.
Returns:: Status that indicates whether the query has succeeded.

Status Instances(std::vector<InstanceID> &instances)¶

List all instances in the connected vineyard cluster.

Parameters:: A – list of instance IDs will be stored in instances.
Returns:: Status that indicates whether the query has succeeded.

inline const std::string &Version() const¶

Get the version of connected vineyard server.

Returns:: Return a version string MAJOR.MINOR.PATCH that follows the semver convention.

virtual Status TryAcquireLock(std::string key, bool &result, std::string &actural_key) = 0¶

Try to acquire a distributed lock.

Parameters:: key – The key of the lock.
Returns:: Status that indicates whether the lock process succeeds.

virtual Status TryReleaseLock(std::string key, bool &result) = 0¶

Try to release a distributed lock.

Parameters:: key – The key of the lock.
Returns:: Status that indicates whether the unlock process succeeds.

Status Debug(const json &debug, json &tree)¶

Issue a debug request.

Parameters:

debug – The payload that will be sent to the debug handler.
result – The result that returned by the debug handler.

Returns:

Status that indicates whether the debug action succeeds.

inline bool compression_enabled() const¶

inline void set_compression_enabled(bool enabled = true)¶

inline void set_timeout_seconds(int seconds)¶

inline int get_timeout_seconds() const¶

Protected Functions

Status doWrite(const std::string &message_out)¶

Status doRead(std::string &message_in)¶

Status doRead(json &root)¶

Protected Attributes

mutable bool connected_¶

std::string ipc_socket_¶

std::string rpc_endpoint_¶

int vineyard_conn_¶

SessionID session_id_¶

InstanceID instance_id_¶

std::string server_version_¶

bool support_rpc_compression_ = false¶

mutable std::recursive_mutex client_mutex_¶

bool compression_enabled_ = false¶

int timeout_seconds_ = 300¶

class Client : public vineyard::BasicIPCClient, protected vineyard::detail::UsageTracker<ObjectID, Payload, Client>¶

Vineyard’s IPC Client connects to to UNIX domain socket of the vineyard server. Vineyard’s IPC Client talks to vineyard server and manipulate objects in vineyard.

Public Functions

inline Client()¶

~Client() override¶

Status Connect()¶

Connect to vineyard using the UNIX domain socket file specified by the environment variable VINEYARD_IPC_SOCKET.

Returns:: Status that indicates whether the connect has succeeded.

Status Connect(std::string const &username, std::string const &password)¶

Connect to vineyard using the UNIX domain socket file specified by the environment variable VINEYARD_IPC_SOCKET.

Returns:: Status that indicates whether the connect has succeeded.

Status Connect(const std::string &ipc_socket)¶

Connect to vineyardd using the given UNIX domain socket ipc_socket.

Parameters:: ipc_socket – Location of the UNIX domain socket.
Returns:: Status that indicates whether the connect has succeeded.

Status Connect(const std::string &ipc_socket, std::string const &username, std::string const &password)¶

Connect to vineyardd using the given UNIX domain socket ipc_socket.

Parameters:: ipc_socket – Location of the UNIX domain socket.
Returns:: Status that indicates whether the connect has succeeded.

virtual void Disconnect() override¶: Disconnect this client.

Status Open(std::string const &ipc_socket)¶

Create a new anonymous session in vineyardd and connect to it .

Parameters:: ipc_socket – Location of the UNIX domain socket.
Returns:: Status that indicates whether the connection of has succeeded.

Status Open(std::string const &ipc_socket, std::string const &username, std::string const &password)¶

Create a new anonymous session in vineyardd and connect to it .

Parameters:: ipc_socket – Location of the UNIX domain socket.
Returns:: Status that indicates whether the connection of has succeeded.

Status Fork(Client &client)¶: Create a new client using self UNIX domain socket.

virtual Status GetMetaData(const ObjectID id, ObjectMeta &meta_data, const bool sync_remote = false) override¶

Obtain metadata from vineyard server.

Parameters:

id – The object id to get.
meta_data – The result metadata will be store in meta_data as return value.
sync_remote – Whether to trigger an immediate remote metadata synchronization before get specific metadata. Default is false.

Returns:

Status that indicates whether the get action has succeeded.

Status FetchAndGetMetaData(const ObjectID id, ObjectMeta &meta_data, const bool sync_remote = false)¶

Obtain metadata from vineyard server.

Parameters:

id – The object id to get.
meta_data – The result metadata will be store in meta_data as return value.
sync_remote – Whether to trigger an immediate remote metadata synchronization before get specific metadata. Default is false.

Returns:

Status that indicates whether the get action has succeeded.

Status GetMetaData(const std::vector<ObjectID> &ids, std::vector<ObjectMeta>&, const bool sync_remote = false)¶

Obtain multiple metadatas from vineyard server.

Parameters:

ids – The object ids to get.
meta_data – The result metadata will be store in meta_data as return value.
sync_remote – Whether to trigger an immediate remote metadata synchronization before get specific metadata. Default is false.

Returns:

Status that indicates whether the get action has succeeded.

Status CreateBlob(size_t size, std::unique_ptr<BlobWriter> &blob)¶

Create a blob in vineyard server. When creating a blob, vineyard server’s bulk allocator will prepare a block of memory of the requested size, then map the memory to client’s process to share the allocated memory.

Parameters:

size – The size of requested blob.
blob – The result mutable blob will be set in blob.

Returns:

Status that indicates whether the create action has succeeded.

Status CreateBlobs(const std::vector<size_t> &sizes, std::vector<std::unique_ptr<BlobWriter>> &blobs)¶

Create blobs in vineyard server. When creating a blob, vineyard server’s bulk allocator will prepare a block of memory of the requested size, then map the memory to client’s process to share the allocated memory.

Parameters:

sizes – The size of requested blobs.
blob – The result mutable blob will be set in blob.

Returns:

Status that indicates whether the create action has succeeded.

Status GetBlob(ObjectID const id, std::shared_ptr<Blob> &blob)¶

Get a blob from vineyard server.

Parameters:: id – the blob to get.
Returns:: Status that indicates whether the get action has succeeded.

Status GetBlob(ObjectID const id, bool unsafe, std::shared_ptr<Blob> &blob)¶

Get a blob from vineyard server, and optionally bypass the “sealed” check.

Parameters:: id – the blob to get.
Returns:: Status that indicates whether the get action has succeeded.

Status GetBlobs(std::vector<ObjectID> const ids, std::vector<std::shared_ptr<Blob>> &blobs)¶

Get a blob from vineyard server.

Parameters:: id – the blob to get.
Returns:: Status that indicates whether the get action has succeeded.

Status GetBlobs(std::vector<ObjectID> const ids, const bool unsafe, std::vector<std::shared_ptr<Blob>> &blobs)¶

Get a blob from vineyard server, and optionally bypass the “sealed” check.

Parameters:: id – the blob to get.
Returns:: Status that indicates whether the get action has succeeded.

Status CreateDiskBlob(size_t size, const std::string &path, std::unique_ptr<BlobWriter> &blob)¶

Claim a shared blob that backed by a file on disk. Users need to provide either a filename to mmap, or an expected size to allocate the file on disk.

When mapping existing file as a blob, if the existing file size is smaller than specified “size”, the file will be enlarged using ftruncate().

Note that when deleting the blob that backed by files, the file won’t be automatically deleted by vineyard.

Parameters:

size – expected file size to allocate.
path – use existing file as the mmap buffer.

Returns:

Status that indicates whether the get action has succeeded.

Status GetNextStreamChunk(ObjectID const id, size_t const size, std::unique_ptr<MutableBuffer> &blob)¶

Allocate a chunk of given size in vineyard for a stream. When the request cannot be satisfied immediately, e.g., vineyard doesn’t have enough memory or the specified has accumulated too many chunks, the request will be blocked until the request been processed.

Parameters:

id – The id of the stream.
size – The size of the chunk to allocate.
blob – The allocated mutable buffer will be set in blob.

Returns:

Status that indicates whether the allocation has succeeded.

Status PullNextStreamChunk(ObjectID const id, std::unique_ptr<Buffer> &chunk)¶

Parameters:

id – The id of the stream.
chunk – The immutable chunk generated by the writer of the stream.

Returns:

Status that indicates whether the polling has succeeded.

std::shared_ptr<Object> GetObject(const ObjectID id, const bool sync_remote = true)¶

Get an object from vineyard. The ObjectFactory will be used to resolve the constructor of the object.

Parameters:

id – The object id to get.
sync_remote – Whether to trigger an immediate remote metadata

Returns:

A std::shared_ptr<Object> that can be safely cast to the underlying concrete object type. When the object doesn’t exists an std::runtime_error exception will be raised.

std::shared_ptr<Object> FetchAndGetObject(const ObjectID id, const bool sync_remote = true)¶

Get an object from vineyard. The ObjectFactory will be used to resolve the constructor of the object.

Parameters:

id – The object id to get.
sync_remote – Whether to trigger an immediate remote metadata

Returns:

A std::shared_ptr<Object> that can be safely cast to the underlying concrete object type. When the object doesn’t exists an std::runtime_error exception will be raised.

Status GetObject(const ObjectID id, std::shared_ptr<Object> &object, const bool sync_remote = true)¶

Get an object from vineyard. The ObjectFactory will be used to resolve the constructor of the object.

Parameters:

id – The object id to get.
object – The result object will be set in parameter object.
sync_remote – Whether to trigger an immediate remote metadata

Returns:

When errors occur during the request, this method won’t throw exceptions, rather, it results a status to represents the error.

Status FetchAndGetObject(const ObjectID id, std::shared_ptr<Object> &object, const bool sync_remote = true)¶

Get an object from vineyard. The ObjectFactory will be used to resolve the constructor of the object.

Parameters:

id – The object id to get.
object – The result object will be set in parameter object.
sync_remote – Whether to trigger an immediate remote metadata

Returns:

When errors occur during the request, this method won’t throw exceptions, rather, it results a status to represents the error.

template<typename T> inline std::shared_ptr<T> GetObject(const ObjectID id, const bool sync_remote = true)¶

Get an object from vineyard. The type parameter T will be used to resolve the constructor of the object.

Parameters:

id – The object id to get.
sync_remote – Whether to trigger an immediate remote metadata

Returns:

A std::shared_ptr<Object> that can be safely cast to the underlying concrete object type. When the object doesn’t exists an std::runtime_error exception will be raised.

template<typename T> inline std::shared_ptr<T> FetchAndGetObject(const ObjectID id, const bool sync_remote = true)¶

Get an object from vineyard. The type parameter T will be used to resolve the constructor of the object.

Parameters:

id – The object id to get.
sync_remote – Whether to trigger an immediate remote metadata

Returns:

A std::shared_ptr<Object> that can be safely cast to the underlying concrete object type. When the object doesn’t exists an std::runtime_error exception will be raised.

template<typename T> inline Status GetObject(const ObjectID id, std::shared_ptr<T> &object, const bool sync_remote = true)¶

Get an object from vineyard. The type parameter T will be used to resolve the constructor of the object.

This method can be used to get concrete object from vineyard without explicitly dynamic_cast, and the template type parameter can be deduced in many situations:

std::shared_ptr<Array<int>> int_array;
client.GetObject(id, int_array);

Parameters:

id – The object id to get.
object – The result object will be set in parameter object.
sync_remote – Whether to trigger an immediate remote metadata

Returns:

When errors occur during the request, this method won’t throw exceptions, rather, it results a status to represents the error.

template<typename T> inline Status FetchAndGetObject(const ObjectID id, std::shared_ptr<T> &object, const bool sync_remote = true)¶

Get an object from vineyard. The type parameter T will be used to resolve the constructor of the object.

This method can be used to get concrete object from vineyard without explicitly dynamic_cast, and the template type parameter can be deduced in many situations:

std::shared_ptr<Array<int>> int_array;
client.FetchAndGetObject(id, int_array);

Parameters:

id – The object id to get.
object – The result object will be set in parameter object.
sync_remote – Whether to trigger an immediate remote metadata

Returns:

When errors occur during the request, this method won’t throw exceptions, rather, it results a status to represents the error.

std::vector<std::shared_ptr<Object>> GetObjects(const std::vector<ObjectID> &ids, const bool sync_remote = true)¶

Get multiple objects from vineyard.

Parameters:

ids – The object IDs to get.
sync_remote – Whether to trigger an immediate remote metadata

Returns:

A list of objects.

std::vector<std::shared_ptr<Object>> GetObjects(const std::vector<ObjectMeta> &metas)¶

std::vector<ObjectMeta> ListObjectMeta(std::string const &pattern, const bool regex = false, size_t const limit = 5, bool nobuffer = false)¶

List object metadatas in vineyard, using the given typename patterns.

Parameters:

pattern – The pattern string that will be used to matched against objects’ typename.
regex – Whether the pattern is a regular expression pattern. Default is false. When regex is false, the pattern will be treated as a glob pattern.
limit – The number limit for how many objects will be returned at most.

Returns:

A vector of object metadatas that listed from vineyard server.

std::vector<std::shared_ptr<Object>> ListObjects(std::string const &pattern, const bool regex = false, size_t const limit = 5)¶

List objects in vineyard, using the given typename patterns.

Parameters:

pattern – The pattern string that will be used to matched against objects’ typename.
regex – Whether the pattern is a regular expression pattern. Default is false. When regex is false, the pattern will be treated as a glob pattern.
limit – The number limit for how many objects will be returned at most.

Returns:

A vector of objects that listed from vineyard server.

bool IsSharedMemory(const void *target) const¶

Check if the given address belongs to the shared memory region.

Return true if the address (client-side address) comes from the vineyard server.

Parameters:: target – The pointer that been queried.

bool IsSharedMemory(const uintptr_t target) const¶

Check if the given address belongs to the shared memory region.

Return true if the address (client-side address) comes from the vineyard server.

Parameters:: target – The pointer that been queried.

bool IsSharedMemory(const void *target, ObjectID &object_id) const¶

Check if the given address belongs to the shared memory region.

Return true if the address (client-side address) comes from the vineyard server.

Parameters:

target – The pointer that been queried.
object_id – Return the object id of the queried pointer, if found.

bool IsSharedMemory(const uintptr_t target, ObjectID &object_id) const¶

Check if the given address belongs to the shared memory region.

Return true if the address (client-side address) comes from the vineyard server.

Parameters:

target – The pointer that been queried.
object_id – Return the object id of the queried pointer, if found.

Status IsInUse(ObjectID const &id, bool &is_in_use)¶

Check if the blob is a cold blob (no client is using it).

Return true if the the blob is in-use.

Status IsSpilled(ObjectID const &id, bool &is_spilled)¶

Check if the blob is a spilled blob (those no client is using and be dumped on disk).

Return true if the the blob is spilled.

Status AllocatedSize(const ObjectID id, size_t &size)¶: Get the allocated size for the given object.

Status CreateArena(const size_t size, int &fd, size_t &available_size, uintptr_t &base, uintptr_t &space)¶

Status ReleaseArena(const int fd, std::vector<size_t> const &offsets, std::vector<size_t> const &sizes)¶

Status ShallowCopy(ObjectID const id, ObjectID &target_id, Client &source_client)¶: Move the selected objects from the source session to the target.

Status ShallowCopy(PlasmaID const plasma_id, ObjectID &target_id, PlasmaClient &source_client)¶

Status Release(std::vector<ObjectID> const &ids)¶: Decrease the reference count of the object. It will trigger OnRelease behavior when reference count reaches zero. See UsageTracker.

virtual Status Release(ObjectID const &id) override¶

Status DelData(const ObjectID id, const bool force = false, const bool deep = true)¶

Delete metadata in vineyard. When the object is a used by other object, it will be deleted only when the force parameter is specified.

Parameters:

id – The ID to delete.
force – Whether to delete the object forcely. Forcely delete an object means the object and objects which use this object will be delete. Default is false.
deep – Whether to delete the member of this object. Default is true. Note that when deleting object which has direct blob members, the processing on those blobs yields a “deep” behavior.

Returns:

Status that indicates whether the delete action has succeeded.

Status DelData(const ObjectID id, const bool force, const bool deep, const bool memory_trim)¶

Status DelData(const std::vector<ObjectID> &ids, const bool force = false, const bool deep = true)¶

Delete multiple metadatas in vineyard.

Parameters:

ids – The IDs to delete.
force – Whether to delete the object forcely. Forcely delete an object means the object and objects which use this object will be delete. Default is false.
deep – Whether to delete the member of this object. Default is true. Note that when deleting objects which have direct blob members, the processing on those blobs yields a “deep” behavior.

Returns:

Status that indicates whether the delete action has succeeded.

Status DelData(const std::vector<ObjectID> &ids, const bool force, const bool deep, const bool memory_trim)¶

Status CreateGPUBuffer(const size_t size, ObjectID &id, Payload &payload, std::shared_ptr<MutableBuffer> &buffer)¶

Create a GPU buffer on vineyard server. See also CreateBuffer.

Parameters:

size – The size of requested GPU buffer.
id – The result GPU buffer object id.
payload – The result GPU buffer payload.
buffer – The result mutable GPU buffer object related to the GPU blob.

Returns:

Status that indicates whether the get action has succeeded.

Status GetGPUBuffers(const std::set<ObjectID> &ids, const bool unsafe, std::map<ObjectID, std::shared_ptr<Buffer>> &buffers)¶

Get a set of blobs from vineyard server. See also GetBuffer.

Parameters:

ids – Object ids for the blobs to get.
buffers – The result GPU buffer objects related to GPU blobs.

Returns:

Status that indicates whether the get action has succeeded.

Status GetGPUBuffer(const ObjectID id, const bool unsafe, std::shared_ptr<Buffer> &buffer)¶

Get a single GPU blob from vineyard server. See also GetBuffer.

Parameters:

id – Object id for the blob to get.
buffer – The result gpu buffer object related to the GPU blob.

Returns:

Status that indicates whether the get action has succeeded.

virtual Status TryAcquireLock(std::string key, bool &result, std::string &actual_key) override¶

Try to acquire a distributed lock.

Parameters:: key – The key of the lock.
Returns:: Status that indicates whether the lock process succeeds.

virtual Status TryReleaseLock(std::string key, bool &result) override¶

Try to release a distributed lock.

Parameters:: key – The key of the lock.
Returns:: Status that indicates whether the unlock process succeeds.

Status PullNextStreamChunk(ObjectID const id, ObjectID &chunk)¶

Parameters:

id – The id of the stream.
chunk – The immutable chunk generated by the writer of the stream.

Returns:

Status that indicates whether the polling has succeeded.

Status PullNextStreamChunk(ObjectID const id, ObjectMeta &chunk)¶

Parameters:

id – The id of the stream.
chunk – The immutable chunk generated by the writer of the stream.

Returns:

Status that indicates whether the polling has succeeded.

Status PullNextStreamChunk(ObjectID const id, std::shared_ptr<Object> &chunk)¶

Parameters:

id – The id of the stream.
chunk – The immutable chunk generated by the writer of the stream.

Returns:

Status that indicates whether the polling has succeeded.

Public Static Functions

static Client &Default()¶

Get a default client reference, using the UNIX domain socket file specified by the environment variable VINEYARD_IPC_SOCKET.

Returns:: A reference of the default Client instance.

Protected Functions

Status OnRelease(ObjectID const &id)¶: Required by UsageTracker. When reference count reaches zero, send the ReleaseRequest to server.

Status OnDelete(ObjectID const &id)¶: Required by UsageTracker. Currently, the deletion does not respect the reference count, it will send the DelData to server and do the deletion forcely.

Status PostSeal(ObjectMeta const &meta_data)¶: Increase reference count after a new object is sealed.

Status GetDependency(ObjectID const &id, std::set<ObjectID> &bids)¶: Send request to server to get all underlying blobs of a object.

Status CreateBuffer(const size_t size, ObjectID &id, Payload &payload, std::shared_ptr<MutableBuffer> &buffer)¶

Status CreateBuffers(const std::vector<size_t> &sizes, std::vector<ObjectID> &ids, std::vector<Payload> &payloads, std::vector<std::shared_ptr<MutableBuffer>> &buffers)¶

Status GetBuffer(const ObjectID id, std::shared_ptr<Buffer> &buffer)¶

Get a blob from vineyard server. When obtaining blobs from vineyard server, the memory address in the server process will be mmapped to the client’s process to share the memory.

Parameters:

id – Object id for the blob to get.
buffer – The result immutable blob will be set in blob. Note that blob is special, since it can be get as immutable object before sealing.

Returns:

Status that indicates whether the create action has succeeded.

Status GetBuffers(const std::set<ObjectID> &ids, std::map<ObjectID, std::shared_ptr<Buffer>> &buffers)¶

Get a set of blobs from vineyard server. See also GetBuffer.

Parameters:

ids – Object ids for the blobs to get.
buffers – The result immutable blobs will be added to buffers.

Returns:

Status that indicates whether the get action has succeeded.

Status GetBufferSizes(const std::set<ObjectID> &ids, std::map<ObjectID, size_t> &sizes)¶

Get the size of blobs from vineyard server.

Parameters:

ids – Object ids for the blobs to get.
sizes – The result sizes of immutable blobs will be added to sizes.

Returns:

Status that indicates whether the get action has succeeded.

Status DropBuffer(const ObjectID id, const int fd)¶

An (unsafe) internal-usage method that drop the buffer, without checking the dependency. To achieve the “DeleteObject” semantic for blobs, use DelData instead.

Note that the target buffer could be both sealed and unsealed.

Status ShrinkBuffer(const ObjectID id, const size_t size)¶

An (unsafe) internal-usage method that shrink the buffer, without allocating a new buffer and copying. The buffer to be shrink shouldn’t be sealed.

In the underlying server-side implementation, madvise is used.

Status Seal(ObjectID const &object_id)¶: mark the blob as sealed to control the visibility of a blob, client can never Get an unsealed blob.

inline virtual bool IsIPC() const override¶

Check if the client is an IPC client.

Returns:: True means the client is an IPC client.

Friends

friend class detail::UsageTracker< ObjectID, Payload, Client >

class RPCClient : public vineyard::ClientBase ¶

Public Functions

~RPCClient() override¶

virtual void Disconnect() override¶: Disconnect this client.

Status Connect()¶

Connect to vineyard using the TCP endpoint specified by the environment variable VINEYARD_RPC_ENDPOINT. If the environment variable VINEYARD_RDMA_ENDPOINT is set, the RDMA endpoint will also be used.

Returns:: Status that indicates whether the connect has succeeded.

Status Connect(std::string const &username, std::string const &password)¶

Returns:: Status that indicates whether the connect has succeeded.

Status Connect(const std::string &rpc_endpoint)¶

Connect to vineyardd using the given TCP endpoint rpc_endpoint. If the environment variable VINEYARD_RDMA_ENDPOINT is set, the RDMA endpoint will also be used.

Parameters:: rpc_endpoint – The TPC endpoint of vineyard server, in the format of host:port.
Returns:: Status that indicates whether the connect has succeeded.

Status Connect(const std::string &rpc_endpoint, std::string const &username, std::string const &password, const std::string &rdma_endpoint = "")¶

Connect to vineyardd using the given TCP endpoint rpc_endpoint.

Parameters:

rpc_endpoint – The TPC endpoint of vineyard server, in the format of host:port.
rdma_endpoint – The RDMA endpoint of RDMA server, in the format of host:port.

Returns:

Status that indicates whether the connect has succeeded.

Status Connect(const std::string &rpc_endpoint, const SessionID session_id, std::string const &username = "", std::string const &password = "", const std::string &rdma_endpoint = "")¶

Connect to vineyardd using the given TCP endpoint rpc_endpoint.

Parameters:

rpc_endpoint – The TPC endpoint of vineyard server, in the format of host:port.
session_id – Connect to specified session.
rdma_endpoint – The RDMA endpoint of RDMA server, in the format of host:port.

Returns:

Status that indicates whether the connect has succeeded.

Status Connect(const std::string &host, uint32_t port, const std::string &rdma_host = "", uint32_t rdma_port = -1)¶

Connect to vineyardd using the given TCP host and port.

Parameters:

host – The host of vineyard server.
port – The TCP port of vineyard server’s RPC service.
rdma_host – The host of RDMA server.
rdma_port – The RDMA port of RDMA server’s RDMA service.

Returns:

Status that indicates whether the connect has succeeded.

Status Connect(const std::string &host, uint32_t port, std::string const &username, std::string const &password, const std::string &rdma_host = "", uint32_t rdma_port = -1)¶

Connect to vineyardd using the given TCP host and port.

Parameters:

host – The host of vineyard server.
port – The TCP port of vineyard server’s RPC service.
rdma_host – The host of RDMA server.
rdma_port – The RDMA port of RDMA server’s RDMA service.

Returns:

Status that indicates whether the connect has succeeded.

Status Connect(const std::string &host, uint32_t port, const SessionID session_id, std::string const &username = "", std::string const &password = "", const std::string &rdma_host = "", uint32_t rdma_port = -1)¶

Connect to vineyardd using the given TCP host and port.

Parameters:

host – The host of vineyard server.
port – The TCP port of vineyard server’s RPC service.
session_id – Connect to specified session.
rdma_host – The host of RDMA server.
rdma_port – The RDMA port of RDMA server’s RDMA service.

Returns:

Status that indicates whether the connect has succeeded.

Status Fork(RPCClient &client)¶: Create a new client using self endpoint.

virtual Status GetMetaData(const ObjectID id, ObjectMeta &meta_data, const bool sync_remote = false) override¶

Obtain metadata from vineyard server. Note that unlike IPC client, RPC client doesn’t map shared memorys to the client’s process.

Parameters:

id – The object id to get.
meta_data – The result metadata will be store in meta_data as return value.
sync_remote – Whether to trigger an immediate remote metadata synchronization before get specific metadata. Default is false.

Returns:

Status that indicates whether the get action has succeeded.

Status GetMetaData(const std::vector<ObjectID> &id, std::vector<ObjectMeta> &meta_data, const bool sync_remote = false)¶

Obtain multiple metadatas from vineyard server.

Parameters:

ids – The object ids to get.
meta_data – The result metadata will be store in meta_data as return value.
sync_remote – Whether to trigger an immediate remote metadata synchronization before get specific metadata. Default is false.

Returns:

Status that indicates whether the get action has succeeded.

std::shared_ptr<Object> GetObject(const ObjectID id, const bool sync_remote = true)¶

Get an object from vineyard. The ObjectFactory will be used to resolve the constructor of the object.

In RPCClient, all blob fields in the result object are unaccessible, access those fields will trigger an std::runtime_error.

Parameters:

id – The object id to get.
sync_remote – Whether to trigger an immediate remote metadata synchronization before get specific object. Default is true.

Returns:

A std::shared_ptr<Object> that can be safely cast to the underlying concrete object type. When the object doesn’t exists an std::runtime_error exception will be raised.

Status GetObject(const ObjectID id, std::shared_ptr<Object> &object, const bool sync_remote = true)¶

Get an object from vineyard. The ObjectFactory will be used to resolve the constructor of the object.

Parameters:

id – The object id to get.
object – The result object will be set in parameter object.
sync_remote – Whether to trigger an immediate remote metadata synchronization before get specific object. Default is true.

Returns:

When errors occur during the request, this method won’t throw exceptions, rather, it results a status to represents the error.

Status BatchedGetObjects(const std::vector<ObjectMeta> metas, std::vector<std::shared_ptr<Object>> &objects)¶

std::vector<std::shared_ptr<Object>> GetObjects(const std::vector<ObjectID> &ids, const bool sync_remote = true)¶

Get multiple objects from vineyard.

Parameters:

ids – The object IDs to get.
sync_remote – Whether to trigger an immediate remote metadata synchronization before get specific object. Default is true.

Returns:

A list of objects.

template<typename T> inline std::shared_ptr<T> GetObject(const ObjectID id, const bool sync_remote = true)¶

Get an object from vineyard. The type parameter T will be used to resolve the constructor of the object.

Parameters:

id – The object id to get.
sync_remote – Whether to trigger an immediate remote metadata synchronization before get specific object. Default is true.

Returns:

A std::shared_ptr<Object> that can be safely cast to the underlying concrete object type. When the object doesn’t exists an std::runtime_error exception will be raised.

template<typename T> inline Status GetObject(const ObjectID id, std::shared_ptr<T> &object, const bool sync_remote = true)¶

Get an object from vineyard. The type parameter T will be used to resolve the constructor of the object.

This method can be used to get concrete object from vineyard without explicitly dynamic_cast, and the template type parameter can be deduced in many situations:

std::shared_ptr<Array<int>> int_array;
client.GetObject(id, int_array);

Parameters:

id – The object id to get.
object – The result object will be set in parameter object.
sync_remote – Whether to trigger an immediate remote metadata synchronization before get specific object. Default is true.

Returns:

When errors occur during the request, this method won’t throw exceptions, rather, it results a status to represents the error.

std::vector<ObjectMeta> ListObjectMeta(std::string const &pattern, const bool regex = false, size_t const limit = 5, bool nobuffer = false)¶

List object metadatas in vineyard, using the given typename patterns.

Parameters:

pattern – The pattern string that will be used to matched against objects’ typename.
regex – Whether the pattern is a regular expression pattern. Default is false. When regex is false, the pattern will be treated as a glob pattern.
limit – The number limit for how many objects will be returned at most.

Returns:

A vector of object metadatas that listed from vineyard server.

std::vector<std::shared_ptr<Object>> ListObjects(std::string const &pattern, const bool regex = false, size_t const limit = 5)¶

List objects in vineyard, using the given typename patterns.

Parameters:

pattern – The pattern string that will be used to matched against objects’ typename.
regex – Whether the pattern is a regular expression pattern. Default is false. When regex is false, the pattern will be treated as a glob pattern.
limit – The number limit for how many objects will be returned at most.

Returns:

A vector of objects that listed from vineyard server.

inline virtual const InstanceID remote_instance_id() const override¶

Get the remote instance id of the connected vineyard server.

Note that for RPC client the instance id is not available, thus we have the “remote instance id” to indicate which server we are connecting to.

Returns:: The vineyard server’s instance id.

inline virtual bool IsRPC() const override¶

Check if the client is a RPC client.

Returns:: True means the client is a RPC client.

bool IsFetchable(const ObjectMeta &meta)¶

Whether the instance connected by rpc client is the same as object metadata’s instance.

Returns:: True means the instance is the same as object metadata’s instance.

Status CreateRemoteBlob(std::shared_ptr<RemoteBlobWriter> const &buffer, ObjectMeta &meta)¶: Create a remote blob on the vineyard server.

Status CreateRemoteBlobs(std::vector<std::shared_ptr<RemoteBlobWriter>> const &buffers, std::vector<ObjectMeta> &metas)¶: Create remote blobs on the vineyard server.

Status GetRemoteBlob(const ObjectID &id, std::shared_ptr<RemoteBlob> &buffer)¶

Get the remote blob of the connected vineyard server, using the RPC socket.

Note that getting remote blobs requires an expensive copy over network.

Status GetRemoteBlob(const ObjectID &id, const bool unsafe, std::shared_ptr<RemoteBlob> &buffer)¶

Get the remote blob of the connected vineyard server, using the RPC socket, and optionally bypass the “seal” check.

Note that getting remote blobs requires an expensive copy over network.

Status GetRemoteBlob(const ObjectID &id, const bool unsafe, std::shared_ptr<MutableBuffer> &buffer)¶

Status GetRemoteBlobs(std::vector<ObjectID> const &ids, std::vector<std::shared_ptr<RemoteBlob>> &remote_blobs)¶

Get the remote blobs of the connected vineyard server, using the RPC socket.

Note that getting remote blobs requires an expensive copy over network.

Status GetRemoteBlobs(std::set<ObjectID> const &ids, std::map<ObjectID, std::shared_ptr<RemoteBlob>> &remote_blobs)¶

Status GetRemoteBlobs(std::vector<ObjectID> const &ids, const bool unsafe, std::vector<std::shared_ptr<RemoteBlob>> &remote_blobs)¶

Get the remote blobs of the connected vineyard server, using the RPC socket. and optionally bypass the “seal” check.

Note that getting remote blobs requires an expensive copy over network.

Status GetRemoteBlobs(std::vector<ObjectID> const &ids, const bool unsafe, std::vector<std::shared_ptr<MutableBuffer>> &buffers)¶

Status GetRemoteBlobs(std::set<ObjectID> const &ids, const bool unsafe, std::map<ObjectID, std::shared_ptr<RemoteBlob>> &remote_blobs)¶

virtual Status TryAcquireLock(std::string key, bool &result, std::string &actural_key) override¶

Try to acquire a distributed lock.

Parameters:: key – The key of the lock.
Returns:: Status that indicates whether the lock process succeeds.

virtual Status TryReleaseLock(std::string key, bool &result) override¶

Try to release a distributed lock.

Parameters:: key – The key of the lock.
Returns:: Status that indicates whether the unlock process succeeds.

inline const std::string rdma_endpoint()¶

Get the RDMA endpoint of the connected vineyard server.

Returns:: The RDMA endpoint of the connected vineyard server.

Vineyard Server¶

struct InstanceStatus¶

Public Functions

explicit InstanceStatus(const json &tree)¶

Initialize the status value using a json returned from the vineyard server.

Parameters:: tree – JSON that returned from the vineyard server.

Public Members

const InstanceID instance_id¶: The connected instance id.

const std::string deployment¶: The deployment manner, can be local or distributed.

const size_t memory_usage¶: The current memory usage in vineyard server, in bytes.

const size_t memory_limit¶: The memory upper bound of this vineyard server, in bytes.

const size_t deferred_requests¶: How many requests are deferred in the queue.

const size_t ipc_connections¶: How many Client connects to this vineyard server.

const size_t rpc_connections¶: How many RPCClient connects to this vineyard server.

Blob¶

class Blob : public vineyard::Registered<Blob>¶

The unit to store data payload in vineyard.

When the client gets a blob from vineyard, the vineyard server maps a chunk of memory from its memory space to the client space in a zero-copy fashion.

Public Functions

size_t size() const¶

Get the size of the blob, i.e., the number of bytes of the data payload in the blob.

Note that the size of blob is the “allocated size” of the blob, and may (usually) not be the same value of the requested size.

Returns:: The (allocated) size of the blob.

size_t allocated_size() const¶

Get the allocated size of the blob, i.e., the number of bytes of the data payload in the blob.

Returns:: The allocated size of the blob.

const char *data() const¶

Get the const data pointer of the data payload in the blob.

Returns:: The const data pointer.

const std::shared_ptr<vineyard::Buffer> &Buffer() const¶

Get the buffer of the blob.

Returns:: The buffer which holds the data payload of the blob.

const std::shared_ptr<arrow::Buffer> ArrowBuffer() const¶

Get the arrow buffer of the blob.

Returns:: The buffer which holds the data payload of the blob.

const std::shared_ptr<vineyard::Buffer> BufferOrEmpty() const¶

Get the buffer of the blob, ensure a valid shared_ptr been returned even the blob is empty (size == 0).

Returns:: The buffer which holds the data payload of the blob.

const std::shared_ptr<arrow::Buffer> ArrowBufferOrEmpty() const¶

Get the arrow buffer of the blob, ensure a valid shared_ptr been returned even the blob is empty (size == 0).

Returns:: The buffer which holds the data payload of the blob.

virtual void Construct(ObjectMeta const &meta) override¶

Construct the blob locally for the given object meta.

Parameters:: meta – The given object meta.

void Dump() const¶: Dump the buffer for debugging.

Public Static Functions

static inline std::unique_ptr<Object> Create()¶

static std::shared_ptr<Blob> MakeEmpty(Client &client)¶

Create an empty blob in the vineyard server.

Parameters:: client – The client connected to the vineyard server.

static std::shared_ptr<Blob> FromAllocator(Client &client, const ObjectID object_id, const uintptr_t pointer, const size_t size)¶

Create the blob from a buffer from the client-side allocator.

Parameters:

object_id – The object ID of this blob.
pointer – The address of buffer in the client-side allocator.
size – The estimated size of the buffer.

static std::shared_ptr<Blob> FromPointer(Client &client, const uintptr_t pointer, const size_t size)¶

Create the blob from a given buffer. If the buffer already lies in the vineyardd, it would return immediately without copying, otherwise a blob writer will be created and the content of the buffer will be copied into.

Parameters:

pointer – The address of the buffer.
size – The estimated size of the buffer.

class BlobWriter : public vineyard::ObjectBuilder ¶

The writer to write a blob in vineyard.

The writer is initialized in the client with a local buffer and its size, and a blob in vineyard will be created when Build is invoked.

Public Functions

ObjectID id() const¶

Return the object id of this blob builder. Note that before sealing the blob builder the object id cannot be used to get “Blob” objects.

Returns:: The ObjectID of the blob writer.

size_t size() const¶

Get the size of the blob, i.e., the number of bytes of the data payload in the blob.

Returns:: The size of the blob.

char *data()¶

Get the data pointer of the data payload in the blob.

Returns:: The data pointer.

const char *data() const¶

Get the const data pointer of the data payload in the blob.

Returns:: The const data pointer.

const std::shared_ptr<MutableBuffer> &Buffer() const¶

Get the mutable buffer of the blob.

Returns:: The mutable buffer of the blob, which can be modified to update the content in the blob.

virtual Status Build(Client &client) override¶

Build a blob in vineyard server.

Parameters:: client – The client connected to the vineyard server.

Status Abort(Client &client)¶

Abort the blob builder.

Parameters:: client – Release the blob builder object if it is not sealed.

Status Shrink(Client &client, const size_t size)¶: Shrink the blob builder to the given size without reallocating and copying.

void AddKeyValue(std::string const &key, std::string const &value)¶

Add key-value metadata for the blob.

Parameters:

key – The key of the metadata.
value – The value of the metadata.

void AddKeyValue(std::string const &key, std::string &&value)¶

Add key-value metadata for the blob.

Parameters:

key – The key of the metadata.
value – The value of the metadata.

void Dump() const¶: Dump the buffer for debugging.

Stream¶

class ByteStream : public vineyard::BareRegistered<ByteStream>, public vineyard::Stream<Blob>¶

Public Functions

inline void SetBufferSizeLimit(size_t limit)¶

Status WriteBytes(const char *ptr, size_t len)¶

Status WriteLine(const std::string &line)¶

Status FlushBuffer()¶

Status ReadLine(std::string &line)¶

Public Static Functions

static inline std::unique_ptr<Object> Create()¶

Basic Data Types¶

template<typename T> class Array : public vineyard::Registered<Array<T>>¶

The array type in vineyard.

Template Parameters:: T – The type for the elements.

Public Functions

inline virtual void Construct(const ObjectMeta &meta) override¶

Construct an object from metadata. The metadata meta should come from client’s GetMetaData method.

The implementation of Construct method is usually boilerplate. Vineyard provides a code generator to help developers code their own data structures and can be shared via vineyard.

Parameters:: meta – The metadata that be used to construct the object.

inline const T &operator[](size_t loc) const¶

Get the element at the given location.

Parameters:: loc – The given location to get the element.

inline size_t size() const¶

Get the size of the array.

Returns:: The size.

inline const T *data() const¶

Get the pointer to the beginning of the data buffer.

Parameters:: The – pointer to the data buffer

Public Static Functions

static inline std::unique_ptr<Object> Create()¶

template<typename T> class ArrayBuilder : public vineyard::ArrayBaseBuilder<T>¶

ArrayBuilder is designed for constructing arrays that supported by vineyard.

Template Parameters:: T – The type for the elements.

Public Functions

inline ArrayBuilder(Client &client, size_t size)¶

inline ArrayBuilder(Client &client, std::vector<T> const &vec)¶

Initialize the ArrayBuilder from an existing std::vector of type T.

Parameters:

client – The client connected to the vineyard server.
vec – The local std::vector of type T.

inline ArrayBuilder(Client &client, const T *data, size_t size)¶

Initialize the ArrayBuilder from an existing C array of type T.

Parameters:

client – The client connected to the vineyard server.
data – The pointer to the array.
size – The size of the array.

inline ~ArrayBuilder()¶

inline const size_t size() const¶

Get the size of the array, i.e., number of elements in the array.

Returns:: The size.

inline T &operator[](size_t idx)¶

Get the element located in the given index of the array.

Parameters:: idx – The give index.
Returns:: The element at the given index.

inline T *data() noexcept¶

Get the data pointer to the array.

Returns:: The data pointer.

inline const T *data() const noexcept¶

Get the const data pointer to the array.

Returns:: The const data pointer.

inline Status Build(Client &client) override¶

Build the array object.

Parameters:: client – The client connected to the vineyard server.

template<typename K, typename V, typename H = prime_number_hash_wy<K>, typename E = std::equal_to<K>> class Hashmap : public vineyard::Registered<Hashmap<K, V, prime_number_hash_wy<K>, std::equal_to<K>>>, public vineyard::prime_number_hash_wy<K>, public std::equal_to<K>¶

The hash map in vineyard.

Template Parameters:

K – The type for the key.
V – The type for the value.
std::hash<K> – The hash function for the key.
std::equal_to<K> – The compare function for the key.

Public Types

using KeyHash = H ¶

using KeyEqual = E ¶

using T = std::pair<K, V>¶

using Entry = ska::detailv3::sherwood_v3_entry<T>¶

using EntryPointer = const Entry*¶

using Hasher = ska::detailv3::KeyOrValueHasher<K, std::pair<K, V>, H>¶

using Equal = ska::detailv3::KeyOrValueEquality<K, std::pair<K, V>, E>¶

using value_type = T ¶

using size_type = size_t¶

using difference_type = std::ptrdiff_t¶

using hasher = H ¶

using key_equal = E ¶

using reference = value_type&¶

using const_reference = const value_type&¶

using pointer = value_type*¶

using const_pointer = value_type*¶

using flat_hash_table_type = ska::detailv3::sherwood_v3_table<T, K, H, Hasher, E, Equal, std::allocator<T>, typename std::allocator_traits<std::allocator<T>>::template rebind_alloc<ska::detailv3::sherwood_v3_entry<T>>>¶

Public Functions

inline virtual void Construct(const ObjectMeta &meta) override¶

Construct an object from metadata. The metadata meta should come from client’s GetMetaData method.

The implementation of Construct method is usually boilerplate. Vineyard provides a code generator to help developers code their own data structures and can be shared via vineyard.

Parameters:: meta – The metadata that be used to construct the object.

inline virtual void PostConstruct(const ObjectMeta &meta) override¶: Set the hash policy after the construction of the HashMap.

inline iterator begin() const¶: The beginning iterator.

inline iterator end() const¶: The ending iterator.

inline iterator find(const K &key)¶: Find the iterator by key.

inline const iterator find(const K &key) const¶: Return the const iterator by key.

inline size_t count(const K &key) const¶: Return the number of occurancies of the key.

inline size_t size() const¶: Return the size of the HashMap, i.e., the number of elements stored in the HashMap.

inline size_t bucket_count() const¶: Return the max size of the HashMap, i.e., the number of allocated cells for elements stored in the HashMap.

inline float load_factor() const¶: Return the load factor of the HashMap.

inline bool empty() const¶: Check whether the HashMap is empty.

inline const V &at(const K &key) const¶: Get the value by key. Here the existence of the key is checked.

Public Static Functions

static inline std::unique_ptr<Object> Create()¶

struct iterator¶

The iterator to iterate key-value mappings in the HashMap.

Public Functions

iterator() = default¶

inline explicit iterator(EntryPointer current)¶

inline iterator &operator++()¶

inline iterator operator++(int)¶

inline const value_type &operator*() const¶

inline const value_type *operator->() const¶

Public Members

EntryPointer current = EntryPointer()¶

Friends

inline friend bool operator==(const iterator &lhs, const iterator &rhs)¶

inline friend bool operator!=(const iterator &lhs, const iterator &rhs)¶

template<typename K, typename V, typename H = prime_number_hash_wy<K>, typename E = std::equal_to<K>> class HashmapBuilder : public vineyard::HashmapBaseBuilder<K, V, prime_number_hash_wy<K>, std::equal_to<K>>¶

HashmapBuilder is used for constructing hashmaps that supported by vineyard.

Template Parameters:

K – The type for the key.
V – The type for the value.
std::hash<K> – The hash function for the key.
std::equal_to<K> – The compare function for the key.

Public Functions

inline explicit HashmapBuilder(Client &client)¶

inline HashmapBuilder(Client &client, ska::flat_hash_map<K, V, H, E> &&hashmap)¶

inline V &operator[](const K &key)¶: Get the mapping value of the given key.

inline V &operator[](K &&key)¶: Get the mapping value of the given key.

template<class ...Args> inline bool emplace(Args&&... args)¶: Emplace key-value pair into the hashmap.

inline V &at(const K &key)¶: Get the mapping value of the given key.

inline const V &at(const K &key) const¶: Get the const mapping value of the given key.

inline size_t size() const¶: Get the size of the hashmap.

inline void reserve(size_t size)¶: Reserve the size for the hashmap.

inline size_t bucket_count() const¶: Return the maximum possible size of the HashMap, i.e., the number of elements that can be stored in the HashMap.

inline float load_factor() const¶: Return the load factor of the HashMap.

inline bool empty() const¶: Check whether the hashmap is empty.

inline ska::flat_hash_map<K, V, H, E>::iterator begin()¶: Return the beginning iterator.

inline ska::flat_hash_map<K, V, H, E>::const_iterator begin() const¶: Return the const beginning iterator.

inline ska::flat_hash_map<K, V, H, E>::const_iterator cbegin() const¶: Return the const beginning iterator.

inline ska::flat_hash_map<K, V, H, E>::iterator end()¶: Return the ending iterator.

inline ska::flat_hash_map<K, V, H, E>::const_iterator end() const¶: Return the const ending iterator.

inline ska::flat_hash_map<K, V, H, E>::const_iterator cend() const¶: Return the const ending iterator.

inline ska::flat_hash_map<K, V, H, E>::iterator find(const K &key)¶: Find the value by key.

inline void AssociateDataBuffer(std::shared_ptr<Blob> data_buffer)¶: Associated with a given data buffer.

inline Status Build(Client &client) override¶: Build the hashmap object.

template<typename T> class Tensor : public vineyard::ITensor, public vineyard::BareRegistered<Tensor<T>>¶

Public Types

using value_t = T ¶

using value_pointer_t = T*¶

using value_const_pointer_t = const T*¶

using ArrowTensorT = typename detail::ArrowTensorType<T>::type¶

Public Functions

inline virtual void Construct(const ObjectMeta &meta) override¶

Construct an object from metadata. The metadata meta should come from client’s GetMetaData method.

The implementation of Construct method is usually boilerplate. Vineyard provides a code generator to help developers code their own data structures and can be shared via vineyard.

Parameters:: meta – The metadata that be used to construct the object.

inline std::vector<int64_t> strides() const¶

Get the strides of the tensor.

Returns:: The strides of the tensor. The definition of the tensor’s strides can be found in https://pytorch.org/docs/stable/tensor_attributes.html

inline virtual std::vector<int64_t> const &shape() const override¶

Get the shape of the tensor.

Returns:: The shape vector where the ith element represents the size of the ith axis.

inline virtual std::vector<int64_t> const &partition_index() const override¶

Get the index of this partition in the global tensor.

Returns:: The index vector where the ith element represents the index in the ith axis.

inline virtual AnyType value_type() const override¶

Get the type of tensor’s elements.

Returns:: The type of the tensor’s elements.

inline value_const_pointer_t data() const¶

Get the data pointer to the tensor’s data buffer.

Returns:: The data pointer.

inline const value_t operator[](size_t index) const¶

Get the data in the tensor by index.

Returns:: The data reference.

inline virtual const std::shared_ptr<arrow::Buffer> buffer() const override¶

Get the buffer of the tensor.

Returns:: The shared pointer to an arrow buffer which holds the data buffer of the tensor.

inline virtual const std::shared_ptr<arrow::Buffer> auxiliary_buffer() const override¶

inline const std::shared_ptr<ArrowTensorT> ArrowTensor()¶: Return a view of the original tensor so that it can be used as arrow’s Tensor.

Public Static Functions

static inline std::unique_ptr<Object> Create()¶

template<typename T> class TensorBuilder : public vineyard::ITensorBuilder, public vineyard::TensorBaseBuilder<T>¶

TensorBuilder is used for building tensors that supported by vineyard.

Template Parameters:: T –

Public Types

using value_t = T ¶

using value_pointer_t = T*¶

using value_const_pointer_t = const T*¶

Public Functions

inline TensorBuilder(Client &client, std::vector<int64_t> const &shape)¶

Initialize the TensorBuilder with the tensor shape.

Parameters:

client – The client connected to the vineyard server.
shape – The shape of the tensor.

inline TensorBuilder(Client &client, std::vector<int64_t> const &shape, std::vector<int64_t> const &partition_index)¶

Initialize the TensorBuilder for a partition of a GlobalTensor.

Parameters:

client – The client connected to the vineyard server.
shape – The shape of the partition.
partition_index – The partition index in the global tensor.

inline std::vector<int64_t> const &shape() const¶

Get the shape of the tensor.

Returns:: The shape vector where the ith element represents the size of the ith axis.

inline std::vector<int64_t> const &partition_index() const¶

Get the index of this partition in the global tensor.

Returns:: The index vector where the ith element represents the index in the ith axis.

inline void set_shape(std::vector<int64_t> const &shape)¶

Set the shape of the tensor.

Parameters:: shape – The vector for the shape, where the ith element represents the size of the shape in the ith axis.

inline void set_partition_index(std::vector<int64_t> const &partition_index)¶

Set the index in the global tensor.

Parameters:: partition_index – The vector of indices, where the ith element represents the index in the ith axis.

inline std::vector<int64_t> strides() const¶

Get the strides of the tensor.

Returns:: The strides of the tensor. The definition of the tensor’s strides can be found in https://pytorch.org/docs/stable/tensor_attributes.html

inline value_pointer_t data() const¶: Get the data pointer of the tensor.

inline Status Build(Client &client) override¶

Build the tensor.

Parameters:: client – The client connceted to the vineyard server.

class DataFrame : public vineyard::Registered<DataFrame>¶

Public Functions

inline virtual void Construct(const ObjectMeta &meta) override¶

Construct an object from metadata. The metadata meta should come from client’s GetMetaData method.

The implementation of Construct method is usually boilerplate. Vineyard provides a code generator to help developers code their own data structures and can be shared via vineyard.

Parameters:: meta – The metadata that be used to construct the object.

const std::vector<json> &Columns() const¶

Get the column names.

Returns:: The vector of column names.

std::shared_ptr<ITensor> Index() const¶

Get the index of dataframe.

Returns:: The shared pointer to the index tensor.

std::shared_ptr<ITensor> Column(json const &column) const¶

Get the column of the given column name.

Parameters:: column – The given column name.
Returns:: The shared pointer to the column tensor.

const std::pair<size_t, size_t> partition_index() const¶

Get the partition index of the global dataframe.

Returns:: The pair of the partition_index on rows and the partition_index on columns.

const std::pair<size_t, size_t> shape() const¶

Get the shape of the dataframe.

Returns:: The pair of the number of rows and the number of columns.

const std::shared_ptr<arrow::RecordBatch> AsBatch(bool copy = false) const¶: Get a RecordBatch view for the dataframe.

Public Static Functions

static inline std::unique_ptr<Object> Create()¶

class DataFrameBuilder : public vineyard::DataFrameBaseBuilder¶

DataFrameBuilder is used for constructing dataframes that supported by vineyard.

Public Functions

explicit DataFrameBuilder(Client &client)¶

const std::pair<size_t, size_t> partition_index() const¶

Get the partition index of the global dataframe.

Returns:: The pair of the partition_index on rows and the partition_index on columns.

void set_partition_index(size_t partition_index_row, size_t partition_index_column)¶

Set the index in the global dataframe.

Parameters:

partition_index_row – The row index.
partition_index_column – The column index.

void set_row_batch_index(size_t row_batch_index)¶

Set the row batch index in the global dataframe. Note that the row batch index gives the order of batches on rows.

Parameters:: row_batch_index – The row batch index.

void set_index(std::shared_ptr<ITensorBuilder> builder)¶

Set the index of dataframe by add a index column to dataframe.

Parameters:: builder – The index tensor builder.

std::shared_ptr<ITensorBuilder> Column(json const &column) const¶

Get the column of the given column name.

Parameters:: column – The given column name.
Returns:: The shared pointer to the column tensor.

void AddColumn(json const &column, std::shared_ptr<ITensorBuilder> builder)¶

Add a column to the dataframe by registering a tensor builder to the column name. When the dataframe is built, the tensor builder will be employed to build the column.

Parameters:

column – The name of the column.
builder – The tensor builder for the column.

void DropColumn(json const &column)¶

Drop the column with the given column name.

Parameters:: column – The name of column to be dropped.

virtual Status Build(Client &client) override¶

Build the dataframe object.

Parameters:: client – The client connected to the vineyard server.

class Sequence : public vineyard::Registered<Sequence>¶

The sequence type in vineyard.

Public Functions

inline virtual void Construct(const ObjectMeta &meta) override¶

Construct an object from metadata. The metadata meta should come from client’s GetMetaData method.

The implementation of Construct method is usually boilerplate. Vineyard provides a code generator to help developers code their own data structures and can be shared via vineyard.

Parameters:: meta – The metadata that be used to construct the object.

inline const size_t Size() const¶

Get the size of the sequence, i.e., the number of elements it contains.

Returns:: The size of the sequence.

inline const std::shared_ptr<Object> At(size_t index) const¶

Get the value at the given index.

Parameters:: index – The given index to get the value.

inline const std::shared_ptr<Object> First() const¶

Get the first element of the pair.

Returns:: The shared pointer to the first object.

inline const std::shared_ptr<Object> Second() const¶

Get the second element of the pair.

Returns:: The shared pointer to the second object.

inline const iterator begin() const¶

Get the beginning iterator.

Returns:: The beginning iterator.

inline const iterator end() const¶

Get the ending iterator.

Returns:: The ending iterator.

Public Static Functions

static inline std::unique_ptr<Object> Create()¶

class iterator : public std::iterator<std::bidirectional_iterator_tag, std::shared_ptr<Object>, size_t, const std::shared_ptr<Object>*, std::shared_ptr<Object>>¶

The iterator for the sequence object to iterate from the first to the last element.

Public Functions

inline explicit iterator(Sequence const *sequence, size_t index = 0)¶

inline iterator &operator++()¶

inline bool operator==(iterator other) const¶

inline bool operator!=(iterator other) const¶

inline reference operator*() const¶

class SequenceBuilder : public vineyard::SequenceBaseBuilder¶

SequenceBuilder is designed for generating sequences.

Public Functions

inline explicit SequenceBuilder(Client &client)¶

inline explicit SequenceBuilder(Client &client, size_t const size)¶

Initialize the SequenceBuilder with a given size.

Parameters:

client – The client connected to the vineyard server.
size – The size of the sequence to build.

inline const size_t Size() const¶

Get the size of the sequence, i.e., the number of elements it contains.

Returns:: The size of the sequence.

inline void SetSize(size_t size)¶

Set the size for the sequence. Note that the size of a sequence can be set only once.

Parameters:: size – The size for the sequence.

inline std::shared_ptr<ObjectBuilder> At(size_t index)¶

Get the builder at the given index. Here the index is bound-checked.

Parameters:: index – The given index.
Returns:: The builder at the given index.

inline void SetValue(size_t idx, std::shared_ptr<ObjectBuilder> const &value)¶

Set the builder for the value at the given index. When building the sequence, the builder will be invoked to build the value.

Parameters:

idx – The index of the value.
value – The builder for the value for the given index.

inline void SetValue(size_t idx, std::shared_ptr<Object> const &value)¶

Set the builder for the value at the given index. When building the sequence, the builder will be invoked to build the value.

Parameters:

idx – The index of the value.
value – The value for the given index.

template<typename T> class Scalar : public vineyard::Registered<Scalar<T>>¶

The scalar type in vineyard. Note that the value of the scalar is writing into the meta tree as a string directly, instead of storing the value in the shared memory with a blob.

Template Parameters:: T – The type for the value.

Public Functions

inline virtual void Construct(const ObjectMeta &meta) override¶

Construct an object from metadata. The metadata meta should come from client’s GetMetaData method.

The implementation of Construct method is usually boilerplate. Vineyard provides a code generator to help developers code their own data structures and can be shared via vineyard.

Parameters:: meta – The metadata that be used to construct the object.

inline const T Value() const¶

Get the value of the scalar.

Returns:: The value of the scalar.

inline AnyType Type() const¶

Get the type of the scalar.

Returns:: The type of the scalar.

Public Static Functions

static inline std::unique_ptr<Object> Create()¶

template<typename T> class ScalarBuilder : public vineyard::ScalarBaseBuilder<T>¶

ScalarBuilder is used for building scalars that supported by vineyard.

Template Parameters:: T – The type for the scalar.

Public Functions

inline explicit ScalarBuilder(Client &client)¶

inline explicit ScalarBuilder(Client &client, T const &value)¶

Initialize the scalar with the value.

Parameters:

client – The client connected to the vineyard server.
value – The value for the scalar.

inline void SetValue(T const &value)¶

Set the value of the scalar.

Parameters:: value – The value for the scalar.

Distributed Data Types¶

class GlobalTensor : public vineyard::BareRegistered<GlobalTensor>, public vineyard::Collection<ITensor>¶

GlobalTensor is a holder for a set of tensor chunks that are distributed over many vineyard nodes.

Public Functions

void PostConstruct(const ObjectMeta &meta) override¶

std::vector<int64_t> const &shape() const¶

std::vector<int64_t> const &partition_shape() const¶

const std::vector<std::shared_ptr<ITensor>> LocalPartitions(Client &client) const¶: backwards compatibility

Public Static Functions

static inline std::unique_ptr<Object> Create()¶

class GlobalTensorBuilder : public vineyard::CollectionBuilder<ITensor>¶

GlobalTensorBuilder is designed for building global tensors.

Public Functions

inline explicit GlobalTensorBuilder(Client &client)¶

std::vector<int64_t> const &partition_shape() const¶: Get the partition shape of the global tensor. Here the ith element represents how many partitions are made on the ith axis.

void set_partition_shape(std::vector<int64_t> const &partition_shape)¶: Set the partition shape of the global tensor. Here the ith element represents how many partitions are made on the ith axis.

std::vector<int64_t> const &shape() const¶: Get the entire shape of the global tensor.

void set_shape(std::vector<int64_t> const &shape)¶: Set the entire shape of the global tensor.

void AddPartition(const ObjectID partition_id)¶: Backwards compatibility.

void AddPartitions(const std::vector<ObjectID> &partition_ids)¶

class GlobalDataFrame : public vineyard::BareRegistered<GlobalDataFrame>, public vineyard::Collection<DataFrame>¶

GlobalDataFrame is a DataFrame that refers a set of dataframe chunks in many vineyardd nodes.

Public Functions

void PostConstruct(const ObjectMeta &meta) override¶

const std::pair<size_t, size_t> partition_shape() const¶

Set the partition shape of the global dataframe.

Parameters:

partition_shape_row – The number of partitions on rows.
partition_shape_column – The number of partitions on columns.

const std::vector<std::shared_ptr<DataFrame>> LocalPartitions(Client &client) const¶: backwards compatibility

Public Static Functions

static inline std::unique_ptr<Object> Create()¶

class GlobalDataFrameBuilder : public vineyard::CollectionBuilder<DataFrame>¶

GlobalDataFrameBuilder is designed for building global dataframes.

Public Functions

inline explicit GlobalDataFrameBuilder(Client &client)¶

const std::pair<size_t, size_t> partition_shape() const¶

Get the partition shape of the global dataframe.

Returns:: The pair of <number_of_partitions_on_rows, number_of_partitions_on_columns>.

void set_partition_shape(const size_t partition_shape_row, const size_t partition_shape_column)¶

Set the partition shape of the global dataframe.

Parameters:

partition_shape_row – The number of partitions on rows.
partition_shape_column – The number of partitions on columns.

void AddPartition(const ObjectID partition_id)¶: Backwards compatibility.

void AddPartitions(const std::vector<ObjectID> &partition_ids)¶