Python API Reference#

Objects#

class vineyard.ObjectID#

Opaque type for vineyard’s object id. The object ID is generated by vineyard server, the underlying type of ObjectID is a 64-bit unsigned integer. Wrapper utilities are provided to interact with the external python world.

>>> id = ObjectID("000043c5c6d5e646")
>>> id
000043c5c6d5e646
>>> repr(id)
'000043c5c6d5e646'
>>> print(id)
ObjectID <"000043c5c6d5e646">
>>> int(id)
74516723525190

__eq__()#: Return self==value.

__hash__()#: Return hash(self).

__init__()#

__repr__()#: Return repr(self).

__str__()#: Return str(self).

class vineyard.Object#

Base class for vineyard objects.

property id#: The object id of this object.

property isglobal#: Whether the object is a global object.

property islocal#: Whether the object is a local object.

property ispersist#: Whether the object is a persistent object. The word “persistent” means the object could be seen by clients that connect to other vineyard server instances.

member()#

member(self, name: str) → Object

Get the member object of this object.

Parameters: name – str The name of the member object.
Returns: The member object.
Return type: Object

See also

ObjectMeta.get, ObjectMeta.__getitem__

property meta#: The metadata of this object.

property nbytes#: The nbytes of this object.

property signature#: The object signature of this object.

property typename#: The typename of this object. typename is the string value of the C++ type, e.g., vineyard::Array<int>, vineyard::Table.

class vineyard.ObjectBuilder#: Base class for vineyard object builders.

Metadata#

class vineyard.ObjectMeta#

ObjectMeta is the type for metadata of an Object. The ObjectMeta can be treat as a dict-like type. If the the metadata if the metadata obtained from vineyard, the metadata is readonly. Otherwise key-value attributes or object members could be associated with the metadata to construct a new vineyard object.

We can inspect the key-value attributes and members of an ObjectMeta:

>>> meta = client.get_meta(hashmap_id)
>>> meta
ObjectMeta {
    "id": "0000347aebe92dd0",
    "instance_id": "0",
    ...
}
>>> meta['num_elements_']
'5'
>>> meta['entries']
ObjectMeta {
    "id": "0000347aebe92dd0",
    "instance_id": "0",
    ...
}

ObjectMeta value can be iterated over:

>>> list(k for k in meta['entries'])
['transient', 'num_slots_minus_one_', 'max_lookups_', 'num_elements_', 'entries_',
 'nbytes', 'typename', 'instance_id', 'id']

__contains__()#

Check if given key exists in the object metadata.

Parameters: key – str The name to be looked up.
Returns: True if the queried key exists in this object metadata, otherwise False.
Return type: bool

__getitem__()#

__getitem__(self, key: str) → string or Object

Get meta or member’s meta from metadata.

Parameters

key – str The name to be looked up.

Returns

string – If the given key is a key of meta, returns the meta value.
Object – If the given key is a key of member, return the meta of this member.

__init__()#

__init__(global_: bool = False)

Create an empty metadata, the metadata will be used to create a vineyard object.

Parameters: global_: bool, if the object meta is for creating a global object.

__repr__()#: Return repr(self).

__setitem__()#

__setitem__(self, key: str, value) → None

Add a metadata to the ObjectMeta.

Parameters

key – str The name of the new metadata entry.
value –
str, int, float, bool or list of int The value of the new metadata entry.
- When the value is a str, it will be convert to string at first.
- When the value is a list of str, int or float, it will be first dumpped as string using json.dumps.

__setitem__(self, key: str, ObjectID, Object or ObjectMeta) → None

Add a member object.

Parameters

key – str The name of the member object.
object – Object, ObjectID or ObjectMeta The reference to the member object or the object id of the member object.

__str__()#: Return str(self).

add_member()#

add_member(self, key: str, ObjectID, Object or ObjectMeta) → None

Add a member object.

Parameters

key – str The name of the member object.
object – Object, ObjectID or ObjectMeta The reference to the member object or the object id of the member object.

get()#

get(self, key: str, default=None) → string or Object

Get meta or member’s meta from metadata, return default value if the given key is not presented.

Parameters

key – str The name to be looked up.

Returns

str – When the given key belongs to a metadata pair. Note that the metadata value of type int or float will be returned in string format as well.
ObjectMeta – When the given key is mapped to a member object.
See Also – ObjectMeta.__getitem__

get_member()#

get_member(self, key: str) → Object

Get member object from metadata, return None if the given key is not presented, and raise exception RuntimeError if the given key is associated with a plain metadata, rather than member object.

Parameters: key – str The name to be looked up.
Raises: RuntimeError – When the given key is associated with a plain metadata, rather than member object.

See also

ObjectMeta.__getitem__, ObjectMeta.get

property id#: The corresponding object ID of this metadata.

property instance_id#: The "instance_id" of vineyard instance that the metadata been placed on.

property isglobal#: True if the object is a global object, otherwise a local object or remote object.

property islocal#: True if the object is a local object, otherwise a global object or remote object.

property memory_usage#: Get the total memory usage of buffers in this object meta.

property nbytes#: The "nbytes" attribute of this metadata.

set_global()#

Mark the building object as a global object.

Parameters: global – bool, default is True

property signature#: The corresponding object signature of this metadata.

property typename#: The "typename" attribute of this metadata.

Vineyard client#

vineyard.connect()#

vineyard.connect(endpoint: str) → IPCClient

Connect to vineyard via UNIX domain socket for IPC service:

client = vineyard.connect('/var/run/vineyard.sock')

Parameters:

endpoint: str: UNIX domain socket path to setup an IPC connection.

Returns:

IPCClient: The connected IPC client.

vineyard.connect(host: str, port: int or str) → RPCClient

Connect to vineyard via TCP socket.

Parameters:

host: str: Hostname to connect to.
port: int or str: The TCP that listened by vineyard TCP service.

Returns:

RPCClient: The connected RPC client.

vineyard.connect(endpoint: (str, int or str)) → RPCClient

Connect to vineyard via TCP socket.

Parameters:

endpoint: tuple(str, int or str): Endpoint to connect to. The parameter is a tuple, in which the first element is the host, and the second parameter, can be int a str, is the port.

Returns:

RPCClient: The connected RPC client.

vineyard.connect() → IPCClient or RPCClient

Connect to vineyard via UNIX domain socket or TCP endpoint. This method normally usually no arguments, and will first tries to resolve IPC socket from the environment variable VINEYARD_IPC_SOCKET and connect to it. If it fails to establish a connection with vineyard server, the method will tries to resolve RPC endpoint from the environment variable VINEYARD_RPC_ENDPOINT.

If both tries are failed, this method will raise a ConnectionFailed exception.

In rare cases, user may be not sure about if the IPC socket or RPC endpoint is available, i.e., the variable might be None. In such cases this method can accept a None as arguments, and do resolution as described above.

Raises:: ConnectionFailed

class vineyard.IPCClient#

IPC client that connects to vineyard instance’s UNIX domain socket.

allocated_size()#

allocated_size(target: Object or ObjectID) → int

Get the allocated size of the given object.

Parameters: target – Object or ObjectID The given Object.
Returns: int

clear()#

clear() → None

Drop all objects that visible to the current instance in the vineyard cluster.

close()#: Close the client.

property connected#: Whether the client instance has been connected to the vineyard server.

create_blob()#

create_blob(size: int) → Blob

Allocate a blob in vineyard server.

Parameters: size – int The size of blob that will be allocated on vineyardd.
Returns: BlobBuilder

create_empty_blob()#

create_empty_blob() → Blob

Create an empty blob in vineyard server.

Returns: Blob

create_metadata()#

create_metadata(metadata: ObjectMeta) → ObjectMeta

Create metadata in vineyardd.

Parameters: metadata – ObjectMeta The metadata that will be created on vineyardd.
Returns: The result created metadata.

create_metadata(metadata: ObjectMeta, instance_id: InstanceID) → ObjectMeta

Create metadata in vineyardd with a specified instance id.

Parameters

metadata – ObjectMeta The metadata that will be created on vineyardd.
instance_id – InstanceID The instance to place the object metadata.

Returns

The result created metadata.

delete()#

delete(object_id: ObjectID or List[ObjectID], force: bool = false,
deep: bool = true) -> None

Delete the specific vineyard object.

Parameters

object_id – ObjectID or list of ObjectID Objects that will be deleted. The object_id can be a single ObjectID, or a list of ObjectID.
force – bool Forcedly delete an object means the member will be recursively deleted even if the member object is also referred by others. The default value is True.
deep –
bool Deeply delete an object means we will deleting the members recursively. The default value is True.

Note that when deleting objects which have direct blob members, the processing on those blobs yields a “deep” behavior.

delete(object_meta: ObjectMeta, force: bool = false, deep: bool = true)
-> None

Delete the specific vineyard object.

Parameters: object_meta – The corresponding object meta to delete.

delete(object: Object, force: bool = false, deep: bool = true) → None

Delete the specific vineyard object.

Parameters: object – The corresponding object meta to delete.

drop_name()#

drop_name(name: str or ObjectName) → None

Remove the association of the given name.

Parameters: name – str The name that will be removed.

exists()#

exists(object_id: ObjectID) → bool

Whether the given object exists.

Parameters: object_id – ObjectID The object id to check if exists.
Returns: True when the specified object exists.
Return type: bool

find_shared_memory()#

find_shared_memory(target: ptr) → bool

Find the corresponding blob (if exists) of the given pointer.

Parameters: target – address, in int format The given address.
Returns: ObjectID

get(object_id, resolver=None, **kw)#

Get vineyard object as python value.

>>> arr_id = vineyard.ObjectID('00002ec13bc81226')
>>> arr = client.get(arr_id)
>>> arr
array([0, 1, 2, 3, 4, 5, 6, 7])

Parameters

client – IPCClient or RPCClient The vineyard client to use.
object_id – ObjectID The object id that will be obtained from vineyard.
resolver – When retrieving vineyard object, an optional resolver can be specified. If no resolver given, the default resolver context will be used.
kw – User-specific argument that will be passed to the builder.

Returns

A python object that return by the resolver, by resolving an vineyard object.

get_blob()#

get_blob(object_id: ObjectID, unsafe: bool = false) → Blob

Getting a blob from vineyard using the given object ID.

Parameters

object_id – ObjectID The blob to get.
unsafe – bool unsafe means getting the blob even the blob is not sealed yet. Default is False.

Returns

Blob

See also

IPCClient.get_blob RPCClient.get_remote_blob RPCClient.get_remote_blobs

get_blobs()#

get_blobs(object_ids: List[ObjectID], unsafe: bool = false) → List[Blob]

Getting blobs from vineyard using the given object IDs.

Parameters

object_ids – List[ObjectID] The blobs to get.
unsafe – bool unsafe means getting the blob even the blob is not sealed yet. Default is False.

Returns

List[Blob]

See also

IPCClient.get_blobs RPCClient.get_remote_blob RPCClient.get_remote_blobs

get_meta()#

get_meta(object_id: ObjectID, sync_remote: bool = False) → ObjectMeta

Get object metadata from vineyard.

Parameters

object_id – ObjectID The object id to get.
sync_remote – bool If the target object is a remote object, code_remote=True will force a meta synchronization on the vineyard server. Default is False.

Returns

ObjectMeta

get_metas()#

get_metas(object_ids: List[ObjectID], sync_remote: bool = False)
-> List[ObjectMeta]

Get metadatas of multiple objects from vineyard.

Parameters

object_ids – List[ObjectID] The object ids to get.
sync_remote – bool If the target object is a remote object, code_remote=True will force a meta synchronization on the vineyard server. Default is False.

Returns

List[ObjectMeta]

get_name()#

get_name(name: str or ObjectName, wait: bool = False) → ObjectID

Get the associated object id of the given name.

Parameters

name – str The name that will be queried.
wait – bool Whether to wait util the name appears, if wait, the request will be blocked until the name been registered.

Returns

The associated object id with the name.

Return type

ObjectID

get_object()#

get_object(object_id: ObjectID) → Object

Get object from vineyard.

Parameters: object_id – ObjectID The object id to get.
Returns: Object

get_objects()#

get_objects(object_ids: List[ObjectID]) → List[Object]

Get multiple objects from vineyard.

Parameters: object_ids – List[ObjectID]
Returns: List[Object]

property instance_id#: The instance id of the connected vineyard server.

property ipc_socket#: The UNIX domain socket location of connected vineyard server.

is_shared_memory()#

is_shared_memory(target: ptr) → bool

Check if the address is on the shared memory region.

Parameters: target – address, in int format The given address.
Returns: bool

list_metadatas()#

list_metadatas(pattern: str, regex: bool = False, limit: int = 5,
nobuffer: bool = False) -> List[Object]

List all objects in current vineyard server.

Parameters

pattern – str The pattern string that will be matched against the object’s typename.
regex – bool Whether the pattern is a regex expression, otherwise the pattern will be used as wildcard pattern. Default value is False.
limit – int The limit to list. Default value is 5.
nobuffer – bool Whether to fill the buffers in returned object metadatas. Default value is False.

Returns

List[ObjectMeta]

list_names()#

list_names(pattern: str, regex: bool = False, limit: int = 5)
-> Dict[str, ObjectID]

List all names in current vineyard server.

Parameters

pattern – str The pattern string that will be matched against the name.
regex – bool Whether the pattern is a regex expression, otherwise the pattern will be used as wildcard pattern. Default value is False.
limit – int The limit to list. Default value is 5.

Returns

Dict[str, ObjectID]

list_objects()#

list_objects(pattern: str, regex: bool = False, limit: int = 5)
-> List[Object]

List all objects in current vineyard server.

Parameters

pattern – str The pattern string that will be matched against the object’s typename.
regex – bool Whether the pattern is a regex expression, otherwise the pattern will be used as wildcard pattern. Default value is False.
limit – int The limit to list. Default value is 5.

Returns

List[Object]

property meta#

The metadata information of the vineyard server. The value is a nested dict, the first-level key is the instance id, and the second-level key is the cluster metadata fields.

>>> client.meta
{
    14: {
        'hostid': '54058007061210',
        'hostname': '127.0.0.1',
        'timestamp': '6882550126788354072'
    },
    15: {
        'hostid': '48843417291806',
        'hostname': '127.0.0.1',
        'timestamp': '6882568290204737414'
    }
}

persist()#

persist(object_id: ObjectID) → None

Persist the object of the given object id. After persisting, the object will be visible by clients that connect to other vineyard server instances.

Parameters: object_id – ObjectID The object that will be persist.

persist(object_meta: ObjectMeta) → None

Persist the given object.

Parameters: object_meta – ObjectMeta The object that will be persist.

persist(object: Object) → None

Persist the given object.

Parameters: object – Object The object that will be persist.

put(value, builder=None, **kw)#

Put python value to vineyard.

>>> arr = np.arange(8)
>>> arr_id = client.put(arr)
>>> arr_id
00002ec13bc81226

Parameters

client – IPCClient The vineyard client to use.
value – The python value that will be put to vineyard. Supported python value types are decided by modules that registered to vineyard. By default, python value can be put to vineyard after serialized as a bytes buffer using pickle.
builder – When putting python value to vineyard, an optional builder can be specified to tell vineyard how to construct the corresponding vineyard Object. If not specified, the default builder context will be used to select a proper builder.
kw – User-specific argument that will be passed to the builder.

Returns

The result object id will be returned.

Return type

ObjectID

put_name()#

put_name(object: ObjectID or ObjectMeta or Object,
name: str or ObjectName) -> None

Associate the given object id with a name. An ObjectID can be associated with more than one names.

Parameters

object_id – ObjectID
name – str

reset()#

reset() → None

Alias of ClientBase.clear().

property rpc_endpoint#: The RPC endpoint of the connected vineyard server.

shallow_copy()#

shallow_copy(object_id: ObjectID) → ObjectID

Create a shallow copy of the given vineyard object.

Parameters: object_id – ObjectID The vineyard object that is requested to be shallow-copied.
Returns: The object id of newly shallow-copied vineyard object.
Return type: ObjectID

shallow_copy(object_id: ObjectID, extra_metadata: dict) → ObjectID

Create a shallow copy of the given vineyard object, with extra metadata.

Parameters

object_id – ObjectID The vineyard object that is requested to be shallow-copied.
extra_metadata – dict Extra metadata to apply to the newly created object. The fields of extra metadata must be primitive types, e.g., string, number, and cannot be array or dict.

Returns

The object id of newly shallow-copied vineyard object.

Return type

ObjectID

property status#: The status the of connected vineyard server, returns a InstanceStatus.

See also

InstanceStatus

sync_meta()#

sync_meta() → None

Synchronize remote metadata to local immediately.

property version#: The version number string of connected vineyard server, in the format of semver: MAJOR.MINOR.PATCH.

class vineyard.RPCClient#

RPC client that connects to vineyard instance’s RPC endpoints.

The RPC client can only access the metadata of objects, any access to the blob payload will trigger a RuntimeError exception.

clear()#

clear() → None

Drop all objects that visible to the current instance in the vineyard cluster.

close()#: Close the client.

property connected#: Whether the client instance has been connected to the vineyard server.

create_metadata()#

create_metadata(metadata: ObjectMeta) → ObjectMeta

Create metadata in vineyardd.

Parameters: metadata – ObjectMeta The metadata that will be created on vineyardd.
Returns: The result created metadata.

create_metadata(metadata: ObjectMeta, instance_id: InstanceID) → ObjectMeta

Create metadata in vineyardd with a specified instance id.

Parameters

metadata – ObjectMeta The metadata that will be created on vineyardd.
instance_id – InstanceID The instance to place the object metadata.

Returns

The result created metadata.

create_remote_blob()#

create_remote_blob(blob_builder: RemoteBlobBuilder) → ObjectID

Put the remote blob to connected remote vineyard instance. The blob_builder is assumed to be ready and modification on the blob_builder after creation won’t take effect.

Note that creating remote blobs requires network transfer and may yields significate overhead.

vineyard_rpc_client = vineyard.connect(*vineyard_endpoint.split(':'))

buffer_writer = RemoteBlobBuilder(len(payload))
buffer_writer.copy(0, payload)
blob_id = vineyard_rpc_client.create_remote_blob(buffer_writer)

Parameters: blob_builder – RemoteBlobBuilder The remote blob to create.
Returns: ObjectID

See also

RPCClient.get_remote_blob RPCClient.get_remote_blobs

delete()#

delete(object_id: ObjectID or List[ObjectID], force: bool = false,
deep: bool = true) -> None

Delete the specific vineyard object.

Parameters

object_id – ObjectID or list of ObjectID Objects that will be deleted. The object_id can be a single ObjectID, or a list of ObjectID.
force – bool Forcedly delete an object means the member will be recursively deleted even if the member object is also referred by others. The default value is True.
deep –
bool Deeply delete an object means we will deleting the members recursively. The default value is True.

Note that when deleting objects which have direct blob members, the processing on those blobs yields a “deep” behavior.

delete(object_meta: ObjectMeta, force: bool = false, deep: bool = true)
-> None

Delete the specific vineyard object.

Parameters: object_meta – The corresponding object meta to delete.

delete(object: Object, force: bool = false, deep: bool = true) → None

Delete the specific vineyard object.

Parameters: object – The corresponding object meta to delete.

drop_name()#

drop_name(name: str or ObjectName) → None

Remove the association of the given name.

Parameters: name – str The name that will be removed.

exists()#

exists(object_id: ObjectID) → bool

Whether the given object exists.

Parameters: object_id – ObjectID The object id to check if exists.
Returns: True when the specified object exists.
Return type: bool

get(object_id, resolver=None, **kw)#

Get vineyard object as python value.

>>> arr_id = vineyard.ObjectID('00002ec13bc81226')
>>> arr = client.get(arr_id)
>>> arr
array([0, 1, 2, 3, 4, 5, 6, 7])

Parameters

client – IPCClient or RPCClient The vineyard client to use.
object_id – ObjectID The object id that will be obtained from vineyard.
resolver – When retrieving vineyard object, an optional resolver can be specified. If no resolver given, the default resolver context will be used.
kw – User-specific argument that will be passed to the builder.

Returns

A python object that return by the resolver, by resolving an vineyard object.

get_meta()#

get_meta(object_id: ObjectID) → ObjectMeta

Get object metadata from vineyard.

Parameters: object_id – ObjectID The object id to get.
Returns: ObjectMeta

get_metas()#

get_metas(object_ids: List[ObjectID] -> List[ObjectMeta]

Get metadatas of multiple objects from vineyard.

Parameters: object_ids – List[ObjectID]
Returns: List[ObjectMeta]

get_name()#

get_name(name: str or ObjectName, wait: bool = False) → ObjectID

Get the associated object id of the given name.

Parameters

name – str The name that will be queried.
wait – bool Whether to wait util the name appears, if wait, the request will be blocked until the name been registered.

Returns

The associated object id with the name.

Return type

ObjectID

get_object()#

get_object(object_id: ObjectID) → Object

Get object from vineyard.

Parameters: object_id – ObjectID The object id to get.
Returns: Object

get_objects()#

get_objects(object_ids: List[ObjectID]) → List[Object]

Get multiple objects from vineyard.

Parameters: object_ids – List[ObjectID]
Returns: List[Object]

get_remote_blob()#

get_remote_blob(object_id: ObjectID, unsafe: bool = false) → RemoteBlob

Getting a remote blob from vineyard using the given object ID.

Note that getting remote blobs requires network transfer and may yields significate overhead.

Parameters

object_id – ObjectID The remote blob to get.
unsafe – bool unsafe means getting the blob even the blob is not sealed yet. Default is False.

Returns

RemoteBlob

See also

IPCClient.get_blob IPCClient.get_blobs RPCClient.get_remote_blobs

get_remote_blobs()#

get_remote_blobs(object_ids: List[ObjectID], unsafe: bool = false)
-> List[RemoteBlob]

Getting remote blobs from vineyard using the given object IDs.

Note that getting remote blobs requires network transfer and may yields significate overhead.

Parameters

object_ids – List[ObjectID] The remote blobs to get.
unsafe – bool unsafe means getting the blob even the blob is not sealed yet. Default is False.

Returns

List[RemoteBlob]

See also

IPCClient.get_blob IPCClient.get_blobs RPCClient.get_remote_blob

property instance_id#: The instance id of the connected vineyard server.

property ipc_socket#: The UNIX domain socket location of connected vineyard server.

list_metadatas()#

list_metadatas(pattern: str, regex: bool = False, limit: int = 5,
nobuffer: bool = False) -> List[Object]

List all objects in current vineyard server.

Parameters

pattern – str The pattern string that will be matched against the object’s typename.
regex – bool Whether the pattern is a regex expression, otherwise the pattern will be used as wildcard pattern. Default value is False.
limit – int The limit to list. Default value is 5.
nobuffer – bool Whether to fill the buffers in returned object metadatas. Default value is False.

Returns

List[ObjectMeta]

list_names()#

list_names(pattern: str, regex: bool = False, limit: int = 5)
-> Dict[str, ObjectID]

List all names in current vineyard server.

Parameters

pattern – str The pattern string that will be matched against the name.
regex – bool Whether the pattern is a regex expression, otherwise the pattern will be used as wildcard pattern. Default value is False.
limit – int The limit to list. Default value is 5.

Returns

Dict[str, ObjectID]

list_objects()#

list_objects(pattern: str, regex: bool = False, limit: int = 5)
-> List[Object]

List all objects in current vineyard server.

Parameters

pattern – str The pattern string that will be matched against the object’s typename.
regex – bool Whether the pattern is a regex expression, otherwise the pattern will be used as wildcard pattern. Default value is False.
limit – int The limit to list. Default value is 5.

Returns

List[Object]

property meta#

The metadata information of the vineyard server. The value is a nested dict, the first-level key is the instance id, and the second-level key is the cluster metadata fields.

>>> client.meta
{
    14: {
        'hostid': '54058007061210',
        'hostname': '127.0.0.1',
        'timestamp': '6882550126788354072'
    },
    15: {
        'hostid': '48843417291806',
        'hostname': '127.0.0.1',
        'timestamp': '6882568290204737414'
    }
}

persist()#

persist(object_id: ObjectID) → None

Persist the object of the given object id. After persisting, the object will be visible by clients that connect to other vineyard server instances.

Parameters: object_id – ObjectID The object that will be persist.

persist(object_meta: ObjectMeta) → None

Persist the given object.

Parameters: object_meta – ObjectMeta The object that will be persist.

persist(object: Object) → None

Persist the given object.

Parameters: object – Object The object that will be persist.

put(value, builder=None, **kw)#

Put python value to vineyard.

>>> arr = np.arange(8)
>>> arr_id = client.put(arr)
>>> arr_id
00002ec13bc81226

Parameters

client – IPCClient The vineyard client to use.
value – The python value that will be put to vineyard. Supported python value types are decided by modules that registered to vineyard. By default, python value can be put to vineyard after serialized as a bytes buffer using pickle.
builder – When putting python value to vineyard, an optional builder can be specified to tell vineyard how to construct the corresponding vineyard Object. If not specified, the default builder context will be used to select a proper builder.
kw – User-specific argument that will be passed to the builder.

Returns

The result object id will be returned.

Return type

ObjectID

put_name()#

put_name(object: ObjectID or ObjectMeta or Object,
name: str or ObjectName) -> None

Associate the given object id with a name. An ObjectID can be associated with more than one names.

Parameters

object_id – ObjectID
name – str

property remote_instance_id#: The instance id of the connected remote vineyard server.

reset()#

reset() → None

Alias of ClientBase.clear().

property rpc_endpoint#: The RPC endpoint of the connected vineyard server.

shallow_copy()#

shallow_copy(object_id: ObjectID) → ObjectID

Create a shallow copy of the given vineyard object.

Parameters: object_id – ObjectID The vineyard object that is requested to be shallow-copied.
Returns: The object id of newly shallow-copied vineyard object.
Return type: ObjectID

shallow_copy(object_id: ObjectID, extra_metadata: dict) → ObjectID

Create a shallow copy of the given vineyard object, with extra metadata.

Parameters

object_id – ObjectID The vineyard object that is requested to be shallow-copied.
extra_metadata – dict Extra metadata to apply to the newly created object. The fields of extra metadata must be primitive types, e.g., string, number, and cannot be array or dict.

Returns

The object id of newly shallow-copied vineyard object.

Return type

ObjectID

property status#: The status the of connected vineyard server, returns a InstanceStatus.

See also

InstanceStatus

sync_meta()#

sync_meta() → None

Synchronize remote metadata to local immediately.

property version#: The version number string of connected vineyard server, in the format of semver: MAJOR.MINOR.PATCH.

Vineyard cluster#

class vineyard.InstanceStatus#

InstanceStatus represents the status of connected vineyard instance, including the instance identity, memory statistics and workloads on this instance.

>>> status = client.status
>>> print(status)
InstanceStatus:
    instance_id: 5
    deployment: local
    memory_usage: 360
    memory_limit: 268435456
    deferred_requests: 0
    ipc_connections: 1
    rpc_connections: 0
>>> status.instance_id
5
>>> status.deployment
'local'
>>> status.memory_usage
360
>>> status.memory_limit
268435456
>>> status.deferred_requests
0
>>> status.ipc_connections
1
>>> status.rpc_connections
0

__init__(*args, **kwargs)#

__repr__()#: Return repr(self).

__str__()#: Return str(self).

property deferred_requests#: Report number of waiting requests of current vineyardd instance.

property deployment#: The deployment mode of the connected vineyardd cluster, can be "local" and "distributed".

property instance_id#: Return the instance id of vineyardd that the client is connected to.

property ipc_connections#: Report number of alive IPC connections on the current vineyardd instance.

property memory_limit#: Report memory limit (in bytes) of current vineyardd instance.

property memory_usage#: Report memory usage (in bytes) of current vineyardd instance.

property rpc_connections#: Report number of alive RPC connections on the current vineyardd instance.

Blob#

class vineyard.Blob#

Blob in vineyard is a consecutive readonly shared memory.

property address#: The memory address value of this blob.

property buffer#: The readonly buffer behind this blob. The result buffer has type pyarrow::Buffer.

static empty()#: Create an empty blob (with size as 0).

property is_empty#: Whether the blob is an empty blob, i.e., the size of this blob is 0.

property size#: Size of the blob.

class vineyard.BlobBuilder#

BlobBuilder is the builder for creating a finally immutable blob in vineyard server.

A BlobBuilder can only be explicitly created using the IPCClient.create_blob().

See also

IPCClient.create_blob IPCClient.create_empty_blob

abort()#: Abort the blob builder if it is not sealed yet.

property address#: The memory address value of this blob builder.

property buffer#: The writeable buffer behind this blob builder. The result buffer has type pyarrow::Buffer, and it is a mutable one.

copy()#

copy(self, offset: int, ptr: int, size: int)

Copy the given address to the given offset.

property id#: ObjectID of this blob builder.

property size#: Size of this blob builder.

class vineyard.RemoteBlob#

RemoteBlob is a holder for Blob in cases like the Blob doesn’t exist in the local vineyardd instance but the client still want th access the data with a tolerable network-transfer overhead.

property address#: The memory address value of this blob.

property buffer#: The readonly buffer behind this blob. The result buffer has type pyarrow::Buffer.

property instance_id#: The instance ID where the blob is actually placed at.

property is_empty#: Whether the blob is an empty blob, i.e., the size of this blob is 0.

property size#: Object ID of the remote blob.

class vineyard.RemoteBlobBuilder#

RemoteBlobBuilder is the builder for creating a finally immutable blob in vineyard server over a RPC client.

A RemoteBlobBuilder can only be explicitly initialized using RemoteBlobBuilder(), then be filled the content and finally be sent to remote vineyard instance over the network.

See also

IPCClient.create_blob IPCClient.create_empty_blob

abort()#: Abort the blob builder if it is not sealed yet.

property address#: The memory address value of this blob builder.

property buffer#: The writeable buffer behind this blob builder. The result buffer has type pyarrow::Buffer, and it is a mutable one.

copy()#

copy(self, offset: int, ptr: int, size: int)

Copy the given address to the given offset.

property size#: Size of this blob builder.

Resolvers and builders#

class vineyard.core.resolver.ResolverContext[source]#

vineyard.core.resolver.get_current_resolvers()[source]#: Obtain current resolver context.

vineyard.core.resolver.resolver_context(resolvers=None, base=None)[source]#

Open a new context for register resolvers, without populting outside the global environment.

The resolver_context can be useful when users have more than more resolver for a certain type, e.g., the vineyard::Tensor object can be resolved as numpy.ndarray or xgboost::DMatrix.

We could have

def numpy_resolver(obj):
    ...

default_resolver_context.register('vineyard::Tensor', numpy_resolver)

and

def xgboost_resolver(obj):
    ...

default_resolver_context.register('vineyard::Tensor', xgboost_resolver)

Obviously there’s a conflict, and the stackable resolver_context could help there,

with resolver_context({'vineyard::Tensor', xgboost_resolver}):
    ...

Assuming the default context resolves vineyard::Tensor to numpy.ndarray, inside the with resolver_context the vineyard::Tensor will be resolved to xgboost::DMatrix, and after exiting the context the global environment will be restored back as default.

The with resolver_context is nestable as well.

See also

builder_context driver_context

class vineyard.core.builder.BuilderContext[source]#

register(type_id, builder)[source]#

Parameters

type_id (Python type) – Like int, or numpy.ndarray
builder (callable, e.g., a method, callable object) – A builder translates a python object to vineyard, it accepts a Python value as parameter, and returns an vineyard object as result.

run(client, value, **kw)[source]#

Follows the MRO to find the proper builder for given python value.

Here “Follows the MRO” implies:

If the type of python value has been found in the context, the registered builder will be used.
If not, it follows the MRO chain from down to top to find a registered Python type and used the associated builder.
When the traversal reaches the object type, since there’s a default builder that serialization the python value, the parameter will be serialized and be put into a blob.

vineyard.core.builder.get_current_builders()[source]#: Obtain the current builder context.

vineyard.core.builder.builder_context(builders=None, base=None)[source]#: Open a new context for register builders, without populting outside global environment.

See also

resolver_context driver_context

class vineyard.core.driver.DriverContext[source]#

vineyard.core.driver.get_current_drivers()[source]#: Obtain current driver context.

vineyard.core.driver.driver_context(drivers=None, base=None)[source]#: Open a new context for register drivers, without populting outside global environment.

See also

builder_context resolver_context

Shared memory#

class vineyard.shared_memory.SharedMemory(vineyard_client, name=None, create=False, size=0)[source]#

property buf#: A memoryview of contents of the shared memory block.

freeze()[source]#: Seal the shared memory to make it visible for other processes.

property name#: Unique name that identifies the shared memory block.

property size#: Size in bytes.

unlink()[source]#: Requests that the underlying shared memory block be destroyed.

class vineyard.shared_memory.ShareableList(vineyard_client, sequence=None, *, name=None)[source]#

Pattern for a mutable list-like object shareable via a shared memory block. It differs from the built-in list type in that these lists can not change their overall length (i.e. no append, insert, etc.)

Because values are packed into a memoryview as bytes, the struct packing format for any storable value must require no more than 8 characters to describe its format.

The ShareableList in vineyard differs slightly with its equivalent in the multiprocessing.shared_memory.ShareableList, as it becomes immutable after obtaining from the vineyard backend.

See also

multiprocessing.shared_memory.ShareableList

freeze()[source]#: Make the shareable list immutable and visible for other vineyard clients.

Deployment#

vineyard.init(num_instances=1, **kw)[source]#

Launching a local vineyardd instance and get a client as easy as possible

In a clean environment, simply use:

vineyard.init()

It will launch a local vineyardd and return a connected client to the vineyardd.

It will also setup the environment variable VINEYARD_IPC_SOCKET.

For the case to establish a local vineyard cluster consists of multiple vineyardd instances, using the num_instances parameter:

client1, client2, client3 = vineyard.init(num_instances=3)

In this case, three vineyardd instances will be launched.

The init method can only be called once in a process, to get the established sockets or clients later in the process, use get_current_socket or get_current_client respectively.

vineyard.shutdown()[source]#: Shutdown the vineyardd instances launched by previous vineyard.init().

vineyard.get_current_client()[source]#

Get current vineyard IPC clients established by vineyard.init().

Raises: ValueError if vineyard is not initialized. –

vineyard.get_current_socket()[source]#

Get current vineyard UNIX-domain socket established by vineyard.init().

Raises: ValueError if vineyard is not initialized. –

vineyard.deploy.local.start_vineyardd(etcd_endpoints=None, etcd_prefix=None, vineyardd_path=None, size='256M', socket=None, rpc=True, rpc_socket_port=9600, debug=False)[source]#

Launch a local vineyard cluster.

Parameters

etcd_endpoint – str Launching vineyard using specified etcd endpoints. If not specified, vineyard will launch its own etcd instance.
etcd_prefix – str Specify a common prefix to establish a local vineyard cluster.
vineyardd_path – str Location of vineyard server program. If not specified, vineyard will use its own bundled vineyardd binary.
size –
int The memory size limit for vineyard’s shared memory. The memory size can be a plain integer or as a fixed-point number using one of these suffixes:
```
E, P, T, G, M, K.
```
You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi, Mi, Ki.

For example, the following represent roughly the same value:
```
128974848, 129k, 129M, 123Mi, 1G, 10Gi, ...
```
socket –
str The UNIX domain socket socket path that vineyard server will listen on. Default is None.

When the socket parameter is None, a random path under temporary directory will be generated and used.
rpc_socket_port – int The port that vineyard will use to privode RPC service.
debug – bool Whether print debug logs.

Returns

Yields a tuple with the subprocess as the first element and the UNIX-domain IPC socket as the second element.

Return type

(proc, socket)

vineyard.deploy.distributed.start_vineyardd(hosts=None, etcd_endpoints=None, vineyardd_path=None, size='256M', socket='/var/run/vineyard.sock', rpc_socket_port=9600, debug=False)[source]#

Launch a local vineyard cluster in a distributed fashion.

Parameters

hosts – list of str A list of machines to launch vineyard server.
etcd_endpoint – str Launching vineyard using specified etcd endpoints. If not specified, vineyard will launch its own etcd instance.
vineyardd_path – str Location of vineyard server program. If not specified, vineyard will use its own bundled vineyardd binary.
size –
int The memory size limit for vineyard’s shared memory. The memory size can be a plain integer or as a fixed-point number using one of these suffixes:
```
E, P, T, G, M, K.
```
You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi, Mi, Ki.

For example, the following represent roughly the same value:
```
128974848, 129k, 129M, 123Mi, 1G, 10Gi, ...
```
socket – str The UNIX domain socket socket path that vineyard server will listen on.
rpc_socket_port – int The port that vineyard will use to privode RPC service.
debug – bool Whether print debug logs.

vineyard.deploy.kubernetes.start_vineyardd(namespace='vineyard', size='512Mi', socket='/var/run/vineyard.sock', rpc_socket_port=9600, vineyard_image='vineyardcloudnative/vineyardd:latest', vineyard_image_pull_policy='IfNotPresent', vineyard_image_pull_secrets=None, k8s_client=None)[source]#

Launch a vineyard cluster on kubernetes.

Parameters

namespace – str namespace in kubernetes
size –
int The memory size limit for vineyard’s shared memory. The memory size can be a plain integer or as a fixed-point number using one of these suffixes:
```
E, P, T, G, M, K.
```
You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi, Mi, Ki.

For example, the following represent roughly the same value:
```
128974848, 129k, 129M, 123Mi, 1G, 10Gi, ...
```
socket – str The UNIX domain socket socket path that vineyard server will listen on.
rpc_socket_port – int The port that vineyard will use to privode RPC service.
k8s_client – kubernetes.client.api.ApiClient A kubernetes client. If not specified, vineyard will try to resolve the kubernetes configuration from current context.
vineyard_image – str The docker image of vineyardd to launch the daemonset.
vineyard_image_pull_policy – str The docker image pull policy of vineyardd.
vineyard_image_pull_secrets – str The docker image pull secrets of vineyardd.

Returns

A list of created kubernetes resources during the deploying process. The resources can be later release using delete_kubernetes_objects().

See also

vineyard.deploy.kubernetes.delete_kubernetes_objects

vineyard.deploy.kubernetes.delete_kubernetes_objects(targets, k8s_client=None, verbose=False, wait=False, timeout_seconds=60, **kwargs)[source]#

Delete the given kubernetes resources.

Parameters

target – List List of Kubernetes objects
k8s_client – The kubernetes client. If not specified, vineyard will try to resolve the kubernetes configuration from current context.
verbose – bool Whether to print the deletion logs.
wait – bool Whether to wait for the deletion to complete.
timeout_seconds – int The timeout in seconds for waiting for the deletion to complete.

See also

vineyard.deploy.kubernetes.start_vineyardd vineyard.deploy.kubernetes.delete_kubernetes_object

I/O Drivers#

vineyard.io.open(path, *args, mode='r', handlers=None, **kwargs)[source]#

Open a path as a reader or writer, depends on the parameter mode. If mode is r, it will open a stream for read, and open a stream for write when mode is w.

Parameters

path (str) – Path to open.
mode (char) – Mode about how to open the path, r is for read and w for write.
vineyard_ipc_socket (str) – Vineyard’s IPC socket location.
vineyard_endpoint (str) – Vineyard’s RPC socket address.
handlers – A dict that will be filled with a handler that contains the process handler of the underlying read/write process that can be joined using join to capture the possible errors during the I/O proceeding.

See also

vineyard.io.read, vineyard.io.write

vineyard.io.read(path, *args, handlers=None, **kwargs)[source]#

Open a path and read it as a single stream.

Parameters

path (str) – Path to read, the last reader registered for the scheme of the path will be used.
vineyard_ipc_socket (str) – The local or remote vineyard’s IPC socket location that the remote readers will use to establish connections with the vineyard server.
vineyard_endpoint (str, optional) – An optional address of vineyard’s RPC socket, which will be used for retrieving server’s information on the client side. If not provided, the vineyard_ipc_socket will be used, or it will tries to discovery vineyard’s IPC or RPC endpoints from environment variables.

vineyard.io.write(path, stream, *args, handlers=None, **kwargs)[source]#

Write the stream to a given path.

Parameters

path (str) – Path to write, the last writer registered for the scheme of the path will be used.
stream (vineyard stream) – Stream that produces the data to write.
vineyard_ipc_socket (str) – The local or remote vineyard’s IPC socket location that the remote readers will use to establish connections with the vineyard server.
vineyard_endpoint (str, optional) – An optional address of vineyard’s RPC socket, which will be used for retrieving server’s information on the client side. If not provided, the vineyard_ipc_socket will be used, or it will tries to discovery vineyard’s IPC or RPC endpoints from environment variables.

Streams#

class vineyard.io.byte.ByteStream(meta: ObjectMeta, params: Optional[Dict] = None)[source]#

class vineyard.io.dataframe.DataframeStream(meta: ObjectMeta, params: Optional[Dict] = None)[source]#

class vineyard.io.recordbatch.RecordBatchStream(meta: ObjectMeta, params: Optional[Dict] = None)[source]#