Python API Reference

Objects

class vineyard.ObjectID

Opaque type for vineyard’s object id. The object ID is generated by vineyard server, the underlying type of ObjectID is a 64-bit unsigned integer. Wrapper utilities are provided to interact with the external python world.

>>> id = ObjectID("000043c5c6d5e646")
>>> id
000043c5c6d5e646
>>> repr(id)
'000043c5c6d5e646'
>>> print(id)
ObjectID <"000043c5c6d5e646">
>>> int(id)
74516723525190

__eq__()

Return self==value.

__hash__()

Return hash(self).

__init__()

__repr__()

Return repr(self).

__str__()

Return str(self).

class vineyard.Object

Base class for vineyard objects.

static from_()

from_(meta: ObjectMeta) → Object

Construct a new object from the meta. :param meta: ObjectMeta

The meta of the object.

Returns:

Object

property id

The object id of this object.

property isglobal

Whether the object is a global object.

property islocal

Whether the object is a local object.

property ispersist

Whether the object is a persistent object. The word “persistent” means the object could be seen by clients that connect to other vineyard server instances.

member()

member(self, name: str) → Object

Get the member object of this object.

Parameters:

name – str The name of the member object.

Returns:

The member object.

Return type:

Object

See also

ObjectMeta.get, ObjectMeta.__getitem__

property meta

The metadata of this object.

property nbytes

The nbytes of this object.

property signature

The object signature of this object.

property typename

The typename of this object. typename is the string value of the C++ type, e.g., vineyard::Array<int>, vineyard::Table.

class vineyard.ObjectBuilder

Metadata

class vineyard.ObjectMeta

ObjectMeta is the type for metadata of an Object. The ObjectMeta can be treat as a dict-like type. If the the metadata if the metadata obtained from vineyard, the metadata is readonly. Otherwise key-value attributes or object members could be associated with the metadata to construct a new vineyard object.

We can inspect the key-value attributes and members of an ObjectMeta:

>>> meta = client.get_meta(hashmap_id)
>>> meta
ObjectMeta {
    "id": "0000347aebe92dd0",
    "instance_id": "0",
    ...
}
>>> meta['num_elements_']
'5'
>>> meta['entries']
ObjectMeta {
    "id": "0000347aebe92dd0",
    "instance_id": "0",
    ...
}

ObjectMeta value can be iterated over:

>>> list(k for k in meta['entries'])
['transient', 'num_slots_minus_one_', 'max_lookups_', 'num_elements_', 'entries_',
 'nbytes', 'typename', 'instance_id', 'id']

__contains__()

Check if given key exists in the object metadata.

Parameters:

key – str The name to be looked up.

Returns:

True if the queried key exists in this object metadata, otherwise False.

Return type:

bool

__getitem__()

__getitem__(self, key: str) → string or Object

Get meta or member’s meta from metadata.

Parameters:

key – str The name to be looked up.

Returns:

• string – If the given key is a key of meta, returns the meta value.

• Object – If the given key is a key of member, return the meta of this member.

__init__()

__init__(global_: bool = False)

Create an empty metadata, the metadata will be used to create a vineyard object.

Parameters

global_: bool, if the object meta is for creating a global object.

__repr__()

Return repr(self).

__setitem__()

__setitem__(self, key: str, value) → None

Add a metadata to the ObjectMeta.

Parameters:

• key – str The name of the new metadata entry.

• value –

  str, int, float, bool or list of int The value of the new metadata entry.

  • When the value is a str, it will be convert to string at first.

  • When the value is a list of str, int or float, it will be first dumpped as string using json.dumps.

__setitem__(self, key: str, ObjectID, Object or ObjectMeta) → None

Add a member object.

Parameters:

• key – str The name of the member object.

• object – Object, ObjectID or ObjectMeta The reference to the member object or the object id of the member object.

__str__()

Return str(self).

add_member()

add_member(self, key: str, ObjectID, Object or ObjectMeta) → None

Add a member object.

Parameters:

• key – str The name of the member object.

• object – Object, ObjectID or ObjectMeta The reference to the member object or the object id of the member object.

add_remote_blob()

add_remote_blob(self, RemoteBlob blob) → None

Add the remote blob’s buffer to the metadata.

Parameters:

blob – RemoteBlob The reference to the remote blob.

get()

get(self, key: str, default=None) → string or Object

Get meta or member’s meta from metadata, return default value if the given key is not presented.

Parameters:

key – str The name to be looked up.

Returns:

• str – When the given key belongs to a metadata pair. Note that the metadata value of type int or float will be returned in string format as well.

• ObjectMeta – When the given key is mapped to a member object.

• See Also – ObjectMeta.__getitem__

get_member()

get_member(self, key: str) → Object

Get member object from metadata, return None if the given key is not presented, and raise exception RuntimeError if the given key is associated with a plain metadata, rather than member object.

Parameters:

key – str The name to be looked up.

Raises:

RuntimeError – When the given key is associated with a plain metadata, rather than member object.

See also

ObjectMeta.__getitem__, ObjectMeta.get

property id

The corresponding object ID of this metadata.

property instance_id

The "instance_id" of vineyard instance that the metadata been placed on.

property isglobal

True if the object is a global object, otherwise a local object or remote object.

property islocal

True if the object is a local object, otherwise a global object or remote object.

member()

get_member(self, key: str) → Object

Get member object from metadata, return None if the given key is not presented, and raise exception RuntimeError if the given key is associated with a plain metadata, rather than member object.

Parameters:

key – str The name to be looked up.

Raises:

RuntimeError – When the given key is associated with a plain metadata, rather than member object.

See also

ObjectMeta.__getitem__, ObjectMeta.get

property memory_usage

Get the total memory usage of buffers in this object meta.

property nbytes

The "nbytes" attribute of this metadata.

set_global()

Mark the building object as a global object.

Parameters:

global – bool, default is True

property signature

The corresponding object signature of this metadata.

property typename

The "typename" attribute of this metadata.

Vineyard client

vineyard.connect(*args, **kwargs)

Connect to vineyard by specified UNIX-domain socket or TCP endpoint.

If no arguments are provided and failed to resolve both the environment variables VINEYARD_IPC_SOCKET, VINEYARD_RPC_ENDPOINT, VINEYARD_CONFIG, and the default configuration file /var/run/vineyard-config.yaml and /var/run/vineyard/vineyard-config.yaml, it will launch a standalone vineyardd server in the background and then connect to it.

The connect() method has various overloading:

connect(socket: str,

username: str = None,

password: str = None) -> IPCClient

Connect to vineyard via UNIX domain socket for IPC service:

client = vineyard.connect('/var/run/vineyard.sock')

Parameters:

socket: str

UNIX domain socket path to setup an IPC connection.

username: str

Username to login, default to None.

password: str

Password to login, default to None.

Returns:

IPCClient: The connected IPC client.

connect(host: str,

port: int or str,

username: str = None,

password: str = None) -> RPCClient

Connect to vineyard via TCP socket.

Parameters:

host: str

Hostname to connect to.

port: int or str

The TCP that listened by vineyard TCP service.

username: str

Username to login, default to None.

password: str

Password to login, default to None.

Returns:

RPCClient: The connected RPC client.

connect(endpoint: (str, int or str),

username: str = None,

password: str = None) -> RPCClient

Connect to vineyard via TCP socket.

Parameters:

endpoint: tuple(str, int or str)

Endpoint to connect to. The parameter is a tuple, in which the first element is the host, and the second parameter is the port (can be int a str).

username: str

Username to login, default to None.

password: str

Password to login, default to None.

Returns:

RPCClient: The connected RPC client.

connect(username: str = None,

password: str = None) -> IPCClient or RPCClient

Connect to vineyard via UNIX domain socket or TCP endpoint. This method normally usually no arguments, and will first tries to resolve IPC socket from the environment variable VINEYARD_IPC_SOCKET and connect to it. If it fails to establish a connection with vineyard server, the method will tries to resolve RPC endpoint from the environment variable VINEYARD_RPC_ENDPOINT. If both tries are failed, this method will try to resolve the configuration file that contains IPC socket and RPC endpoint from the environment variable VINEYARD_CONFIG, and then connect to the vineyard server with the resolved configuration.

If above all are failed, this method will raise a ConnectionFailed exception.

In rare cases, user may be not sure about if the IPC socket or RPC endpoint is available, i.e., the variable might be None. In such cases this method can accept a None as arguments, and do resolution as described above.

Parameters:

username: str

Username to login, default to None.

password: str

Password to login, default to None.

Raises:

ConnectionFailed

class vineyard.Client(socket: str | None = None, port: int | str | None = None, host: str | None = None, endpoint: Tuple[str, str | int] | None = None, rdma_endpoint: str | Tuple[str, str | int] | None = None, session: int | None = None, username: str | None = None, password: str | None = None, config: str | None = None)

Client is responsible for managing IPC and RPC clients for Vineyard and provides a high-level interface to fetch objects from the Vineyard cluster.

allocated_size(object_id: Object | ObjectID) → int

allocated_size(target: Object or ObjectID) → int

Get the allocated size of the given object.

Parameters:

target – Object or ObjectID The given Object.

Returns:

int

clear() → None

clear() → None

Drop all objects that visible to the current instance in the vineyard cluster.

close() → None

Close the client.

property compression: bool

Whether the compression is enabled for underlying RPC client.

property connected

Whether the client instance has been connected to the vineyard server.

create_blob(size: int | List[int]) → BlobBuilder | List[BlobBuilder]

create_blob(size: int) → BlobBuilder

Allocate a blob in vineyard server.

Parameters:

size: int

The size of blob that will be allocated on vineyardd.

Returns:

BlobBuilder

create_blob(sizes: List[int]) → List[BlobBuilder]

Allocate blobs in vineyard server.

Parameters:

size: List[int]

The size of blobs that will be allocated on vineyardd.

Returns:

List[BlobBuilder]

create_blob(size: int) → BlobBuilder

Allocate a blob in vineyard server.

Parameters:

size: int

The size of blob that will be allocated on vineyardd.

Returns:

BlobBuilder

create_blob(sizes: List[int]) → List[BlobBuilder]

Allocate blobs in vineyard server.

Parameters:

size: List[int]

The size of blobs that will be allocated on vineyardd.

Returns:

List[BlobBuilder]

create_empty_blob() → BlobBuilder

create_empty_blob() → Blob

Create an empty blob in vineyard server.

Returns:

Blob

create_metadata(metadata: ObjectMeta | List[ObjectMeta], instance_id: int | None = None) → ObjectMeta | List[ObjectMeta]

create_metadata(metadata: ObjectMeta | List[ObjectMeta])

-> Union[ObjectMeta, List[ObjectMeta]]

Create metadata in vineyardd.

Parameters:

metadata – ObjectMeta The metadata that will be created on vineyardd.

Returns:

The result created metadata.

Return type:

Union[ObjectMeta, List[ObjectMeta]]

create_metadata(metadata: Union[ObjectMeta, List[ObjectMeta]],

instance_id: InstanceID)

-> Union[ObjectMeta, List[ObjectMeta]]

Create metadata in vineyardd with a specified instance id.

Parameters:

• metadata – ObjectMeta The metadata that will be created on vineyardd.

• instance_id – InstanceID The instance to place the object metadata.

Returns:

The result created metadata.

Return type:

Union[ObjectMeta, List[ObjectMeta]]

create_metadata(metadata: ObjectMeta | List[ObjectMeta])

-> Union[ObjectMeta, List[ObjectMeta]]

Create metadata in vineyardd.

Parameters:

metadata – ObjectMeta The metadata that will be created on vineyardd.

Returns:

The result created metadata.

Return type:

Union[ObjectMeta, List[ObjectMeta]]

create_metadata(metadata: Union[ObjectMeta, List[ObjectMeta]],

instance_id: InstanceID)

-> Union[ObjectMeta, List[ObjectMeta]]

Create metadata in vineyardd with a specified instance id.

Parameters:

• metadata – ObjectMeta The metadata that will be created on vineyardd.

• instance_id – InstanceID The instance to place the object metadata.

Returns:

The result created metadata.

Return type:

Union[ObjectMeta, List[ObjectMeta]]

create_remote_blob(blob_builder: RemoteBlobBuilder | List[RemoteBlobBuilder]) → ObjectMeta | List[ObjectMeta]

create_remote_blob(blob_builder: RemoteBlobBuilder) → ObjectMeta

Put the remote blob to connected remote vineyard instance. The blob_builder is assumed to be ready and modification on the blob_builder after creation won’t take effect.

Note that creating remote blobs requires network transfer and may yields significate overhead.

vineyard_rpc_client = vineyard.connect(*vineyard_endpoint.split(':'))

buffer_writer = RemoteBlobBuilder(len(payload))
buffer_writer.copy(0, payload)
blob_meta = vineyard_rpc_client.create_remote_blob(buffer_writer)

Parameters:

blob_builder: RemoteBlobBuilder

The remote blob to create.

Returns:

ObjectMeta

See Also:

RPCClient.get_remote_blob RPCClient.get_remote_blobs

create_remote_blob(blob_builders: List[RemoteBlobBuilder]) → List[ObjectMeta]

Put the remote blobs to connected remote vineyard instance. The blob_builders is assumed to be ready and modification on the blob_builder after creation won’t take effect.

Note that creating remote blobs requires network transfer and may yields significate overhead.

Returns:

List[ObjectMeta]

See Also:

RPCClient.get_remote_blob RPCClient.get_remote_blobs

create_remote_blob(blob_builder: RemoteBlobBuilder) → ObjectMeta

Put the remote blob to connected remote vineyard instance. The blob_builder is assumed to be ready and modification on the blob_builder after creation won’t take effect.

Note that creating remote blobs requires network transfer and may yields significate overhead.

vineyard_rpc_client = vineyard.connect(*vineyard_endpoint.split(':'))

buffer_writer = RemoteBlobBuilder(len(payload))
buffer_writer.copy(0, payload)
blob_meta = vineyard_rpc_client.create_remote_blob(buffer_writer)

Parameters:

blob_builder: RemoteBlobBuilder

The remote blob to create.

Returns:

ObjectMeta

See Also:

RPCClient.get_remote_blob RPCClient.get_remote_blobs

create_remote_blob(blob_builders: List[RemoteBlobBuilder]) → List[ObjectMeta]

Put the remote blobs to connected remote vineyard instance. The blob_builders is assumed to be ready and modification on the blob_builder after creation won’t take effect.

Note that creating remote blobs requires network transfer and may yields significate overhead.

Returns:

List[ObjectMeta]

See Also:

RPCClient.get_remote_blob RPCClient.get_remote_blobs

delete(object: ObjectID | Object | ObjectMeta | List[ObjectID], force: bool = False, deep: bool = True, memory_trim: bool = False) → None

delete(object_id: ObjectID or List[ObjectID], force: bool = false,

deep: bool = true, memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:

• object_id – ObjectID or list of ObjectID Objects that will be deleted. The object_id can be a single ObjectID, or a list of ObjectID.

• force – bool Forcedly delete an object means the member will be recursively deleted even if the member object is also referred by others. The default value is True.

• deep –

  bool Deeply delete an object means we will deleting the members recursively. The default value is True.

  Note that when deleting objects which have direct blob members, the processing on those blobs yields a “deep” behavior.

• memory_trim –

  bool Whether to trim the memory pool inside the shared memory allocator to return the unused physical memory back to the OS kernel, like the malloc_trim API from glibc. Default value is False.

  Note that the memory trimming operation is a best-effort operation and may not release any memory at all.

delete(object_meta: ObjectMeta, force: bool = false, deep: bool = true,

memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:

object_meta – The corresponding object meta to delete.

delete(object: Object, force: bool = false, deep: bool = true,

memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:

object – The corresponding object meta to delete.

delete(object_id: ObjectID or List[ObjectID], force: bool = false,

deep: bool = true, memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:

• object_id – ObjectID or list of ObjectID Objects that will be deleted. The object_id can be a single ObjectID, or a list of ObjectID.

• force – bool Forcedly delete an object means the member will be recursively deleted even if the member object is also referred by others. The default value is True.

• deep –

  bool Deeply delete an object means we will deleting the members recursively. The default value is True.

  Note that when deleting objects which have direct blob members, the processing on those blobs yields a “deep” behavior.

• memory_trim –

  bool Whether to trim the memory pool inside the shared memory allocator to return the unused physical memory back to the OS kernel, like the malloc_trim API from glibc. Default value is False.

  Note that the memory trimming operation is a best-effort operation and may not release any memory at all.

delete(object_meta: ObjectMeta, force: bool = false, deep: bool = true,

memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:

object_meta – The corresponding object meta to delete.

delete(object: Object, force: bool = false, deep: bool = true,

memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:

object – The corresponding object meta to delete.

delete(object_id: ObjectID or List[ObjectID], force: bool = false,

deep: bool = true, memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:

• object_id – ObjectID or list of ObjectID Objects that will be deleted. The object_id can be a single ObjectID, or a list of ObjectID.

• force – bool Forcedly delete an object means the member will be recursively deleted even if the member object is also referred by others. The default value is True.

• deep –

  bool Deeply delete an object means we will deleting the members recursively. The default value is True.

  Note that when deleting objects which have direct blob members, the processing on those blobs yields a “deep” behavior.

• memory_trim –

  bool Whether to trim the memory pool inside the shared memory allocator to return the unused physical memory back to the OS kernel, like the malloc_trim API from glibc. Default value is False.

  Note that the memory trimming operation is a best-effort operation and may not release any memory at all.

delete(object_meta: ObjectMeta, force: bool = false, deep: bool = true,

memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:

object_meta – The corresponding object meta to delete.

delete(object: Object, force: bool = false, deep: bool = true,

memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:

object – The corresponding object meta to delete.

delete(object_id: ObjectID or List[ObjectID], force: bool = false,

deep: bool = true, memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:

• object_id – ObjectID or list of ObjectID Objects that will be deleted. The object_id can be a single ObjectID, or a list of ObjectID.

• force – bool Forcedly delete an object means the member will be recursively deleted even if the member object is also referred by others. The default value is True.

• deep –

  bool Deeply delete an object means we will deleting the members recursively. The default value is True.

  Note that when deleting objects which have direct blob members, the processing on those blobs yields a “deep” behavior.

• memory_trim –

  bool Whether to trim the memory pool inside the shared memory allocator to return the unused physical memory back to the OS kernel, like the malloc_trim API from glibc. Default value is False.

  Note that the memory trimming operation is a best-effort operation and may not release any memory at all.

delete(object_meta: ObjectMeta, force: bool = false, deep: bool = true,

memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:

object_meta – The corresponding object meta to delete.

delete(object: Object, force: bool = false, deep: bool = true,

memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:

object – The corresponding object meta to delete.

drop_name(name: str) → None

drop_name(name: str or ObjectName) → None

Remove the association of the given name.

Parameters:

name – str The name that will be removed.

exists(object: ObjectID) → bool

exists(object_id: ObjectID) → bool

Whether the given object exists.

Parameters:

object_id – ObjectID The object id to check if exists.

Returns:

True when the specified object exists.

Return type:

bool

find_shared_memory(pointer: int) → ObjectID

find_shared_memory(target: ptr) → bool

Find the corresponding blob (if exists) of the given pointer.

Parameters:

target – address, in int format The given address.

Returns:

ObjectID

find_shared_memory(target: ptr) → bool

Find the corresponding blob (if exists) of the given pointer.

Parameters:

target – address, in int format The given address.

Returns:

ObjectID

get(object_id: ObjectID | None = None, name: str | None = None, resolver: ResolverContext | None = None, fetch: bool = False, **kwargs)

Get vineyard object as python value.

>>> arr_id = vineyard.ObjectID('00002ec13bc81226')
>>> arr = client.get(arr_id)
>>> arr
array([0, 1, 2, 3, 4, 5, 6, 7])

Parameters:

• client – IPCClient or RPCClient The vineyard client to use.

• object_id – ObjectID The object id that will be obtained from vineyard.

• name – ObjectID The object name that will be obtained from vineyard, ignored if object_id is not None.

• resolver – When retrieving vineyard object, an optional resolver can be specified. If no resolver given, the default resolver context will be used.

• fetch – Whether to trigger a migration when the target object is located on remote instances.

• kw – User-specific argument that will be passed to the builder.

Returns:

A python object that return by the resolver, by resolving an vineyard object.

get_blob(object_id: ObjectID, unsafe: bool = False) → Blob

get_blob(object_id: ObjectID, unsafe: bool = false) → Blob

Getting a blob from vineyard using the given object ID.

Parameters:

• object_id – ObjectID The blob to get.

• unsafe – bool unsafe means getting the blob even the blob is not sealed yet. Default is False.

Returns:

Blob

See also

IPCClient.get_blob RPCClient.get_remote_blob RPCClient.get_remote_blobs

get_blobs(object_ids: List[ObjectID], unsafe: bool = False) → List[Blob]

get_blobs(object_ids: List[ObjectID], unsafe: bool = false) → List[Blob]

Getting blobs from vineyard using the given object IDs.

Parameters:

• object_ids – List[ObjectID] The blobs to get.

• unsafe – bool unsafe means getting the blob even the blob is not sealed yet. Default is False.

Returns:

List[Blob]

See also

IPCClient.get_blobs RPCClient.get_remote_blob RPCClient.get_remote_blobs

get_meta(object_id: ObjectID, sync_remote: bool = True) → ObjectMeta

get_meta(object_id: ObjectID, sync_remote: bool = False) → ObjectMeta

Get object metadata from vineyard.

Parameters:

• object_id – ObjectID The object id to get.

• sync_remote – bool If the target object is a remote object, code_remote=True will force a meta synchronization on the vineyard server. Default is False.

Returns:

ObjectMeta

get_metas(object_ids: List[ObjectID], sync_remote: bool = True) → List[ObjectMeta]

get_metas(object_ids: List[ObjectID], sync_remote: bool = False)

-> List[ObjectMeta]

Get metadatas of multiple objects from vineyard.

Parameters:

• object_ids – List[ObjectID] The object ids to get.

• sync_remote – bool If the target object is a remote object, code_remote=True will force a meta synchronization on the vineyard server. Default is False.

Returns:

List[ObjectMeta]

get_name(name: str, wait: bool = False) → ObjectID

get_name(name: str or ObjectName, wait: bool = False) → ObjectID

Get the associated object id of the given name.

Parameters:

• name – str The name that will be queried.

• wait – bool Whether to wait util the name appears, if wait, the request will be blocked until the name been registered.

Returns:

The associated object id with the name.

Return type:

ObjectID

get_object(object_id: ObjectID, sync_remote: bool = True, fetch: bool = False) → Object

get_object(object_id: ObjectID, fetch: bool, sync_remote: bool = True) → Object

Get object from vineyard.

Parameters:

• object_id – ObjectID The object id to get.

• fetch – bool whether to trigger the migration of the object from the remote vineyardd to the connected one.

• sync_remote – bool whether to synchronize the metadata from the metadata service.

Returns:

Object

get_objects(object_ids: List[ObjectID], sync_remote: bool = True) → List[Object]

get_objects(object_ids: List[ObjectID], sync_remote: bool = True) → List[Object]

Get multiple objects from vineyard.

Parameters:

• object_ids – List[ObjectID]

• sync_remote – bool whether to synchronize the metadata from the metadata service.

Returns:

List[Object]

get_remote_blob(object_id: ObjectID, unsafe: bool = False) → RemoteBlob

get_remote_blob(object_id: ObjectID, unsafe: bool = false) → RemoteBlob

Getting a remote blob from vineyard using the given object ID.

Note that getting remote blobs requires network transfer and may yields significate overhead.

Parameters:

• object_id – ObjectID The remote blob to get.

• unsafe – bool unsafe means getting the blob even the blob is not sealed yet. Default is False.

Returns:

RemoteBlob

See also

IPCClient.get_blob IPCClient.get_blobs RPCClient.get_remote_blobs

get_remote_blobs(object_ids: List[ObjectID], unsafe: bool = False) → List[RemoteBlob]

get_remote_blobs(object_ids: List[ObjectID], unsafe: bool = false)

-> List[RemoteBlob]

Getting remote blobs from vineyard using the given object IDs.

Note that getting remote blobs requires network transfer and may yields significate overhead.

Parameters:

• object_ids – List[ObjectID] The remote blobs to get.

• unsafe – bool unsafe means getting the blob even the blob is not sealed yet. Default is False.

Returns:

List[RemoteBlob]

See also

IPCClient.get_blob IPCClient.get_blobs RPCClient.get_remote_blob

property instance_id

The instance id of the connected vineyard server.

property ipc_socket

The UNIX domain socket location of connected vineyard server.

property is_ipc

Whether the client is connected to vineyard server via UNIX domain socket.

property is_rpc

Whether the client is connected to vineyard server via RPC endpoint.

is_shared_memory(pointer: int) → bool

is_shared_memory(target: ptr) → bool

Check if the address is on the shared memory region.

Parameters:

target – address, in int format The given address.

Returns:

bool

is_shared_memory(target: ptr) → bool

Check if the address is on the shared memory region.

Parameters:

target – address, in int format The given address.

Returns:

bool

list_metadatas(pattern: str, regex: bool = False, limit: int = 5, nobuffer: bool = False) → List[ObjectMeta]

list_metadatas(pattern: str, regex: bool = False, limit: int = 5,

nobuffer: bool = False) -> List[Object]

List all objects in current vineyard server.

Parameters:

• pattern – str The pattern string that will be matched against the object’s typename.

• regex – bool Whether the pattern is a regex expression, otherwise the pattern will be used as wildcard pattern. Default value is False.

• limit – int The limit to list. Default value is 5.

• nobuffer – bool Whether to fill the buffers in returned object metadatas. Default value is False.

Returns:

List[ObjectMeta]

list_names(pattern: str, regex: bool = False, limit: int = 5) → List[str]

list_names(pattern: str, regex: bool = False, limit: int = 5)

-> Dict[str, ObjectID]

List all names in current vineyard server.

Parameters:

• pattern – str The pattern string that will be matched against the name.

• regex – bool Whether the pattern is a regex expression, otherwise the pattern will be used as wildcard pattern. Default value is False.

• limit – int The limit to list. Default value is 5.

Returns:

Dict[str, ObjectID]

list_objects(pattern: str, regex: bool = False, limit: int = 5) → List[ObjectID]

list_objects(pattern: str, regex: bool = False, limit: int = 5)

-> List[Object]

List all objects in current vineyard server.

Parameters:

• pattern – str The pattern string that will be matched against the object’s typename.

• regex – bool Whether the pattern is a regex expression, otherwise the pattern will be used as wildcard pattern. Default value is False.

• limit – int The limit to list. Default value is 5.

Returns:

List[Object]

memory_trim() → bool

memory_trim() → bool

Trim the memory pool inside the shared memory allocator to return the unused physical memory back to the OS kernel, like the malloc_trim API from glibc.

Returns True if it actually released any memory.

property meta

The metadata information of the vineyard server. The value is a nested dict, the first-level key is the instance id, and the second-level key is the cluster metadata fields.

>>> client.meta
{
    14: {
        'hostid': '54058007061210',
        'hostname': '127.0.0.1',
        'timestamp': '6882550126788354072'
    },
    15: {
        'hostid': '48843417291806',
        'hostname': '127.0.0.1',
        'timestamp': '6882568290204737414'
    }
}

persist(object: ObjectID | Object | ObjectMeta) → None

persist(object_id: ObjectID) → None

Persist the object of the given object id. After persisting, the object will be visible by clients that connect to other vineyard server instances.

Parameters:

object_id – ObjectID The object that will be persist.

persist(object_meta: ObjectMeta) → None

Persist the given object.

Parameters:

object_meta – ObjectMeta The object that will be persist.

persist(object: Object) → None

Persist the given object.

Parameters:

object – Object The object that will be persist.

put(value: Any, builder: BuilderContext | None = None, persist: bool = False, name: str | None = None, **kwargs)

Put python value to vineyard.

>>> arr = np.arange(8)
>>> arr_id = client.put(arr)
>>> arr_id
00002ec13bc81226

Parameters:

• client – IPCClient or RPCClient The vineyard client to use.

• value – The python value that will be put to vineyard. Supported python value types are decided by modules that registered to vineyard. By default, python value can be put to vineyard after serialized as a bytes buffer using pickle.

• builder – optional When putting python value to vineyard, an optional builder can be specified to tell vineyard how to construct the corresponding vineyard Object. If not specified, the default builder context will be used to select a proper builder.

• persist – bool, optional If true, persist the object after creation.

• name – str, optional If given, the name will be automatically associated with the resulted object. Note that only take effect when the object is persisted.

• kw – User-specific argument that will be passed to the builder.

Returns:

The result object id will be returned.

Return type:

ObjectID

put_name(object: Object | ObjectMeta | ObjectID, name: str) → None

put_name(object: ObjectID or ObjectMeta or Object,

name: str or ObjectName) -> None

Associate the given object id with a name. An ObjectID can be associated with more than one names.

Parameters:

• object_id – ObjectID

• name – str

property remote_instance_id: int

The instance id of the connected remote vineyard server.

reset() → None

reset() → None

Alias of ClientBase.clear().

property rpc_endpoint

The RPC endpoint of the connected vineyard server.

shallow_copy(object_id: ObjectID, extra_metadata: dict | None = None) → ObjectID

shallow_copy(object_id: ObjectID) → ObjectID

Create a shallow copy of the given vineyard object.

Parameters:

object_id – ObjectID The vineyard object that is requested to be shallow-copied.

Returns:

The object id of newly shallow-copied vineyard object.

Return type:

ObjectID

shallow_copy(object_id: ObjectID, extra_metadata: dict) → ObjectID

Create a shallow copy of the given vineyard object, with extra metadata.

Parameters:

• object_id – ObjectID The vineyard object that is requested to be shallow-copied.

• extra_metadata – dict Extra metadata to apply to the newly created object. The fields of extra metadata must be primitive types, e.g., string, number, and cannot be array or dict.

Returns:

The object id of newly shallow-copied vineyard object.

Return type:

ObjectID

property spread: bool

Whether the spread is enabled for underlying RPC client.

property status

The status the of connected vineyard server, returns a InstanceStatus.

See also

InstanceStatus

sync_meta() → None

sync_meta() → None

Synchronize remote metadata to local immediately.

property version

The version number string of connected vineyard server, in the format of semver: MAJOR.MINOR.PATCH.

with_compression(enabled: bool = True)

Disable compression for the following put operations.

with_spread(enabled: bool = True)

Enable spread for the following put operations.

class vineyard.IPCClient

IPC client that connects to vineyard instance’s UNIX domain socket.

allocated_size()

allocated_size(target: Object or ObjectID) → int

Get the allocated size of the given object.

Parameters:

target – Object or ObjectID The given Object.

Returns:

int

clear()

clear() → None

Drop all objects that visible to the current instance in the vineyard cluster.

close()

Close the client.

property connected

Whether the client instance has been connected to the vineyard server.

create_blob()

create_blob(size: int) → BlobBuilder

Allocate a blob in vineyard server.

Parameters:

size: int

The size of blob that will be allocated on vineyardd.

Returns:

BlobBuilder

create_blob(sizes: List[int]) → List[BlobBuilder]

Allocate blobs in vineyard server.

Parameters:

size: List[int]

The size of blobs that will be allocated on vineyardd.

Returns:

List[BlobBuilder]

create_blob(size: int) → BlobBuilder

Allocate a blob in vineyard server.

Parameters:

size: int

The size of blob that will be allocated on vineyardd.

Returns:

BlobBuilder

create_blob(sizes: List[int]) → List[BlobBuilder]

Allocate blobs in vineyard server.

Parameters:

size: List[int]

The size of blobs that will be allocated on vineyardd.

Returns:

List[BlobBuilder]

create_empty_blob()

create_empty_blob() → Blob

Create an empty blob in vineyard server.

Returns:

Blob

create_metadata()

create_metadata(metadata: ObjectMeta | List[ObjectMeta])

-> Union[ObjectMeta, List[ObjectMeta]]

Create metadata in vineyardd.

Parameters:

metadata – ObjectMeta The metadata that will be created on vineyardd.

Returns:

The result created metadata.

Return type:

Union[ObjectMeta, List[ObjectMeta]]

create_metadata(metadata: Union[ObjectMeta, List[ObjectMeta]],

instance_id: InstanceID)

-> Union[ObjectMeta, List[ObjectMeta]]

Create metadata in vineyardd with a specified instance id.

Parameters:

• metadata – ObjectMeta The metadata that will be created on vineyardd.

• instance_id – InstanceID The instance to place the object metadata.

Returns:

The result created metadata.

Return type:

Union[ObjectMeta, List[ObjectMeta]]

create_metadata(metadata: ObjectMeta | List[ObjectMeta])

-> Union[ObjectMeta, List[ObjectMeta]]

Create metadata in vineyardd.

Parameters:

metadata – ObjectMeta The metadata that will be created on vineyardd.

Returns:

The result created metadata.

Return type:

Union[ObjectMeta, List[ObjectMeta]]

create_metadata(metadata: Union[ObjectMeta, List[ObjectMeta]],

instance_id: InstanceID)

-> Union[ObjectMeta, List[ObjectMeta]]

Create metadata in vineyardd with a specified instance id.

Parameters:

• metadata – ObjectMeta The metadata that will be created on vineyardd.

• instance_id – InstanceID The instance to place the object metadata.

Returns:

The result created metadata.

Return type:

Union[ObjectMeta, List[ObjectMeta]]

delete()

delete(object_id: ObjectID or List[ObjectID], force: bool = false,

deep: bool = true, memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:

• object_id – ObjectID or list of ObjectID Objects that will be deleted. The object_id can be a single ObjectID, or a list of ObjectID.

• force – bool Forcedly delete an object means the member will be recursively deleted even if the member object is also referred by others. The default value is True.

• deep –

  bool Deeply delete an object means we will deleting the members recursively. The default value is True.

  Note that when deleting objects which have direct blob members, the processing on those blobs yields a “deep” behavior.

• memory_trim –

  bool Whether to trim the memory pool inside the shared memory allocator to return the unused physical memory back to the OS kernel, like the malloc_trim API from glibc. Default value is False.

  Note that the memory trimming operation is a best-effort operation and may not release any memory at all.

delete(object_meta: ObjectMeta, force: bool = false, deep: bool = true,

memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:

object_meta – The corresponding object meta to delete.

delete(object: Object, force: bool = false, deep: bool = true,

memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:

object – The corresponding object meta to delete.

delete(object_id: ObjectID or List[ObjectID], force: bool = false,

deep: bool = true, memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:

• object_id – ObjectID or list of ObjectID Objects that will be deleted. The object_id can be a single ObjectID, or a list of ObjectID.

• force – bool Forcedly delete an object means the member will be recursively deleted even if the member object is also referred by others. The default value is True.

• deep –

  bool Deeply delete an object means we will deleting the members recursively. The default value is True.

  Note that when deleting objects which have direct blob members, the processing on those blobs yields a “deep” behavior.

• memory_trim –

  bool Whether to trim the memory pool inside the shared memory allocator to return the unused physical memory back to the OS kernel, like the malloc_trim API from glibc. Default value is False.

  Note that the memory trimming operation is a best-effort operation and may not release any memory at all.

delete(object_meta: ObjectMeta, force: bool = false, deep: bool = true,

memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:

object_meta – The corresponding object meta to delete.

delete(object: Object, force: bool = false, deep: bool = true,

memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:

object – The corresponding object meta to delete.

delete(object_id: ObjectID or List[ObjectID], force: bool = false,

deep: bool = true, memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:

• object_id – ObjectID or list of ObjectID Objects that will be deleted. The object_id can be a single ObjectID, or a list of ObjectID.

• force – bool Forcedly delete an object means the member will be recursively deleted even if the member object is also referred by others. The default value is True.

• deep –

  bool Deeply delete an object means we will deleting the members recursively. The default value is True.

  Note that when deleting objects which have direct blob members, the processing on those blobs yields a “deep” behavior.

• memory_trim –

  bool Whether to trim the memory pool inside the shared memory allocator to return the unused physical memory back to the OS kernel, like the malloc_trim API from glibc. Default value is False.

  Note that the memory trimming operation is a best-effort operation and may not release any memory at all.

delete(object_meta: ObjectMeta, force: bool = false, deep: bool = true,

memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:

object_meta – The corresponding object meta to delete.

delete(object: Object, force: bool = false, deep: bool = true,

memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:

object – The corresponding object meta to delete.

delete(object_id: ObjectID or List[ObjectID], force: bool = false,

deep: bool = true, memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:

• object_id – ObjectID or list of ObjectID Objects that will be deleted. The object_id can be a single ObjectID, or a list of ObjectID.

• force – bool Forcedly delete an object means the member will be recursively deleted even if the member object is also referred by others. The default value is True.

• deep –

  bool Deeply delete an object means we will deleting the members recursively. The default value is True.

  Note that when deleting objects which have direct blob members, the processing on those blobs yields a “deep” behavior.

• memory_trim –

  bool Whether to trim the memory pool inside the shared memory allocator to return the unused physical memory back to the OS kernel, like the malloc_trim API from glibc. Default value is False.

  Note that the memory trimming operation is a best-effort operation and may not release any memory at all.

delete(object_meta: ObjectMeta, force: bool = false, deep: bool = true,

memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:

object_meta – The corresponding object meta to delete.

delete(object: Object, force: bool = false, deep: bool = true,

memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:

object – The corresponding object meta to delete.

drop_name()

drop_name(name: str or ObjectName) → None

Remove the association of the given name.

Parameters:

name – str The name that will be removed.

exists()

exists(object_id: ObjectID) → bool

Whether the given object exists.

Parameters:

object_id – ObjectID The object id to check if exists.

Returns:

True when the specified object exists.

Return type:

bool

find_shared_memory()

find_shared_memory(target: ptr) → bool

Find the corresponding blob (if exists) of the given pointer.

Parameters:

target – address, in int format The given address.

Returns:

ObjectID

find_shared_memory(target: ptr) → bool

Find the corresponding blob (if exists) of the given pointer.

Parameters:

target – address, in int format The given address.

Returns:

ObjectID

get(object_id: ObjectID | None = None, name: str | None = None, resolver: ResolverContext | None = None, fetch: bool = False, **kwargs)

Get vineyard object as python value.

>>> arr_id = vineyard.ObjectID('00002ec13bc81226')
>>> arr = client.get(arr_id)
>>> arr
array([0, 1, 2, 3, 4, 5, 6, 7])

Parameters:

• client – IPCClient or RPCClient The vineyard client to use.

• object_id – ObjectID The object id that will be obtained from vineyard.

• name – ObjectID The object name that will be obtained from vineyard, ignored if object_id is not None.

• resolver – When retrieving vineyard object, an optional resolver can be specified. If no resolver given, the default resolver context will be used.

• fetch – Whether to trigger a migration when the target object is located on remote instances.

• kw – User-specific argument that will be passed to the builder.

Returns:

A python object that return by the resolver, by resolving an vineyard object.

get_blob()

get_blob(object_id: ObjectID, unsafe: bool = false) → Blob

Getting a blob from vineyard using the given object ID.

Parameters:

• object_id – ObjectID The blob to get.

• unsafe – bool unsafe means getting the blob even the blob is not sealed yet. Default is False.

Returns:

Blob

See also

IPCClient.get_blob RPCClient.get_remote_blob RPCClient.get_remote_blobs

get_blobs()

get_blobs(object_ids: List[ObjectID], unsafe: bool = false) → List[Blob]

Getting blobs from vineyard using the given object IDs.

Parameters:

• object_ids – List[ObjectID] The blobs to get.

• unsafe – bool unsafe means getting the blob even the blob is not sealed yet. Default is False.

Returns:

List[Blob]

See also

IPCClient.get_blobs RPCClient.get_remote_blob RPCClient.get_remote_blobs

get_meta()

get_meta(object_id: ObjectID, sync_remote: bool = False) → ObjectMeta

Get object metadata from vineyard.

Parameters:

• object_id – ObjectID The object id to get.

• sync_remote – bool If the target object is a remote object, code_remote=True will force a meta synchronization on the vineyard server. Default is False.

Returns:

ObjectMeta

get_metas()

get_metas(object_ids: List[ObjectID], sync_remote: bool = False)

-> List[ObjectMeta]

Get metadatas of multiple objects from vineyard.

Parameters:

• object_ids – List[ObjectID] The object ids to get.

• sync_remote – bool If the target object is a remote object, code_remote=True will force a meta synchronization on the vineyard server. Default is False.

Returns:

List[ObjectMeta]

get_name()

get_name(name: str or ObjectName, wait: bool = False) → ObjectID

Get the associated object id of the given name.

Parameters:

• name – str The name that will be queried.

• wait – bool Whether to wait util the name appears, if wait, the request will be blocked until the name been registered.

Returns:

The associated object id with the name.

Return type:

ObjectID

get_object()

get_object(object_id: ObjectID, fetch: bool, sync_remote: bool = True) → Object

Get object from vineyard.

Parameters:

• object_id – ObjectID The object id to get.

• fetch – bool whether to trigger the migration of the object from the remote vineyardd to the connected one.

• sync_remote – bool whether to synchronize the metadata from the metadata service.

Returns:

Object

get_objects()

get_objects(object_ids: List[ObjectID], sync_remote: bool = True) → List[Object]

Get multiple objects from vineyard.

Parameters:

• object_ids – List[ObjectID]

• sync_remote – bool whether to synchronize the metadata from the metadata service.

Returns:

List[Object]

property instance_id

The instance id of the connected vineyard server.

property ipc_socket

The UNIX domain socket location of connected vineyard server.

property is_ipc

Whether the client is connected to vineyard server via UNIX domain socket.

property is_rpc

Whether the client is connected to vineyard server via RPC endpoint.

is_shared_memory()

is_shared_memory(target: ptr) → bool

Check if the address is on the shared memory region.

Parameters:

target – address, in int format The given address.

Returns:

bool

is_shared_memory(target: ptr) → bool

Check if the address is on the shared memory region.

Parameters:

target – address, in int format The given address.

Returns:

bool

list_metadatas()

list_metadatas(pattern: str, regex: bool = False, limit: int = 5,

nobuffer: bool = False) -> List[Object]

List all objects in current vineyard server.

Parameters:

• pattern – str The pattern string that will be matched against the object’s typename.

• regex – bool Whether the pattern is a regex expression, otherwise the pattern will be used as wildcard pattern. Default value is False.

• limit – int The limit to list. Default value is 5.

• nobuffer – bool Whether to fill the buffers in returned object metadatas. Default value is False.

Returns:

List[ObjectMeta]

list_names()

list_names(pattern: str, regex: bool = False, limit: int = 5)

-> Dict[str, ObjectID]

List all names in current vineyard server.

Parameters:

• pattern – str The pattern string that will be matched against the name.

• regex – bool Whether the pattern is a regex expression, otherwise the pattern will be used as wildcard pattern. Default value is False.

• limit – int The limit to list. Default value is 5.

Returns:

Dict[str, ObjectID]

list_objects()

list_objects(pattern: str, regex: bool = False, limit: int = 5)

-> List[Object]

List all objects in current vineyard server.

Parameters:

• pattern – str The pattern string that will be matched against the object’s typename.

• regex – bool Whether the pattern is a regex expression, otherwise the pattern will be used as wildcard pattern. Default value is False.

• limit – int The limit to list. Default value is 5.

Returns:

List[Object]

memory_trim()

memory_trim() → bool

Trim the memory pool inside the shared memory allocator to return the unused physical memory back to the OS kernel, like the malloc_trim API from glibc.

Returns True if it actually released any memory.

property meta

The metadata information of the vineyard server. The value is a nested dict, the first-level key is the instance id, and the second-level key is the cluster metadata fields.

>>> client.meta
{
    14: {
        'hostid': '54058007061210',
        'hostname': '127.0.0.1',
        'timestamp': '6882550126788354072'
    },
    15: {
        'hostid': '48843417291806',
        'hostname': '127.0.0.1',
        'timestamp': '6882568290204737414'
    }
}

persist()

persist(object_id: ObjectID) → None

Persist the object of the given object id. After persisting, the object will be visible by clients that connect to other vineyard server instances.

Parameters:

object_id – ObjectID The object that will be persist.

persist(object_meta: ObjectMeta) → None

Persist the given object.

Parameters:

object_meta – ObjectMeta The object that will be persist.

persist(object: Object) → None

Persist the given object.

Parameters:

object – Object The object that will be persist.

put(value: Any, builder: BuilderContext | None = None, persist: bool = False, name: str | None = None, **kwargs)

Put python value to vineyard.

>>> arr = np.arange(8)
>>> arr_id = client.put(arr)
>>> arr_id
00002ec13bc81226

Parameters:

• client – IPCClient or RPCClient The vineyard client to use.

• value – The python value that will be put to vineyard. Supported python value types are decided by modules that registered to vineyard. By default, python value can be put to vineyard after serialized as a bytes buffer using pickle.

• builder – optional When putting python value to vineyard, an optional builder can be specified to tell vineyard how to construct the corresponding vineyard Object. If not specified, the default builder context will be used to select a proper builder.

• persist – bool, optional If true, persist the object after creation.

• name – str, optional If given, the name will be automatically associated with the resulted object. Note that only take effect when the object is persisted.

• kw – User-specific argument that will be passed to the builder.

Returns:

The result object id will be returned.

Return type:

ObjectID

put_name()

put_name(object: ObjectID or ObjectMeta or Object,

name: str or ObjectName) -> None

Associate the given object id with a name. An ObjectID can be associated with more than one names.

Parameters:

• object_id – ObjectID

• name – str

reset()

reset() → None

Alias of ClientBase.clear().

property rpc_endpoint

The RPC endpoint of the connected vineyard server.

shallow_copy()

shallow_copy(object_id: ObjectID) → ObjectID

Create a shallow copy of the given vineyard object.

Parameters:

object_id – ObjectID The vineyard object that is requested to be shallow-copied.

Returns:

The object id of newly shallow-copied vineyard object.

Return type:

ObjectID

shallow_copy(object_id: ObjectID, extra_metadata: dict) → ObjectID

Create a shallow copy of the given vineyard object, with extra metadata.

Parameters:

• object_id – ObjectID The vineyard object that is requested to be shallow-copied.

• extra_metadata – dict Extra metadata to apply to the newly created object. The fields of extra metadata must be primitive types, e.g., string, number, and cannot be array or dict.

Returns:

The object id of newly shallow-copied vineyard object.

Return type:

ObjectID

property status

The status the of connected vineyard server, returns a InstanceStatus.

See also

InstanceStatus

sync_meta()

sync_meta() → None

Synchronize remote metadata to local immediately.

property version

The version number string of connected vineyard server, in the format of semver: MAJOR.MINOR.PATCH.

class vineyard.RPCClient

RPC client that connects to vineyard instance’s RPC endpoints.

clear()

clear() → None

Drop all objects that visible to the current instance in the vineyard cluster.

close()

Close the client.

property connected

Whether the client instance has been connected to the vineyard server.

create_metadata()

create_metadata(metadata: ObjectMeta | List[ObjectMeta])

-> Union[ObjectMeta, List[ObjectMeta]]

Create metadata in vineyardd.

Parameters:

metadata – ObjectMeta The metadata that will be created on vineyardd.

Returns:

The result created metadata.

Return type:

Union[ObjectMeta, List[ObjectMeta]]

create_metadata(metadata: Union[ObjectMeta, List[ObjectMeta]],

instance_id: InstanceID)

-> Union[ObjectMeta, List[ObjectMeta]]

Create metadata in vineyardd with a specified instance id.

Parameters:

• metadata – ObjectMeta The metadata that will be created on vineyardd.

• instance_id – InstanceID The instance to place the object metadata.

Returns:

The result created metadata.

Return type:

Union[ObjectMeta, List[ObjectMeta]]

create_metadata(metadata: ObjectMeta | List[ObjectMeta])

-> Union[ObjectMeta, List[ObjectMeta]]

Create metadata in vineyardd.

Parameters:

metadata – ObjectMeta The metadata that will be created on vineyardd.

Returns:

The result created metadata.

Return type:

Union[ObjectMeta, List[ObjectMeta]]

create_metadata(metadata: Union[ObjectMeta, List[ObjectMeta]],

instance_id: InstanceID)

-> Union[ObjectMeta, List[ObjectMeta]]

Create metadata in vineyardd with a specified instance id.

Parameters:

• metadata – ObjectMeta The metadata that will be created on vineyardd.

• instance_id – InstanceID The instance to place the object metadata.

Returns:

The result created metadata.

Return type:

Union[ObjectMeta, List[ObjectMeta]]

create_remote_blob()

create_remote_blob(blob_builder: RemoteBlobBuilder) → ObjectMeta

Put the remote blob to connected remote vineyard instance. The blob_builder is assumed to be ready and modification on the blob_builder after creation won’t take effect.

Note that creating remote blobs requires network transfer and may yields significate overhead.

vineyard_rpc_client = vineyard.connect(*vineyard_endpoint.split(':'))

buffer_writer = RemoteBlobBuilder(len(payload))
buffer_writer.copy(0, payload)
blob_meta = vineyard_rpc_client.create_remote_blob(buffer_writer)

Parameters:

blob_builder: RemoteBlobBuilder

The remote blob to create.

Returns:

ObjectMeta

See Also:

RPCClient.get_remote_blob RPCClient.get_remote_blobs

create_remote_blob(blob_builders: List[RemoteBlobBuilder]) → List[ObjectMeta]

Put the remote blobs to connected remote vineyard instance. The blob_builders is assumed to be ready and modification on the blob_builder after creation won’t take effect.

Note that creating remote blobs requires network transfer and may yields significate overhead.

Returns:

List[ObjectMeta]

See Also:

RPCClient.get_remote_blob RPCClient.get_remote_blobs

create_remote_blob(blob_builder: RemoteBlobBuilder) → ObjectMeta

Put the remote blob to connected remote vineyard instance. The blob_builder is assumed to be ready and modification on the blob_builder after creation won’t take effect.

Note that creating remote blobs requires network transfer and may yields significate overhead.

vineyard_rpc_client = vineyard.connect(*vineyard_endpoint.split(':'))

buffer_writer = RemoteBlobBuilder(len(payload))
buffer_writer.copy(0, payload)
blob_meta = vineyard_rpc_client.create_remote_blob(buffer_writer)

Parameters:

blob_builder: RemoteBlobBuilder

The remote blob to create.

Returns:

ObjectMeta

See Also:

RPCClient.get_remote_blob RPCClient.get_remote_blobs

create_remote_blob(blob_builders: List[RemoteBlobBuilder]) → List[ObjectMeta]

Put the remote blobs to connected remote vineyard instance. The blob_builders is assumed to be ready and modification on the blob_builder after creation won’t take effect.

Note that creating remote blobs requires network transfer and may yields significate overhead.

Returns:

List[ObjectMeta]

See Also:

RPCClient.get_remote_blob RPCClient.get_remote_blobs

delete()

delete(object_id: ObjectID or List[ObjectID], force: bool = false,

deep: bool = true, memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:

• object_id – ObjectID or list of ObjectID Objects that will be deleted. The object_id can be a single ObjectID, or a list of ObjectID.

• force – bool Forcedly delete an object means the member will be recursively deleted even if the member object is also referred by others. The default value is True.

• deep –

  bool Deeply delete an object means we will deleting the members recursively. The default value is True.

  Note that when deleting objects which have direct blob members, the processing on those blobs yields a “deep” behavior.

• memory_trim –

  bool Whether to trim the memory pool inside the shared memory allocator to return the unused physical memory back to the OS kernel, like the malloc_trim API from glibc. Default value is False.

  Note that the memory trimming operation is a best-effort operation and may not release any memory at all.

delete(object_meta: ObjectMeta, force: bool = false, deep: bool = true,

memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:

object_meta – The corresponding object meta to delete.

delete(object: Object, force: bool = false, deep: bool = true,

memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:

object – The corresponding object meta to delete.

delete(object_id: ObjectID or List[ObjectID], force: bool = false,

deep: bool = true, memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:

• object_id – ObjectID or list of ObjectID Objects that will be deleted. The object_id can be a single ObjectID, or a list of ObjectID.

• force – bool Forcedly delete an object means the member will be recursively deleted even if the member object is also referred by others. The default value is True.

• deep –

  bool Deeply delete an object means we will deleting the members recursively. The default value is True.

  Note that when deleting objects which have direct blob members, the processing on those blobs yields a “deep” behavior.

• memory_trim –

  bool Whether to trim the memory pool inside the shared memory allocator to return the unused physical memory back to the OS kernel, like the malloc_trim API from glibc. Default value is False.

  Note that the memory trimming operation is a best-effort operation and may not release any memory at all.

delete(object_meta: ObjectMeta, force: bool = false, deep: bool = true,

memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:

object_meta – The corresponding object meta to delete.

delete(object: Object, force: bool = false, deep: bool = true,

memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:

object – The corresponding object meta to delete.

delete(object_id: ObjectID or List[ObjectID], force: bool = false,

deep: bool = true, memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:

• object_id – ObjectID or list of ObjectID Objects that will be deleted. The object_id can be a single ObjectID, or a list of ObjectID.

• force – bool Forcedly delete an object means the member will be recursively deleted even if the member object is also referred by others. The default value is True.

• deep –

  bool Deeply delete an object means we will deleting the members recursively. The default value is True.

  Note that when deleting objects which have direct blob members, the processing on those blobs yields a “deep” behavior.

• memory_trim –

  bool Whether to trim the memory pool inside the shared memory allocator to return the unused physical memory back to the OS kernel, like the malloc_trim API from glibc. Default value is False.

  Note that the memory trimming operation is a best-effort operation and may not release any memory at all.

delete(object_meta: ObjectMeta, force: bool = false, deep: bool = true,

memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:

object_meta – The corresponding object meta to delete.

delete(object: Object, force: bool = false, deep: bool = true,

memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:

object – The corresponding object meta to delete.

delete(object_id: ObjectID or List[ObjectID], force: bool = false,

deep: bool = true, memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:

• object_id – ObjectID or list of ObjectID Objects that will be deleted. The object_id can be a single ObjectID, or a list of ObjectID.

• force – bool Forcedly delete an object means the member will be recursively deleted even if the member object is also referred by others. The default value is True.

• deep –

  bool Deeply delete an object means we will deleting the members recursively. The default value is True.

  Note that when deleting objects which have direct blob members, the processing on those blobs yields a “deep” behavior.

• memory_trim –

  bool Whether to trim the memory pool inside the shared memory allocator to return the unused physical memory back to the OS kernel, like the malloc_trim API from glibc. Default value is False.

  Note that the memory trimming operation is a best-effort operation and may not release any memory at all.

delete(object_meta: ObjectMeta, force: bool = false, deep: bool = true,

memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:

object_meta – The corresponding object meta to delete.

delete(object: Object, force: bool = false, deep: bool = true,

memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:

object – The corresponding object meta to delete.

drop_name()

drop_name(name: str or ObjectName) → None

Remove the association of the given name.

Parameters:

name – str The name that will be removed.

exists()

exists(object_id: ObjectID) → bool

Whether the given object exists.

Parameters:

object_id – ObjectID The object id to check if exists.

Returns:

True when the specified object exists.

Return type:

bool

get(object_id: ObjectID | None = None, name: str | None = None, resolver: ResolverContext | None = None, fetch: bool = False, **kwargs)

Get vineyard object as python value.

>>> arr_id = vineyard.ObjectID('00002ec13bc81226')
>>> arr = client.get(arr_id)
>>> arr
array([0, 1, 2, 3, 4, 5, 6, 7])

Parameters:

• client – IPCClient or RPCClient The vineyard client to use.

• object_id – ObjectID The object id that will be obtained from vineyard.

• name – ObjectID The object name that will be obtained from vineyard, ignored if object_id is not None.

• resolver – When retrieving vineyard object, an optional resolver can be specified. If no resolver given, the default resolver context will be used.

• fetch – Whether to trigger a migration when the target object is located on remote instances.

• kw – User-specific argument that will be passed to the builder.

Returns:

A python object that return by the resolver, by resolving an vineyard object.

get_meta()

get_meta(object_id: ObjectID) → ObjectMeta

Get object metadata from vineyard.

Parameters:

object_id – ObjectID The object id to get.

Returns:

ObjectMeta

get_metas()

get_metas(object_ids: List[ObjectID] -> List[ObjectMeta]

Get metadatas of multiple objects from vineyard.

Parameters:

object_ids – List[ObjectID]

Returns:

List[ObjectMeta]

get_name()

get_name(name: str or ObjectName, wait: bool = False) → ObjectID

Get the associated object id of the given name.

Parameters:

• name – str The name that will be queried.

• wait – bool Whether to wait util the name appears, if wait, the request will be blocked until the name been registered.

Returns:

The associated object id with the name.

Return type:

ObjectID

get_object()

get_object(object_id: ObjectID, sync_remote: bool = True) → Object

Get object from vineyard.

Parameters:

• object_id – ObjectID The object id to get.

• sync_remote – bool whether to synchronize the metadata from the metadata service.

Returns:

Object

get_objects()

get_objects(object_ids: List[ObjectID], sync_remote: bool = True) → List[Object]

Get multiple objects from vineyard.

Parameters:

• object_ids – List[ObjectID]

• sync_remote – bool whether to synchronize the metadata from the metadata service.

Returns:

List[Object]

get_remote_blob()

get_remote_blob(object_id: ObjectID, unsafe: bool = false) → RemoteBlob

Getting a remote blob from vineyard using the given object ID.

Note that getting remote blobs requires network transfer and may yields significate overhead.

Parameters:

• object_id – ObjectID The remote blob to get.

• unsafe – bool unsafe means getting the blob even the blob is not sealed yet. Default is False.

Returns:

RemoteBlob

See also

IPCClient.get_blob IPCClient.get_blobs RPCClient.get_remote_blobs

get_remote_blobs()

get_remote_blobs(object_ids: List[ObjectID], unsafe: bool = false)

-> List[RemoteBlob]

Getting remote blobs from vineyard using the given object IDs.

Note that getting remote blobs requires network transfer and may yields significate overhead.

Parameters:

• object_ids – List[ObjectID] The remote blobs to get.

• unsafe – bool unsafe means getting the blob even the blob is not sealed yet. Default is False.

Returns:

List[RemoteBlob]

See also

IPCClient.get_blob IPCClient.get_blobs RPCClient.get_remote_blob

property instance_id

The instance id of the connected vineyard server.

property ipc_socket

The UNIX domain socket location of connected vineyard server.

is_fetchable()

Whether the rpc client is able to fetch objects from the connected vineyard server. When the instance connected by the rpc client is not the same as the instance of metadata, the rpc client is not able to fetch the object.

property is_ipc

Whether the client is connected to vineyard server via UNIX domain socket.

property is_rpc

Whether the client is connected to vineyard server via RPC endpoint.

list_metadatas()

list_metadatas(pattern: str, regex: bool = False, limit: int = 5,

nobuffer: bool = False) -> List[Object]

List all objects in current vineyard server.

Parameters:

• pattern – str The pattern string that will be matched against the object’s typename.

• regex – bool Whether the pattern is a regex expression, otherwise the pattern will be used as wildcard pattern. Default value is False.

• limit – int The limit to list. Default value is 5.

• nobuffer – bool Whether to fill the buffers in returned object metadatas. Default value is False.

Returns:

List[ObjectMeta]

list_names()

list_names(pattern: str, regex: bool = False, limit: int = 5)

-> Dict[str, ObjectID]

List all names in current vineyard server.

Parameters:

• pattern – str The pattern string that will be matched against the name.

• regex – bool Whether the pattern is a regex expression, otherwise the pattern will be used as wildcard pattern. Default value is False.

• limit – int The limit to list. Default value is 5.

Returns:

Dict[str, ObjectID]

list_objects()

list_objects(pattern: str, regex: bool = False, limit: int = 5)

-> List[Object]

List all objects in current vineyard server.

Parameters:

• pattern – str The pattern string that will be matched against the object’s typename.

• regex – bool Whether the pattern is a regex expression, otherwise the pattern will be used as wildcard pattern. Default value is False.

• limit – int The limit to list. Default value is 5.

Returns:

List[Object]

memory_trim()

memory_trim() → bool

Trim the memory pool inside the shared memory allocator to return the unused physical memory back to the OS kernel, like the malloc_trim API from glibc.

Returns True if it actually released any memory.

property meta

The metadata information of the vineyard server. The value is a nested dict, the first-level key is the instance id, and the second-level key is the cluster metadata fields.

>>> client.meta
{
    14: {
        'hostid': '54058007061210',
        'hostname': '127.0.0.1',
        'timestamp': '6882550126788354072'
    },
    15: {
        'hostid': '48843417291806',
        'hostname': '127.0.0.1',
        'timestamp': '6882568290204737414'
    }
}

persist()

persist(object_id: ObjectID) → None

Persist the object of the given object id. After persisting, the object will be visible by clients that connect to other vineyard server instances.

Parameters:

object_id – ObjectID The object that will be persist.

persist(object_meta: ObjectMeta) → None

Persist the given object.

Parameters:

object_meta – ObjectMeta The object that will be persist.

persist(object: Object) → None

Persist the given object.

Parameters:

object – Object The object that will be persist.

put(value: Any, builder: BuilderContext | None = None, persist: bool = False, name: str | None = None, **kwargs)

Put python value to vineyard.

>>> arr = np.arange(8)
>>> arr_id = client.put(arr)
>>> arr_id
00002ec13bc81226

Parameters:

• client – IPCClient or RPCClient The vineyard client to use.

• value – The python value that will be put to vineyard. Supported python value types are decided by modules that registered to vineyard. By default, python value can be put to vineyard after serialized as a bytes buffer using pickle.

• builder – optional When putting python value to vineyard, an optional builder can be specified to tell vineyard how to construct the corresponding vineyard Object. If not specified, the default builder context will be used to select a proper builder.

• persist – bool, optional If true, persist the object after creation.

• name – str, optional If given, the name will be automatically associated with the resulted object. Note that only take effect when the object is persisted.

• kw – User-specific argument that will be passed to the builder.

Returns:

The result object id will be returned.

Return type:

ObjectID

put_name()

put_name(object: ObjectID or ObjectMeta or Object,

name: str or ObjectName) -> None

Associate the given object id with a name. An ObjectID can be associated with more than one names.

Parameters:

• object_id – ObjectID

• name – str

property remote_instance_id

The instance id of the connected remote vineyard server.

reset()

reset() → None

Alias of ClientBase.clear().

property rpc_endpoint

The RPC endpoint of the connected vineyard server.

shallow_copy()

shallow_copy(object_id: ObjectID) → ObjectID

Create a shallow copy of the given vineyard object.

Parameters:

object_id – ObjectID The vineyard object that is requested to be shallow-copied.

Returns:

The object id of newly shallow-copied vineyard object.

Return type:

ObjectID

shallow_copy(object_id: ObjectID, extra_metadata: dict) → ObjectID

Create a shallow copy of the given vineyard object, with extra metadata.

Parameters:

• object_id – ObjectID The vineyard object that is requested to be shallow-copied.

• extra_metadata – dict Extra metadata to apply to the newly created object. The fields of extra metadata must be primitive types, e.g., string, number, and cannot be array or dict.

Returns:

The object id of newly shallow-copied vineyard object.

Return type:

ObjectID

property status

The status the of connected vineyard server, returns a InstanceStatus.

See also

InstanceStatus

sync_meta()

sync_meta() → None

Synchronize remote metadata to local immediately.

property version

The version number string of connected vineyard server, in the format of semver: MAJOR.MINOR.PATCH.

Vineyard cluster

class vineyard.InstanceStatus

InstanceStatus represents the status of connected vineyard instance, including the instance identity, memory statistics and workloads on this instance.

>>> status = client.status
>>> print(status)
InstanceStatus:
    instance_id: 5
    deployment: local
    memory_usage: 360
    memory_limit: 268435456
    deferred_requests: 0
    ipc_connections: 1
    rpc_connections: 0
>>> status.instance_id
5
>>> status.deployment
'local'
>>> status.memory_usage
360
>>> status.memory_limit
268435456
>>> status.deferred_requests
0
>>> status.ipc_connections
1
>>> status.rpc_connections
0

__init__(*args, **kwargs)

__repr__()

Return repr(self).

__str__()

Return str(self).

property deferred_requests

Report number of waiting requests of current vineyardd instance.

property deployment

The deployment mode of the connected vineyardd cluster, can be "local" and "distributed".

property instance_id

Return the instance id of vineyardd that the client is connected to.

property ipc_connections

Report number of alive IPC connections on the current vineyardd instance.

property memory_limit

Report memory limit (in bytes) of current vineyardd instance.

property memory_usage

Report memory usage (in bytes) of current vineyardd instance.

property rpc_connections

Report number of alive RPC connections on the current vineyardd instance.

Blob

class vineyard.Blob

Blob in vineyard is a consecutive readonly shared memory.

property address

The memory address value of this blob.

static empty()

Create an empty blob (with size as 0).

property is_empty

Whether the blob is an empty blob, i.e., the size of this blob is 0.

property size

Size of the blob.

class vineyard.BlobBuilder

BlobBuilder is the builder for creating a finally immutable blob in vineyard server.

A BlobBuilder can only be explicitly created using the IPCClient.create_blob().

See also

IPCClient.create_blob IPCClient.create_empty_blob

abort()

Abort the blob builder if it is not sealed yet.

property address

The memory address value of this blob builder.

copy()

copy(self, offset: int, ptr: int, size: int, concurrency: int = 6)

Copy the given address to the given offset.

copy(self, offset: int, ptr: int, size: int, concurrency: int = 6)

Copy the given address to the given offset.

copy(self, offset: int, ptr: int, size: int, concurrency: int = 6)

Copy the given address to the given offset.

property id

ObjectID of this blob builder.

shrink()

Shrink the blob builder to the given size if it is not sealed yet.

property size

Size of this blob builder.

class vineyard.RemoteBlob

RemoteBlob is a holder for Blob in cases like the Blob doesn’t exist in the local vineyardd instance but the client still want th access the data with a tolerable network-transfer overhead.

property address

The memory address value of this blob.

property id

Object ID of the remote blob.

property instance_id

The instance ID where the blob is actually placed at.

property is_empty

Whether the blob is an empty blob, i.e., the size of this blob is 0.

property size

The size of this blob.

class vineyard.RemoteBlobBuilder

RemoteBlobBuilder is the builder for creating a finally immutable blob in vineyard server over a RPC client.

A RemoteBlobBuilder can only be explicitly initialized using RemoteBlobBuilder(), then be filled the content and finally be sent to remote vineyard instance over the network.

See also

IPCClient.create_blob IPCClient.create_empty_blob

abort()

Abort the blob builder if it is not sealed yet.

property address

The memory address value of this blob builder.

copy()

copy(self, offset: int, ptr: int, size: int, concurrency: int = 6)

Copy the given address to the given offset.

copy(self, offset: int, ptr: int, size: int, concurrency: int = 6)

Copy the given address to the given offset.

copy(self, offset: int, ptr: int, size: int, concurrency: int = 6)

Copy the given address to the given offset.

property size

Size of this blob builder.

Resolvers and builders

class vineyard.core.resolver.ResolverContext(parent_context: ResolverContext | None = None)

vineyard.core.resolver.get_current_resolvers() → ResolverContext

Obtain current resolver context.

vineyard.core.resolver.resolver_context(resolvers: Dict[str, Callable] | None = None, base: ResolverContext | None = None) → Generator[ResolverContext, Any, Any]

Open a new context for register resolvers, without populating outside the global environment.

The resolver_context can be useful when users have more than more resolver for a certain type, e.g., the vineyard::Tensor object can be resolved as numpy.ndarray or xgboost::DMatrix.

We could have

def numpy_resolver(obj):
    ...

default_resolver_context.register('vineyard::Tensor', numpy_resolver)

and

def xgboost_resolver(obj):
    ...

default_resolver_context.register('vineyard::Tensor', xgboost_resolver)

Obviously there’s a conflict, and the stackable resolver_context could help there,

with resolver_context({'vineyard::Tensor', xgboost_resolver}):
    ...

Assuming the default context resolves vineyard::Tensor to numpy.ndarray, inside the with resolver_context the vineyard::Tensor will be resolved to xgboost::DMatrix, and after exiting the context the global environment will be restored back as default.

The with resolver_context is nestable as well.

See also

builder_context driver_context

class vineyard.core.builder.BuilderContext(parent_context: BuilderContext | None = None)

register(type_id: type, builder: Callable)

Register a Python type to the builder context.

Parameters:

• type_id (Python type) – Like int, or numpy.ndarray

• builder (callable, e.g., a method, callable object) – A builder translates a python object to vineyard, it accepts a Python value as parameter, and returns an vineyard object as result.

run(client, value, **kw)

Follows the MRO to find the proper builder for given python value.

Here “Follows the MRO” implies:

• If the type of python value has been found in the context, the registered builder will be used.

• If not, it follows the MRO chain from down to top to find a registered Python type and used the associated builder.

• When the traversal reaches the object type, since there’s a default builder that serialization the python value, the parameter will be serialized and be put into a blob.

vineyard.core.builder.get_current_builders() → BuilderContext

Obtain the current builder context.

vineyard.core.builder.builder_context(builders: Dict[type, Callable] | None = None, base: BuilderContext | None = None) → Generator[BuilderContext, Any, Any]

Open a new context for register builders, without populating outside global environment.

See also

resolver_context driver_context

class vineyard.core.driver.DriverContext

vineyard.core.driver.get_current_drivers()

Obtain current driver context.

vineyard.core.driver.driver_context(drivers=None, base=None)

Open a new context for register drivers, without populting outside global environment.

See also

builder_context resolver_context

Shared memory

class vineyard.shared_memory.SharedMemory(vineyard_client, name=None, create=False, size=0)

property buf

A memoryview of contents of the shared memory block.

freeze()

Seal the shared memory to make it visible for other processes.

property name

Unique name that identifies the shared memory block.

property size

Size in bytes.

unlink()

Requests that the underlying shared memory block be destroyed.

class vineyard.shared_memory.ShareableList(vineyard_client, sequence=None, *, name=None)

Pattern for a mutable list-like object shareable via a shared memory block. It differs from the built-in list type in that these lists can not change their overall length (i.e. no append, insert, etc.)

Because values are packed into a memoryview as bytes, the struct packing format for any storable value must require no more than 8 characters to describe its format.

The ShareableList in vineyard differs slightly with its equivalent in the multiprocessing.shared_memory.ShareableList, as it becomes immutable after obtaining from the vineyard backend.

See also

multiprocessing.shared_memory.ShareableList

freeze()

Make the shareable list immutable and visible for other vineyard clients.

Deployment

vineyard.connect(*args, **kwargs)

Connect to vineyard by specified UNIX-domain socket or TCP endpoint.

If no arguments are provided and failed to resolve both the environment variables VINEYARD_IPC_SOCKET, VINEYARD_RPC_ENDPOINT, VINEYARD_CONFIG, and the default configuration file /var/run/vineyard-config.yaml and /var/run/vineyard/vineyard-config.yaml, it will launch a standalone vineyardd server in the background and then connect to it.

The connect() method has various overloading:

connect(socket: str,

username: str = None,

password: str = None) -> IPCClient

Connect to vineyard via UNIX domain socket for IPC service:

client = vineyard.connect('/var/run/vineyard.sock')

Parameters:

socket: str

UNIX domain socket path to setup an IPC connection.

username: str

Username to login, default to None.

password: str

Password to login, default to None.

Returns:

IPCClient: The connected IPC client.

connect(host: str,

port: int or str,

username: str = None,

password: str = None) -> RPCClient

Connect to vineyard via TCP socket.

Parameters:

host: str

Hostname to connect to.

port: int or str

The TCP that listened by vineyard TCP service.

username: str

Username to login, default to None.

password: str

Password to login, default to None.

Returns:

RPCClient: The connected RPC client.

connect(endpoint: (str, int or str),

username: str = None,

password: str = None) -> RPCClient

Connect to vineyard via TCP socket.

Parameters:

endpoint: tuple(str, int or str)

Endpoint to connect to. The parameter is a tuple, in which the first element is the host, and the second parameter is the port (can be int a str).

username: str

Username to login, default to None.

password: str

Password to login, default to None.

Returns:

RPCClient: The connected RPC client.

connect(username: str = None,

password: str = None) -> IPCClient or RPCClient

Connect to vineyard via UNIX domain socket or TCP endpoint. This method normally usually no arguments, and will first tries to resolve IPC socket from the environment variable VINEYARD_IPC_SOCKET and connect to it. If it fails to establish a connection with vineyard server, the method will tries to resolve RPC endpoint from the environment variable VINEYARD_RPC_ENDPOINT. If both tries are failed, this method will try to resolve the configuration file that contains IPC socket and RPC endpoint from the environment variable VINEYARD_CONFIG, and then connect to the vineyard server with the resolved configuration.

If above all are failed, this method will raise a ConnectionFailed exception.

In rare cases, user may be not sure about if the IPC socket or RPC endpoint is available, i.e., the variable might be None. In such cases this method can accept a None as arguments, and do resolution as described above.

Parameters:

username: str

Username to login, default to None.

password: str

Password to login, default to None.

Raises:

ConnectionFailed

vineyard.get_current_socket() → str

Get current vineyard UNIX-domain socket established by vineyard.try_init().

Raises:

ValueError if vineyard is not initialized. –

vineyard.deploy.local.start_vineyardd(meta: str | None = 'etcd', etcd_endpoints: str | None = None, etcd_prefix: str | None = None, vineyardd_path: str | None = None, size: str | None = '', socket: str | None = None, rpc: str | None = True, rpc_socket_port: str | None = 9600, debug=False) → Generator[Tuple[Popen, str, str], None, None]

Launch a local vineyard cluster.

Parameters:

• meta – str, optional. Metadata backend, can be “etcd”, “redis” and “local”. Defaults to “etcd”.

• etcd_endpoint – str Launching vineyard using specified etcd endpoints. If not specified, vineyard will launch its own etcd instance.

• etcd_prefix – str Specify a common prefix to establish a local vineyard cluster.

• vineyardd_path – str Location of vineyard server program. If not specified, vineyard will use its own bundled vineyardd binary.

• size –

  int The memory size limit for vineyard’s shared memory. The memory size can be a plain integer or as a fixed-point number using one of these suffixes:

  E, P, T, G, M, K.
  

  You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi, Mi, Ki. Defaults to “”, means not limited.

  For example, the following represent roughly the same value:

  128974848, 129k, 129M, 123Mi, 1G, 10Gi, ...
  

• socket –

  str The UNIX domain socket socket path that vineyard server will listen on. Default is None.

  When the socket parameter is None, a random path under temporary directory will be generated and used.

• rpc_socket_port – int The port that vineyard will use to provided RPC service.

• debug – bool Whether print debug logs.

Returns:

Yields a tuple with the subprocess as the first element and the UNIX-domain IPC socket as the second element.

Return type:

(proc, socket)

vineyard.deploy.distributed.start_vineyardd(hosts=None, etcd_endpoints=None, vineyardd_path=None, size='', socket='/var/run/vineyard.sock', rpc_socket_port=9600, debug=False)

Launch a local vineyard cluster in a distributed fashion.

Parameters:

• hosts – list of str A list of machines to launch vineyard server.

• etcd_endpoint – str Launching vineyard using specified etcd endpoints. If not specified, vineyard will launch its own etcd instance.

• vineyardd_path – str Location of vineyard server program. If not specified, vineyard will use its own bundled vineyardd binary.

• size –

  int The memory size limit for vineyard’s shared memory. The memory size can be a plain integer or as a fixed-point number using one of these suffixes:

  E, P, T, G, M, K.
  

  You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi, Mi, Ki. Defaults to “”, means not limited.

  For example, the following represent roughly the same value:

  128974848, 129k, 129M, 123Mi, 1G, 10Gi, ...
  

• socket – str The UNIX domain socket socket path that vineyard server will listen on.

• rpc_socket_port – int The port that vineyard will use to privode RPC service.

• debug – bool Whether print debug logs.

vineyard.deploy.kubernetes.start_vineyardd(namespace='vineyard', size='512Mi', socket='/var/run/vineyard.sock', rpc_socket_port=9600, vineyard_image='vineyardcloudnative/vineyardd:latest', vineyard_image_pull_policy='IfNotPresent', vineyard_image_pull_secrets=None, k8s_client=None)

Launch a vineyard cluster on kubernetes.

Parameters:

• namespace – str namespace in kubernetes

• size –

  int The memory size limit for vineyard’s shared memory. The memory size can be a plain integer or as a fixed-point number using one of these suffixes:

  E, P, T, G, M, K.
  

  You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi, Mi, Ki.

  For example, the following represent roughly the same value:

  128974848, 129k, 129M, 123Mi, 1G, 10Gi, ...
  

• socket – str The UNIX domain socket socket path that vineyard server will listen on.

• rpc_socket_port – int The port that vineyard will use to provide RPC service.

• k8s_client – kubernetes.client.api.ApiClient A kubernetes client. If not specified, vineyard will try to resolve the kubernetes configuration from current context.

• vineyard_image – str The docker image of vineyardd to launch the daemonset.

• vineyard_image_pull_policy – str The docker image pull policy of vineyardd.

• vineyard_image_pull_secrets – str The docker image pull secrets of vineyardd.

Returns:

A list of created kubernetes resources during the deploying process. The resources can be later release using delete_kubernetes_objects().

See also

vineyard.deploy.kubernetes.delete_kubernetes_objects

vineyard.deploy.kubernetes.delete_kubernetes_objects(targets, k8s_client=None, verbose=False, wait=False, timeout_seconds=60, **kwargs)

Delete the given kubernetes resources.

Parameters:

• target – List List of Kubernetes objects

• k8s_client – The kubernetes client. If not specified, vineyard will try to resolve the kubernetes configuration from current context.

• verbose – bool Whether to print the deletion logs.

• wait – bool Whether to wait for the deletion to complete.

• timeout_seconds – int The timeout in seconds for waiting for the deletion to complete.

See also

vineyard.deploy.kubernetes.start_vineyardd vineyard.deploy.kubernetes.delete_kubernetes_object

I/O Drivers

vineyard.io.open(path, *args, mode='r', handlers=None, chunk_hook: Callable | None = None, **kwargs)

Open a path as a reader or writer, depends on the parameter mode. If mode is r, it will open a stream for read, and open a stream for write when mode is w.

Parameters:

• path (str) – Path to open.

• mode (char) – Mode about how to open the path, r is for read and w for write.

• handlers – A dict that will be filled with a handler that contains the process handler of the underlying read/write process that can be joined using join to capture the possible errors during the I/O proceeding.

• chunk_hook (callable, optional) –

  If the read/write target is a global dataframe (e.g., csv, orc, parquet, etc.), the hook will be called for each chunk to be read or write (usually a pyarrow.RecordBatch). The hook should return a pyarrow.RecordBatch object and should be stateless as the invoke order is not guaranteed. E.g.,

  def exchange_column(batch):
      import pyarrow as pa
  
      columns = batch.columns
      first = columns[0]
      second = columns[1]
      columns = [second, first] + columns[2:]
      return pa.RecordBatch.from_arrays(columns, schema=batch.schema)
  

• vineyard_ipc_socket (str) – Vineyard’s IPC socket location.

• vineyard_endpoint (str) – Vineyard’s RPC socket address.

See also

vineyard.io.read, vineyard.io.write

vineyard.io.read(path, *args, handlers=None, accumulate=False, chunk_hook: Callable | None = None, **kwargs)

Open a path and read it as a single stream.

Parameters:

• path (str) – Path to read, the last reader registered for the scheme of the path will be used.

• handlers (list, optional) – If handlers is not None, launched worker processes will be emplaced into the list for further customized job lifecycle management. Default is None.

• accumulate (bool, optional) – If accumulate is True, it will return a data frame, rather than dataframe stream. Default is False.

• chunk_hook (callable, optional) –

  If the read/write target is a global dataframe (e.g., csv, orc, parquet, etc.), the hook will be called for each chunk to be read or write (usually a pyarrow.RecordBatch). The hook should return a pyarrow.RecordBatch object and should be stateless as the invoke order is not guaranteed. E.g.,

  def exchange_column(batch):
      import pyarrow as pa
  
      columns = batch.columns
      first = columns[0]
      second = columns[1]
      columns = [second, first] + columns[2:]
      return pa.RecordBatch.from_arrays(columns, schema=batch.schema)
  

• vineyard_ipc_socket (str) – The local or remote vineyard’s IPC socket location that the remote readers will use to establish connections with the vineyard server.

• vineyard_endpoint (str, optional) – An optional address of vineyard’s RPC socket, which will be used for retrieving server’s information on the client side. If not provided, the vineyard_ipc_socket will be used, or it will tries to discovery vineyard’s IPC or RPC endpoints from environment variables.

vineyard.io.write(path, stream, *args, handlers=None, chunk_hook: Callable | None = None, **kwargs)

Write the stream to a given path.

Parameters:

• path (str) – Path to write, the last writer registered for the scheme of the path will be used.

• stream (vineyard stream) – Stream that produces the data to write.

• handlers (list, optional) – If handlers is not None, launched worker processes will be emplaced into the list for further customized job lifecycle management. Default is None.

• chunk_hook (callable, optional) –

  If the read/write target is a global dataframe (e.g., csv, orc, parquet, etc.), the hook will be called for each chunk to be read or write (usually a pyarrow.RecordBatch). The hook should return a pyarrow.RecordBatch object and should be stateless as the invoke order is not guaranteed. E.g.,

  def exchange_column(batch):
      import pyarrow as pa
  
      columns = batch.columns
      first = columns[0]
      second = columns[1]
      columns = [second, first] + columns[2:]
      return pa.RecordBatch.from_arrays(columns, schema=batch.schema)
  

• vineyard_ipc_socket (str) – The local or remote vineyard’s IPC socket location that the remote readers will use to establish connections with the vineyard server.

• vineyard_endpoint (str, optional) – An optional address of vineyard’s RPC socket, which will be used for retrieving server’s information on the client side. If not provided, the vineyard_ipc_socket will be used, or it will tries to discovery vineyard’s IPC or RPC endpoints from environment variables.

Streams

class vineyard.io.byte.ByteStream(meta: ObjectMeta, params: Dict | None = None)

class vineyard.io.dataframe.DataframeStream(meta: ObjectMeta, params: Dict | None = None)

class vineyard.io.recordbatch.RecordBatchStream(meta: ObjectMeta, params: Dict[str, Any] | None = None)

Interacting with the CSI Driver

vineyard.csi.read(path: str)

Read vineyard object from path, and return python value. Notice, the API is only used for CSI driver.

Parameters:

path – str The path that represents a vineyard object.

>>> arr = vineyard.read('/a/b/c/d/f')
>>> arr
array([0, 1, 2, 3, 4, 5, 6, 7])

Returns:

A python object that return by the resolver, by resolving an vineyard object.

vineyard.csi.write(value: Any, path: str)

Write python value to vineyard. Notice, the API is only used for CSI driver.

Parameters:

path – str The path that represents a vineyard object.

>>> arr = np.arange(8)
>>> vineyard.write(arr)

Vineyard Cli Tool

You can also use the Python API to interact with internal Vineyard Cli Tool.

$ python -m vineyard.cli <command> [options]

Vineyard LLM KV Cache

Before using the KV Cache, you need to install the vineyard-llm package.