Python API Reference#

Objects#

class vineyard.ObjectID#

Opaque type for vineyard’s object id. The object ID is generated by vineyard server, the underlying type of ObjectID is a 64-bit unsigned integer. Wrapper utilities are provided to interact with the external python world.

>>> id = ObjectID("000043c5c6d5e646")
>>> id
000043c5c6d5e646
>>> repr(id)
'000043c5c6d5e646'
>>> print(id)
ObjectID <"000043c5c6d5e646">
>>> int(id)
74516723525190

__eq__()#: Return self==value.

__hash__()#: Return hash(self).

__init__()#

__repr__()#: Return repr(self).

__str__()#: Return str(self).

class vineyard.Object#

Base class for vineyard objects.

property id#: The object id of this object.

property isglobal#: Whether the object is a global object.

property islocal#: Whether the object is a local object.

property ispersist#: Whether the object is a persistent object. The word “persistent” means the object could be seen by clients that connect to other vineyard server instances.

member()#

member(self, name: str) → Object

Get the member object of this object.

Parameters:: name – str The name of the member object.
Returns:: The member object.
Return type:: Object

See also

ObjectMeta.get, ObjectMeta.__getitem__

property meta#: The metadata of this object.

property nbytes#: The nbytes of this object.

property signature#: The object signature of this object.

property typename#: The typename of this object. typename is the string value of the C++ type, e.g., vineyard::Array<int>, vineyard::Table.

class vineyard.ObjectBuilder#

Metadata#

class vineyard.ObjectMeta#

ObjectMeta is the type for metadata of an Object. The ObjectMeta can be treat as a dict-like type. If the the metadata if the metadata obtained from vineyard, the metadata is readonly. Otherwise key-value attributes or object members could be associated with the metadata to construct a new vineyard object.

We can inspect the key-value attributes and members of an ObjectMeta:

>>> meta = client.get_meta(hashmap_id)
>>> meta
ObjectMeta {
    "id": "0000347aebe92dd0",
    "instance_id": "0",
    ...
}
>>> meta['num_elements_']
'5'
>>> meta['entries']
ObjectMeta {
    "id": "0000347aebe92dd0",
    "instance_id": "0",
    ...
}

ObjectMeta value can be iterated over:

>>> list(k for k in meta['entries'])
['transient', 'num_slots_minus_one_', 'max_lookups_', 'num_elements_', 'entries_',
 'nbytes', 'typename', 'instance_id', 'id']

__contains__()#

Check if given key exists in the object metadata.

Parameters:: key – str The name to be looked up.
Returns:: True if the queried key exists in this object metadata, otherwise False.
Return type:: bool

__getitem__()#

__getitem__(self, key: str) → string or Object

Get meta or member’s meta from metadata.

Parameters:

key – str The name to be looked up.

Returns:

string – If the given key is a key of meta, returns the meta value.
Object – If the given key is a key of member, return the meta of this member.

__init__()#

__init__(global_: bool = False)

Create an empty metadata, the metadata will be used to create a vineyard object.

Parameters: global_: bool, if the object meta is for creating a global object.

__repr__()#: Return repr(self).

__setitem__()#

__setitem__(self, key: str, value) → None

Add a metadata to the ObjectMeta.

Parameters:

key – str The name of the new metadata entry.
value –
str, int, float, bool or list of int The value of the new metadata entry.
- When the value is a str, it will be convert to string at first.
- When the value is a list of str, int or float, it will be first dumpped as string using json.dumps.

__setitem__(self, key: str, ObjectID, Object or ObjectMeta) → None

Add a member object.

Parameters:

key – str The name of the member object.
object – Object, ObjectID or ObjectMeta The reference to the member object or the object id of the member object.

__str__()#: Return str(self).

add_member()#

add_member(self, key: str, ObjectID, Object or ObjectMeta) → None

Add a member object.

Parameters:

key – str The name of the member object.
object – Object, ObjectID or ObjectMeta The reference to the member object or the object id of the member object.

get()#

get(self, key: str, default=None) → string or Object

Get meta or member’s meta from metadata, return default value if the given key is not presented.

Parameters:

key – str The name to be looked up.

Returns:

str – When the given key belongs to a metadata pair. Note that the metadata value of type int or float will be returned in string format as well.
ObjectMeta – When the given key is mapped to a member object.
See Also – ObjectMeta.__getitem__

get_member()#

get_member(self, key: str) → Object

Get member object from metadata, return None if the given key is not presented, and raise exception RuntimeError if the given key is associated with a plain metadata, rather than member object.

Parameters:: key – str The name to be looked up.
Raises:: RuntimeError – When the given key is associated with a plain metadata, rather than member object.

See also

ObjectMeta.__getitem__, ObjectMeta.get

property id#: The corresponding object ID of this metadata.

property instance_id#: The "instance_id" of vineyard instance that the metadata been placed on.

property isglobal#: True if the object is a global object, otherwise a local object or remote object.

property islocal#: True if the object is a local object, otherwise a global object or remote object.

member()#

get_member(self, key: str) → Object

Get member object from metadata, return None if the given key is not presented, and raise exception RuntimeError if the given key is associated with a plain metadata, rather than member object.

Parameters:: key – str The name to be looked up.
Raises:: RuntimeError – When the given key is associated with a plain metadata, rather than member object.

See also

ObjectMeta.__getitem__, ObjectMeta.get

property memory_usage#: Get the total memory usage of buffers in this object meta.

property nbytes#: The "nbytes" attribute of this metadata.

set_global()#

Mark the building object as a global object.

Parameters:: global – bool, default is True

property signature#: The corresponding object signature of this metadata.

property typename#: The "typename" attribute of this metadata.

Vineyard client#

vineyard.connect(*args, **kwargs)[source]#

Connect to vineyard by specified UNIX-domain socket or TCP endpoint.

If no arguments are provided and failed to resolve both the environment variables VINEYARD_IPC_SOCKET, VINEYARD_RPC_ENDPOINT, VINEYARD_CONFIG, and the default configuration file /var/run/vineyard-config.yaml and /var/run/vineyard/vineyard-config.yaml, it will launch a standalone vineyardd server in the background and then connect to it.

The connect() method has various overloading:

connect(socket: str,

username: str = None,

password: str = None) -> IPCClient

Connect to vineyard via UNIX domain socket for IPC service:

client = vineyard.connect('/var/run/vineyard.sock')

Parameters:

socket: str: UNIX domain socket path to setup an IPC connection.
username: str: Username to login, default to None.
password: str: Password to login, default to None.

Returns:

IPCClient: The connected IPC client.

connect(host: str,

port: int or str,

username: str = None,

password: str = None) -> RPCClient

Connect to vineyard via TCP socket.

Parameters:

host: str: Hostname to connect to.
port: int or str: The TCP that listened by vineyard TCP service.
username: str: Username to login, default to None.
password: str: Password to login, default to None.

Returns:

RPCClient: The connected RPC client.

connect(endpoint: (str, int or str),

username: str = None,

password: str = None) -> RPCClient

Connect to vineyard via TCP socket.

Parameters:

endpoint: tuple(str, int or str): Endpoint to connect to. The parameter is a tuple, in which the first element is the host, and the second parameter is the port (can be int a str).
username: str: Username to login, default to None.
password: str: Password to login, default to None.

Returns:

RPCClient: The connected RPC client.

connect(username: str = None,

password: str = None) -> IPCClient or RPCClient

Connect to vineyard via UNIX domain socket or TCP endpoint. This method normally usually no arguments, and will first tries to resolve IPC socket from the environment variable VINEYARD_IPC_SOCKET and connect to it. If it fails to establish a connection with vineyard server, the method will tries to resolve RPC endpoint from the environment variable VINEYARD_RPC_ENDPOINT. If both tries are failed, this method will try to resolve the configuration file that contains IPC socket and RPC endpoint from the environment variable VINEYARD_CONFIG, and then connect to the vineyard server with the resolved configuration.

If above all are failed, this method will raise a ConnectionFailed exception.

In rare cases, user may be not sure about if the IPC socket or RPC endpoint is available, i.e., the variable might be None. In such cases this method can accept a None as arguments, and do resolution as described above.

Parameters:

username: str: Username to login, default to None.
password: str: Password to login, default to None.

Raises:

ConnectionFailed

class vineyard.IPCClient#

IPC client that connects to vineyard instance’s UNIX domain socket.

allocated_size()#

allocated_size(target: Object or ObjectID) → int

Get the allocated size of the given object.

Parameters:: target – Object or ObjectID The given Object.
Returns:: int

clear()#

clear() → None

Drop all objects that visible to the current instance in the vineyard cluster.

close()#: Close the client.

property connected#: Whether the client instance has been connected to the vineyard server.

create_blob()#

create_blob(size: int) → BlobBuilder

Allocate a blob in vineyard server.

Parameters:

size: int: The size of blob that will be allocated on vineyardd.

Returns:

BlobBuilder

create_blob(sizes: List[int]) → List[BlobBuilder]

Allocate blobs in vineyard server.

Parameters:

size: List[int]: The size of blobs that will be allocated on vineyardd.

Returns:

List[BlobBuilder]

create_blob(size: int) → BlobBuilder

Allocate a blob in vineyard server.

Parameters:

size: int: The size of blob that will be allocated on vineyardd.

Returns:

BlobBuilder

create_blob(sizes: List[int]) → List[BlobBuilder]

Allocate blobs in vineyard server.

Parameters:

size: List[int]: The size of blobs that will be allocated on vineyardd.

Returns:

List[BlobBuilder]

create_empty_blob()#

create_empty_blob() → Blob

Create an empty blob in vineyard server.

Returns:: Blob

create_metadata()#

create_metadata(metadata: ObjectMeta | List[ObjectMeta])
-> Union[ObjectMeta, List[ObjectMeta]]

Create metadata in vineyardd.

Parameters:: metadata – ObjectMeta The metadata that will be created on vineyardd.
Returns:: The result created metadata.
Return type:: Union[ObjectMeta, List[ObjectMeta]]

create_metadata(metadata: Union[ObjectMeta, List[ObjectMeta]],
instance_id: InstanceID)
-> Union[ObjectMeta, List[ObjectMeta]]

Create metadata in vineyardd with a specified instance id.

Parameters:

metadata – ObjectMeta The metadata that will be created on vineyardd.
instance_id – InstanceID The instance to place the object metadata.

Returns:

The result created metadata.

Return type:

Union[ObjectMeta, List[ObjectMeta]]

create_metadata(metadata: ObjectMeta | List[ObjectMeta])
-> Union[ObjectMeta, List[ObjectMeta]]

Create metadata in vineyardd.

Parameters:: metadata – ObjectMeta The metadata that will be created on vineyardd.
Returns:: The result created metadata.
Return type:: Union[ObjectMeta, List[ObjectMeta]]

create_metadata(metadata: Union[ObjectMeta, List[ObjectMeta]],
instance_id: InstanceID)
-> Union[ObjectMeta, List[ObjectMeta]]

Create metadata in vineyardd with a specified instance id.

Parameters:

metadata – ObjectMeta The metadata that will be created on vineyardd.
instance_id – InstanceID The instance to place the object metadata.

Returns:

The result created metadata.

Return type:

Union[ObjectMeta, List[ObjectMeta]]

delete()#

delete(object_id: ObjectID or List[ObjectID], force: bool = false,
deep: bool = true, memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:

object_id – ObjectID or list of ObjectID Objects that will be deleted. The object_id can be a single ObjectID, or a list of ObjectID.
force – bool Forcedly delete an object means the member will be recursively deleted even if the member object is also referred by others. The default value is True.
deep –
bool Deeply delete an object means we will deleting the members recursively. The default value is True.

Note that when deleting objects which have direct blob members, the processing on those blobs yields a “deep” behavior.
memory_trim –
bool Whether to trim the memory pool inside the shared memory allocator to return the unused physical memory back to the OS kernel, like the malloc_trim API from glibc. Default value is False.

Note that the memory trimming operation is a best-effort operation and may not release any memory at all.

delete(object_meta: ObjectMeta, force: bool = false, deep: bool = true,
memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:: object_meta – The corresponding object meta to delete.

delete(object: Object, force: bool = false, deep: bool = true,
memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:: object – The corresponding object meta to delete.

delete(object_id: ObjectID or List[ObjectID], force: bool = false,
deep: bool = true, memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:

object_id – ObjectID or list of ObjectID Objects that will be deleted. The object_id can be a single ObjectID, or a list of ObjectID.
force – bool Forcedly delete an object means the member will be recursively deleted even if the member object is also referred by others. The default value is True.
deep –
bool Deeply delete an object means we will deleting the members recursively. The default value is True.

Note that when deleting objects which have direct blob members, the processing on those blobs yields a “deep” behavior.
memory_trim –
bool Whether to trim the memory pool inside the shared memory allocator to return the unused physical memory back to the OS kernel, like the malloc_trim API from glibc. Default value is False.

Note that the memory trimming operation is a best-effort operation and may not release any memory at all.

delete(object_meta: ObjectMeta, force: bool = false, deep: bool = true,
memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:: object_meta – The corresponding object meta to delete.

delete(object: Object, force: bool = false, deep: bool = true,
memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:: object – The corresponding object meta to delete.

delete(object_id: ObjectID or List[ObjectID], force: bool = false,
deep: bool = true, memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:

object_id – ObjectID or list of ObjectID Objects that will be deleted. The object_id can be a single ObjectID, or a list of ObjectID.
force – bool Forcedly delete an object means the member will be recursively deleted even if the member object is also referred by others. The default value is True.
deep –
bool Deeply delete an object means we will deleting the members recursively. The default value is True.

Note that when deleting objects which have direct blob members, the processing on those blobs yields a “deep” behavior.
memory_trim –
bool Whether to trim the memory pool inside the shared memory allocator to return the unused physical memory back to the OS kernel, like the malloc_trim API from glibc. Default value is False.

Note that the memory trimming operation is a best-effort operation and may not release any memory at all.

delete(object_meta: ObjectMeta, force: bool = false, deep: bool = true,
memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:: object_meta – The corresponding object meta to delete.

delete(object: Object, force: bool = false, deep: bool = true,
memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:: object – The corresponding object meta to delete.

delete(object_id: ObjectID or List[ObjectID], force: bool = false,
deep: bool = true, memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:

object_id – ObjectID or list of ObjectID Objects that will be deleted. The object_id can be a single ObjectID, or a list of ObjectID.
force – bool Forcedly delete an object means the member will be recursively deleted even if the member object is also referred by others. The default value is True.
deep –
bool Deeply delete an object means we will deleting the members recursively. The default value is True.

Note that when deleting objects which have direct blob members, the processing on those blobs yields a “deep” behavior.
memory_trim –
bool Whether to trim the memory pool inside the shared memory allocator to return the unused physical memory back to the OS kernel, like the malloc_trim API from glibc. Default value is False.

Note that the memory trimming operation is a best-effort operation and may not release any memory at all.

delete(object_meta: ObjectMeta, force: bool = false, deep: bool = true,
memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:: object_meta – The corresponding object meta to delete.

delete(object: Object, force: bool = false, deep: bool = true,
memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:: object – The corresponding object meta to delete.

drop_name()#

drop_name(name: str or ObjectName) → None

Remove the association of the given name.

Parameters:: name – str The name that will be removed.

exists()#

exists(object_id: ObjectID) → bool

Whether the given object exists.

Parameters:: object_id – ObjectID The object id to check if exists.
Returns:: True when the specified object exists.
Return type:: bool

find_shared_memory()#

find_shared_memory(target: ptr) → bool

Find the corresponding blob (if exists) of the given pointer.

Parameters:: target – address, in int format The given address.
Returns:: ObjectID

find_shared_memory(target: ptr) → bool

Find the corresponding blob (if exists) of the given pointer.

Parameters:: target – address, in int format The given address.
Returns:: ObjectID

get(object_id: ObjectID | None = None, name: str | None = None, resolver: ResolverContext | None = None, fetch: bool = False, **kwargs)#

Get vineyard object as python value.

>>> arr_id = vineyard.ObjectID('00002ec13bc81226')
>>> arr = client.get(arr_id)
>>> arr
array([0, 1, 2, 3, 4, 5, 6, 7])

Parameters:

client – IPCClient or RPCClient The vineyard client to use.
object_id – ObjectID The object id that will be obtained from vineyard.
name – ObjectID The object name that will be obtained from vineyard, ignored if object_id is not None.
resolver – When retrieving vineyard object, an optional resolver can be specified. If no resolver given, the default resolver context will be used.
fetch – Whether to trigger a migration when the target object is located on remote instances.
kw – User-specific argument that will be passed to the builder.

Returns:

A python object that return by the resolver, by resolving an vineyard object.

get_blob()#

get_blob(object_id: ObjectID, unsafe: bool = false) → Blob

Getting a blob from vineyard using the given object ID.

Parameters:

object_id – ObjectID The blob to get.
unsafe – bool unsafe means getting the blob even the blob is not sealed yet. Default is False.

Returns:

Blob

See also

IPCClient.get_blob RPCClient.get_remote_blob RPCClient.get_remote_blobs

get_blobs()#

get_blobs(object_ids: List[ObjectID], unsafe: bool = false) → List[Blob]

Getting blobs from vineyard using the given object IDs.

Parameters:

object_ids – List[ObjectID] The blobs to get.
unsafe – bool unsafe means getting the blob even the blob is not sealed yet. Default is False.

Returns:

List[Blob]

See also

IPCClient.get_blobs RPCClient.get_remote_blob RPCClient.get_remote_blobs

get_meta()#

get_meta(object_id: ObjectID, sync_remote: bool = False) → ObjectMeta

Get object metadata from vineyard.

Parameters:

object_id – ObjectID The object id to get.
sync_remote – bool If the target object is a remote object, code_remote=True will force a meta synchronization on the vineyard server. Default is False.

Returns:

ObjectMeta

get_metas()#

get_metas(object_ids: List[ObjectID], sync_remote: bool = False)
-> List[ObjectMeta]

Get metadatas of multiple objects from vineyard.

Parameters:

object_ids – List[ObjectID] The object ids to get.
sync_remote – bool If the target object is a remote object, code_remote=True will force a meta synchronization on the vineyard server. Default is False.

Returns:

List[ObjectMeta]

get_name()#

get_name(name: str or ObjectName, wait: bool = False) → ObjectID

Get the associated object id of the given name.

Parameters:

name – str The name that will be queried.
wait – bool Whether to wait util the name appears, if wait, the request will be blocked until the name been registered.

Returns:

The associated object id with the name.

Return type:

ObjectID

get_object()#

get_object(object_id: ObjectID) → Object

Get object from vineyard.

Parameters:: object_id – ObjectID The object id to get.
Returns:: Object

get_objects()#

get_objects(object_ids: List[ObjectID]) → List[Object]

Get multiple objects from vineyard.

Parameters:: object_ids – List[ObjectID]
Returns:: List[Object]

property instance_id#: The instance id of the connected vineyard server.

property ipc_socket#: The UNIX domain socket location of connected vineyard server.

property is_ipc#: Whether the client is connected to vineyard server via UNIX domain socket.

property is_rpc#: Whether the client is connected to vineyard server via RPC endpoint.

is_shared_memory()#

is_shared_memory(target: ptr) → bool

Check if the address is on the shared memory region.

Parameters:: target – address, in int format The given address.
Returns:: bool

is_shared_memory(target: ptr) → bool

Check if the address is on the shared memory region.

Parameters:: target – address, in int format The given address.
Returns:: bool

list_metadatas()#

list_metadatas(pattern: str, regex: bool = False, limit: int = 5,
nobuffer: bool = False) -> List[Object]

List all objects in current vineyard server.

Parameters:

pattern – str The pattern string that will be matched against the object’s typename.
regex – bool Whether the pattern is a regex expression, otherwise the pattern will be used as wildcard pattern. Default value is False.
limit – int The limit to list. Default value is 5.
nobuffer – bool Whether to fill the buffers in returned object metadatas. Default value is False.

Returns:

List[ObjectMeta]

list_names()#

list_names(pattern: str, regex: bool = False, limit: int = 5)
-> Dict[str, ObjectID]

List all names in current vineyard server.

Parameters:

pattern – str The pattern string that will be matched against the name.
regex – bool Whether the pattern is a regex expression, otherwise the pattern will be used as wildcard pattern. Default value is False.
limit – int The limit to list. Default value is 5.

Returns:

Dict[str, ObjectID]

list_objects()#

list_objects(pattern: str, regex: bool = False, limit: int = 5)
-> List[Object]

List all objects in current vineyard server.

Parameters:

pattern – str The pattern string that will be matched against the object’s typename.
regex – bool Whether the pattern is a regex expression, otherwise the pattern will be used as wildcard pattern. Default value is False.
limit – int The limit to list. Default value is 5.

Returns:

List[Object]

memory_trim()#

memory_trim() → bool

Trim the memory pool inside the shared memory allocator to return the unused physical memory back to the OS kernel, like the malloc_trim API from glibc.

Returns True if it actually released any memory.

property meta#

The metadata information of the vineyard server. The value is a nested dict, the first-level key is the instance id, and the second-level key is the cluster metadata fields.

>>> client.meta
{
    14: {
        'hostid': '54058007061210',
        'hostname': '127.0.0.1',
        'timestamp': '6882550126788354072'
    },
    15: {
        'hostid': '48843417291806',
        'hostname': '127.0.0.1',
        'timestamp': '6882568290204737414'
    }
}

persist()#

persist(object_id: ObjectID) → None

Persist the object of the given object id. After persisting, the object will be visible by clients that connect to other vineyard server instances.

Parameters:: object_id – ObjectID The object that will be persist.

persist(object_meta: ObjectMeta) → None

Persist the given object.

Parameters:: object_meta – ObjectMeta The object that will be persist.

persist(object: Object) → None

Persist the given object.

Parameters:: object – Object The object that will be persist.

put(value: Any, builder: BuilderContext | None = None, persist: bool = False, name: str | None = None, **kwargs)#

Put python value to vineyard.

>>> arr = np.arange(8)
>>> arr_id = client.put(arr)
>>> arr_id
00002ec13bc81226

Parameters:

client – IPCClient or RPCClient The vineyard client to use.
value – The python value that will be put to vineyard. Supported python value types are decided by modules that registered to vineyard. By default, python value can be put to vineyard after serialized as a bytes buffer using pickle.
builder – optional When putting python value to vineyard, an optional builder can be specified to tell vineyard how to construct the corresponding vineyard Object. If not specified, the default builder context will be used to select a proper builder.
persist – bool, optional If true, persist the object after creation.
name – str, optional If given, the name will be automatically associated with the resulted object. Note that only take effect when the object is persisted.
kw – User-specific argument that will be passed to the builder.

Returns:

The result object id will be returned.

Return type:

ObjectID

put_name()#

put_name(object: ObjectID or ObjectMeta or Object,
name: str or ObjectName) -> None

Associate the given object id with a name. An ObjectID can be associated with more than one names.

Parameters:

object_id – ObjectID
name – str

reset()#

reset() → None

Alias of ClientBase.clear().

property rpc_endpoint#: The RPC endpoint of the connected vineyard server.

shallow_copy()#

shallow_copy(object_id: ObjectID) → ObjectID

Create a shallow copy of the given vineyard object.

Parameters:: object_id – ObjectID The vineyard object that is requested to be shallow-copied.
Returns:: The object id of newly shallow-copied vineyard object.
Return type:: ObjectID

shallow_copy(object_id: ObjectID, extra_metadata: dict) → ObjectID

Create a shallow copy of the given vineyard object, with extra metadata.

Parameters:

object_id – ObjectID The vineyard object that is requested to be shallow-copied.
extra_metadata – dict Extra metadata to apply to the newly created object. The fields of extra metadata must be primitive types, e.g., string, number, and cannot be array or dict.

Returns:

The object id of newly shallow-copied vineyard object.

Return type:

ObjectID

property status#: The status the of connected vineyard server, returns a InstanceStatus.

See also

InstanceStatus

sync_meta()#

sync_meta() → None

Synchronize remote metadata to local immediately.

property version#: The version number string of connected vineyard server, in the format of semver: MAJOR.MINOR.PATCH.

class vineyard.RPCClient#

RPC client that connects to vineyard instance’s RPC endpoints.

clear()#

clear() → None

Drop all objects that visible to the current instance in the vineyard cluster.

close()#: Close the client.

property connected#: Whether the client instance has been connected to the vineyard server.

create_metadata()#

create_metadata(metadata: ObjectMeta | List[ObjectMeta])
-> Union[ObjectMeta, List[ObjectMeta]]

Create metadata in vineyardd.

Parameters:: metadata – ObjectMeta The metadata that will be created on vineyardd.
Returns:: The result created metadata.
Return type:: Union[ObjectMeta, List[ObjectMeta]]

create_metadata(metadata: Union[ObjectMeta, List[ObjectMeta]],
instance_id: InstanceID)
-> Union[ObjectMeta, List[ObjectMeta]]

Create metadata in vineyardd with a specified instance id.

Parameters:

metadata – ObjectMeta The metadata that will be created on vineyardd.
instance_id – InstanceID The instance to place the object metadata.

Returns:

The result created metadata.

Return type:

Union[ObjectMeta, List[ObjectMeta]]

create_metadata(metadata: ObjectMeta | List[ObjectMeta])
-> Union[ObjectMeta, List[ObjectMeta]]

Create metadata in vineyardd.

Parameters:: metadata – ObjectMeta The metadata that will be created on vineyardd.
Returns:: The result created metadata.
Return type:: Union[ObjectMeta, List[ObjectMeta]]

create_metadata(metadata: Union[ObjectMeta, List[ObjectMeta]],
instance_id: InstanceID)
-> Union[ObjectMeta, List[ObjectMeta]]

Create metadata in vineyardd with a specified instance id.

Parameters:

metadata – ObjectMeta The metadata that will be created on vineyardd.
instance_id – InstanceID The instance to place the object metadata.

Returns:

The result created metadata.

Return type:

Union[ObjectMeta, List[ObjectMeta]]

create_remote_blob()#

create_remote_blob(blob_builder: RemoteBlobBuilder) → ObjectMeta

Put the remote blob to connected remote vineyard instance. The blob_builder is assumed to be ready and modification on the blob_builder after creation won’t take effect.

Note that creating remote blobs requires network transfer and may yields significate overhead.

vineyard_rpc_client = vineyard.connect(*vineyard_endpoint.split(':'))

buffer_writer = RemoteBlobBuilder(len(payload))
buffer_writer.copy(0, payload)
blob_meta = vineyard_rpc_client.create_remote_blob(buffer_writer)

Parameters:

blob_builder: RemoteBlobBuilder: The remote blob to create.

Returns:

ObjectMeta

See Also:

RPCClient.get_remote_blob RPCClient.get_remote_blobs

create_remote_blob(blob_builders: List[RemoteBlobBuilder]) → List[ObjectMeta]

Put the remote blobs to connected remote vineyard instance. The blob_builders is assumed to be ready and modification on the blob_builder after creation won’t take effect.

Note that creating remote blobs requires network transfer and may yields significate overhead.

Returns:: List[ObjectMeta]
See Also:: RPCClient.get_remote_blob RPCClient.get_remote_blobs

create_remote_blob(blob_builder: RemoteBlobBuilder) → ObjectMeta

Put the remote blob to connected remote vineyard instance. The blob_builder is assumed to be ready and modification on the blob_builder after creation won’t take effect.

Note that creating remote blobs requires network transfer and may yields significate overhead.

vineyard_rpc_client = vineyard.connect(*vineyard_endpoint.split(':'))

buffer_writer = RemoteBlobBuilder(len(payload))
buffer_writer.copy(0, payload)
blob_meta = vineyard_rpc_client.create_remote_blob(buffer_writer)

Parameters:

blob_builder: RemoteBlobBuilder: The remote blob to create.

Returns:

ObjectMeta

See Also:

RPCClient.get_remote_blob RPCClient.get_remote_blobs

create_remote_blob(blob_builders: List[RemoteBlobBuilder]) → List[ObjectMeta]

Put the remote blobs to connected remote vineyard instance. The blob_builders is assumed to be ready and modification on the blob_builder after creation won’t take effect.

Note that creating remote blobs requires network transfer and may yields significate overhead.

Returns:: List[ObjectMeta]
See Also:: RPCClient.get_remote_blob RPCClient.get_remote_blobs

delete()#

delete(object_id: ObjectID or List[ObjectID], force: bool = false,
deep: bool = true, memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:

object_id – ObjectID or list of ObjectID Objects that will be deleted. The object_id can be a single ObjectID, or a list of ObjectID.
force – bool Forcedly delete an object means the member will be recursively deleted even if the member object is also referred by others. The default value is True.
deep –
bool Deeply delete an object means we will deleting the members recursively. The default value is True.

Note that when deleting objects which have direct blob members, the processing on those blobs yields a “deep” behavior.
memory_trim –
bool Whether to trim the memory pool inside the shared memory allocator to return the unused physical memory back to the OS kernel, like the malloc_trim API from glibc. Default value is False.

Note that the memory trimming operation is a best-effort operation and may not release any memory at all.

delete(object_meta: ObjectMeta, force: bool = false, deep: bool = true,
memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:: object_meta – The corresponding object meta to delete.

delete(object: Object, force: bool = false, deep: bool = true,
memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:: object – The corresponding object meta to delete.

delete(object_id: ObjectID or List[ObjectID], force: bool = false,
deep: bool = true, memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:

object_id – ObjectID or list of ObjectID Objects that will be deleted. The object_id can be a single ObjectID, or a list of ObjectID.
force – bool Forcedly delete an object means the member will be recursively deleted even if the member object is also referred by others. The default value is True.
deep –
bool Deeply delete an object means we will deleting the members recursively. The default value is True.

Note that when deleting objects which have direct blob members, the processing on those blobs yields a “deep” behavior.
memory_trim –
bool Whether to trim the memory pool inside the shared memory allocator to return the unused physical memory back to the OS kernel, like the malloc_trim API from glibc. Default value is False.

Note that the memory trimming operation is a best-effort operation and may not release any memory at all.

delete(object_meta: ObjectMeta, force: bool = false, deep: bool = true,
memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:: object_meta – The corresponding object meta to delete.

delete(object: Object, force: bool = false, deep: bool = true,
memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:: object – The corresponding object meta to delete.

delete(object_id: ObjectID or List[ObjectID], force: bool = false,
deep: bool = true, memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:

object_id – ObjectID or list of ObjectID Objects that will be deleted. The object_id can be a single ObjectID, or a list of ObjectID.
force – bool Forcedly delete an object means the member will be recursively deleted even if the member object is also referred by others. The default value is True.
deep –
bool Deeply delete an object means we will deleting the members recursively. The default value is True.

Note that when deleting objects which have direct blob members, the processing on those blobs yields a “deep” behavior.
memory_trim –
bool Whether to trim the memory pool inside the shared memory allocator to return the unused physical memory back to the OS kernel, like the malloc_trim API from glibc. Default value is False.

Note that the memory trimming operation is a best-effort operation and may not release any memory at all.

delete(object_meta: ObjectMeta, force: bool = false, deep: bool = true,
memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:: object_meta – The corresponding object meta to delete.

delete(object: Object, force: bool = false, deep: bool = true,
memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:: object – The corresponding object meta to delete.

delete(object_id: ObjectID or List[ObjectID], force: bool = false,
deep: bool = true, memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:

object_id – ObjectID or list of ObjectID Objects that will be deleted. The object_id can be a single ObjectID, or a list of ObjectID.
force – bool Forcedly delete an object means the member will be recursively deleted even if the member object is also referred by others. The default value is True.
deep –
bool Deeply delete an object means we will deleting the members recursively. The default value is True.

Note that when deleting objects which have direct blob members, the processing on those blobs yields a “deep” behavior.
memory_trim –
bool Whether to trim the memory pool inside the shared memory allocator to return the unused physical memory back to the OS kernel, like the malloc_trim API from glibc. Default value is False.

Note that the memory trimming operation is a best-effort operation and may not release any memory at all.

delete(object_meta: ObjectMeta, force: bool = false, deep: bool = true,
memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:: object_meta – The corresponding object meta to delete.

delete(object: Object, force: bool = false, deep: bool = true,
memory_trim: bool = False) -> None

Delete the specific vineyard object.

Parameters:: object – The corresponding object meta to delete.

drop_name()#

drop_name(name: str or ObjectName) → None

Remove the association of the given name.

Parameters:: name – str The name that will be removed.

exists()#

exists(object_id: ObjectID) → bool

Whether the given object exists.

Parameters:: object_id – ObjectID The object id to check if exists.
Returns:: True when the specified object exists.
Return type:: bool

get(object_id: ObjectID | None = None, name: str | None = None, resolver: ResolverContext | None = None, fetch: bool = False, **kwargs)#

Get vineyard object as python value.

>>> arr_id = vineyard.ObjectID('00002ec13bc81226')
>>> arr = client.get(arr_id)
>>> arr
array([0, 1, 2, 3, 4, 5, 6, 7])

Parameters:

client – IPCClient or RPCClient The vineyard client to use.
object_id – ObjectID The object id that will be obtained from vineyard.
name – ObjectID The object name that will be obtained from vineyard, ignored if object_id is not None.
resolver – When retrieving vineyard object, an optional resolver can be specified. If no resolver given, the default resolver context will be used.
fetch – Whether to trigger a migration when the target object is located on remote instances.
kw – User-specific argument that will be passed to the builder.

Returns:

A python object that return by the resolver, by resolving an vineyard object.

get_meta()#

get_meta(object_id: ObjectID) → ObjectMeta

Get object metadata from vineyard.

Parameters:: object_id – ObjectID The object id to get.
Returns:: ObjectMeta

get_metas()#

get_metas(object_ids: List[ObjectID] -> List[ObjectMeta]

Get metadatas of multiple objects from vineyard.

Parameters:: object_ids – List[ObjectID]
Returns:: List[ObjectMeta]

get_name()#

get_name(name: str or ObjectName, wait: bool = False) → ObjectID

Get the associated object id of the given name.

Parameters:

name – str The name that will be queried.
wait – bool Whether to wait util the name appears, if wait, the request will be blocked until the name been registered.

Returns:

The associated object id with the name.

Return type:

ObjectID

get_object()#

get_object(object_id: ObjectID) → Object

Get object from vineyard.

Parameters:: object_id – ObjectID The object id to get.
Returns:: Object

get_objects()#

get_objects(object_ids: List[ObjectID]) → List[Object]

Get multiple objects from vineyard.

Parameters:: object_ids – List[ObjectID]
Returns:: List[Object]

get_remote_blob()#

get_remote_blob(object_id: ObjectID, unsafe: bool = false) → RemoteBlob

Getting a remote blob from vineyard using the given object ID.

Note that getting remote blobs requires network transfer and may yields significate overhead.

Parameters:

object_id – ObjectID The remote blob to get.
unsafe – bool unsafe means getting the blob even the blob is not sealed yet. Default is False.

Returns:

RemoteBlob

See also

IPCClient.get_blob IPCClient.get_blobs RPCClient.get_remote_blobs

get_remote_blobs()#

get_remote_blobs(object_ids: List[ObjectID], unsafe: bool = false)
-> List[RemoteBlob]

Getting remote blobs from vineyard using the given object IDs.

Note that getting remote blobs requires network transfer and may yields significate overhead.

Parameters:

object_ids – List[ObjectID] The remote blobs to get.
unsafe – bool unsafe means getting the blob even the blob is not sealed yet. Default is False.

Returns:

List[RemoteBlob]

See also

IPCClient.get_blob IPCClient.get_blobs RPCClient.get_remote_blob

property instance_id#: The instance id of the connected vineyard server.

property ipc_socket#: The UNIX domain socket location of connected vineyard server.

is_fetchable()#: Whether the rpc client is able to fetch objects from the connected vineyard server. When the instance connected by the rpc client is not the same as the instance of metadata, the rpc client is not able to fetch the object.

property is_ipc#: Whether the client is connected to vineyard server via UNIX domain socket.

property is_rpc#: Whether the client is connected to vineyard server via RPC endpoint.

list_metadatas()#

list_metadatas(pattern: str, regex: bool = False, limit: int = 5,
nobuffer: bool = False) -> List[Object]

List all objects in current vineyard server.

Parameters:

pattern – str The pattern string that will be matched against the object’s typename.
regex – bool Whether the pattern is a regex expression, otherwise the pattern will be used as wildcard pattern. Default value is False.
limit – int The limit to list. Default value is 5.
nobuffer – bool Whether to fill the buffers in returned object metadatas. Default value is False.

Returns:

List[ObjectMeta]

list_names()#

list_names(pattern: str, regex: bool = False, limit: int = 5)
-> Dict[str, ObjectID]

List all names in current vineyard server.

Parameters:

pattern – str The pattern string that will be matched against the name.
regex – bool Whether the pattern is a regex expression, otherwise the pattern will be used as wildcard pattern. Default value is False.
limit – int The limit to list. Default value is 5.

Returns:

Dict[str, ObjectID]

list_objects()#

list_objects(pattern: str, regex: bool = False, limit: int = 5)
-> List[Object]

List all objects in current vineyard server.

Parameters:

pattern – str The pattern string that will be matched against the object’s typename.
regex – bool Whether the pattern is a regex expression, otherwise the pattern will be used as wildcard pattern. Default value is False.
limit – int The limit to list. Default value is 5.

Returns:

List[Object]

memory_trim()#

memory_trim() → bool

Trim the memory pool inside the shared memory allocator to return the unused physical memory back to the OS kernel, like the malloc_trim API from glibc.

Returns True if it actually released any memory.

property meta#

The metadata information of the vineyard server. The value is a nested dict, the first-level key is the instance id, and the second-level key is the cluster metadata fields.

>>> client.meta
{
    14: {
        'hostid': '54058007061210',
        'hostname': '127.0.0.1',
        'timestamp': '6882550126788354072'
    },
    15: {
        'hostid': '48843417291806',
        'hostname': '127.0.0.1',
        'timestamp': '6882568290204737414'
    }
}

persist()#

persist(object_id: ObjectID) → None

Persist the object of the given object id. After persisting, the object will be visible by clients that connect to other vineyard server instances.

Parameters:: object_id – ObjectID The object that will be persist.

persist(object_meta: ObjectMeta) → None

Persist the given object.

Parameters:: object_meta – ObjectMeta The object that will be persist.

persist(object: Object) → None

Persist the given object.

Parameters:: object – Object The object that will be persist.

put(value: Any, builder: BuilderContext | None = None, persist: bool = False, name: str | None = None, **kwargs)#

Put python value to vineyard.

>>> arr = np.arange(8)
>>> arr_id = client.put(arr)
>>> arr_id
00002ec13bc81226

Parameters:

client – IPCClient or RPCClient The vineyard client to use.
value – The python value that will be put to vineyard. Supported python value types are decided by modules that registered to vineyard. By default, python value can be put to vineyard after serialized as a bytes buffer using pickle.
builder – optional When putting python value to vineyard, an optional builder can be specified to tell vineyard how to construct the corresponding vineyard Object. If not specified, the default builder context will be used to select a proper builder.
persist – bool, optional If true, persist the object after creation.
name – str, optional If given, the name will be automatically associated with the resulted object. Note that only take effect when the object is persisted.
kw – User-specific argument that will be passed to the builder.

Returns:

The result object id will be returned.

Return type:

ObjectID

put_name()#

put_name(object: ObjectID or ObjectMeta or Object,
name: str or ObjectName) -> None

Associate the given object id with a name. An ObjectID can be associated with more than one names.

Parameters:

object_id – ObjectID
name – str

property remote_instance_id#: The instance id of the connected remote vineyard server.

reset()#

reset() → None

Alias of ClientBase.clear().

property rpc_endpoint#: The RPC endpoint of the connected vineyard server.

shallow_copy()#

shallow_copy(object_id: ObjectID) → ObjectID

Create a shallow copy of the given vineyard object.

Parameters:: object_id – ObjectID The vineyard object that is requested to be shallow-copied.
Returns:: The object id of newly shallow-copied vineyard object.
Return type:: ObjectID

shallow_copy(object_id: ObjectID, extra_metadata: dict) → ObjectID

Create a shallow copy of the given vineyard object, with extra metadata.

Parameters:

object_id – ObjectID The vineyard object that is requested to be shallow-copied.
extra_metadata – dict Extra metadata to apply to the newly created object. The fields of extra metadata must be primitive types, e.g., string, number, and cannot be array or dict.

Returns:

The object id of newly shallow-copied vineyard object.

Return type:

ObjectID

property status#: The status the of connected vineyard server, returns a InstanceStatus.

See also

InstanceStatus

sync_meta()#

sync_meta() → None

Synchronize remote metadata to local immediately.

property version#: The version number string of connected vineyard server, in the format of semver: MAJOR.MINOR.PATCH.

Vineyard cluster#

class vineyard.InstanceStatus#

InstanceStatus represents the status of connected vineyard instance, including the instance identity, memory statistics and workloads on this instance.

>>> status = client.status
>>> print(status)
InstanceStatus:
    instance_id: 5
    deployment: local
    memory_usage: 360
    memory_limit: 268435456
    deferred_requests: 0
    ipc_connections: 1
    rpc_connections: 0
>>> status.instance_id
5
>>> status.deployment
'local'
>>> status.memory_usage
360
>>> status.memory_limit
268435456
>>> status.deferred_requests
0
>>> status.ipc_connections
1
>>> status.rpc_connections
0

__init__(*args, **kwargs)#

__repr__()#: Return repr(self).

__str__()#: Return str(self).

property deferred_requests#: Report number of waiting requests of current vineyardd instance.

property deployment#: The deployment mode of the connected vineyardd cluster, can be "local" and "distributed".

property instance_id#: Return the instance id of vineyardd that the client is connected to.

property ipc_connections#: Report number of alive IPC connections on the current vineyardd instance.

property memory_limit#: Report memory limit (in bytes) of current vineyardd instance.

property memory_usage#: Report memory usage (in bytes) of current vineyardd instance.

property rpc_connections#: Report number of alive RPC connections on the current vineyardd instance.

Blob#

class vineyard.Blob#

Blob in vineyard is a consecutive readonly shared memory.

property address#: The memory address value of this blob.

static empty()#: Create an empty blob (with size as 0).

property is_empty#: Whether the blob is an empty blob, i.e., the size of this blob is 0.

property size#: Size of the blob.

class vineyard.BlobBuilder#

BlobBuilder is the builder for creating a finally immutable blob in vineyard server.

A BlobBuilder can only be explicitly created using the IPCClient.create_blob().

See also

IPCClient.create_blob IPCClient.create_empty_blob

abort()#: Abort the blob builder if it is not sealed yet.

property address#: The memory address value of this blob builder.

copy()#

copy(self, offset: int, ptr: int, size: int, concurrency: int = 6)

Copy the given address to the given offset.

copy(self, offset: int, ptr: int, size: int, concurrency: int = 6)

Copy the given address to the given offset.

copy(self, offset: int, ptr: int, size: int, concurrency: int = 6)

Copy the given address to the given offset.

property id#: ObjectID of this blob builder.

shrink()#: Shrink the blob builder to the given size if it is not sealed yet.

property size#: Size of this blob builder.

class vineyard.RemoteBlob#

RemoteBlob is a holder for Blob in cases like the Blob doesn’t exist in the local vineyardd instance but the client still want th access the data with a tolerable network-transfer overhead.

property address#: The memory address value of this blob.

property id#: Object ID of the remote blob.

property instance_id#: The instance ID where the blob is actually placed at.

property is_empty#: Whether the blob is an empty blob, i.e., the size of this blob is 0.

property size#: The size of this blob.

class vineyard.RemoteBlobBuilder#

RemoteBlobBuilder is the builder for creating a finally immutable blob in vineyard server over a RPC client.

A RemoteBlobBuilder can only be explicitly initialized using RemoteBlobBuilder(), then be filled the content and finally be sent to remote vineyard instance over the network.

See also

IPCClient.create_blob IPCClient.create_empty_blob

abort()#: Abort the blob builder if it is not sealed yet.

property address#: The memory address value of this blob builder.

copy()#

copy(self, offset: int, ptr: int, size: int, concurrency: int = 6)

Copy the given address to the given offset.

copy(self, offset: int, ptr: int, size: int, concurrency: int = 6)

Copy the given address to the given offset.

copy(self, offset: int, ptr: int, size: int, concurrency: int = 6)

Copy the given address to the given offset.

property size#: Size of this blob builder.

Resolvers and builders#

class vineyard.core.resolver.ResolverContext(parent_context: ResolverContext | None = None)[source]#

vineyard.core.resolver.get_current_resolvers() → ResolverContext[source]#: Obtain current resolver context.

vineyard.core.resolver.resolver_context(resolvers: Dict[str, Callable] | None = None, base: ResolverContext | None = None) → Generator[ResolverContext, Any, Any][source]#

Open a new context for register resolvers, without populating outside the global environment.

The resolver_context can be useful when users have more than more resolver for a certain type, e.g., the vineyard::Tensor object can be resolved as numpy.ndarray or xgboost::DMatrix.

We could have

def numpy_resolver(obj):
    ...

default_resolver_context.register('vineyard::Tensor', numpy_resolver)

and

def xgboost_resolver(obj):
    ...

default_resolver_context.register('vineyard::Tensor', xgboost_resolver)

Obviously there’s a conflict, and the stackable resolver_context could help there,

with resolver_context({'vineyard::Tensor', xgboost_resolver}):
    ...

Assuming the default context resolves vineyard::Tensor to numpy.ndarray, inside the with resolver_context the vineyard::Tensor will be resolved to xgboost::DMatrix, and after exiting the context the global environment will be restored back as default.

The with resolver_context is nestable as well.

See also

builder_context driver_context

class vineyard.core.builder.BuilderContext(parent_context: BuilderContext | None = None)[source]#

register(type_id: type, builder: Callable)[source]#

Parameters:

type_id (Python type) – Like int, or numpy.ndarray
builder (callable, e.g., a method, callable object) – A builder translates a python object to vineyard, it accepts a Python value as parameter, and returns an vineyard object as result.

run(client, value, **kw)[source]#

Follows the MRO to find the proper builder for given python value.

Here “Follows the MRO” implies:

If the type of python value has been found in the context, the registered builder will be used.
If not, it follows the MRO chain from down to top to find a registered Python type and used the associated builder.
When the traversal reaches the object type, since there’s a default builder that serialization the python value, the parameter will be serialized and be put into a blob.

vineyard.core.builder.get_current_builders() → BuilderContext[source]#: Obtain the current builder context.

vineyard.core.builder.builder_context(builders: Dict[type, Callable] | None = None, base: BuilderContext | None = None) → Generator[BuilderContext, Any, Any][source]#: Open a new context for register builders, without populating outside global environment.

See also

resolver_context driver_context

class vineyard.core.driver.DriverContext[source]#

vineyard.core.driver.get_current_drivers()[source]#: Obtain current driver context.

vineyard.core.driver.driver_context(drivers=None, base=None)[source]#: Open a new context for register drivers, without populting outside global environment.

See also

builder_context resolver_context

Shared memory#

class vineyard.shared_memory.SharedMemory(vineyard_client, name=None, create=False, size=0)[source]#

property buf#: A memoryview of contents of the shared memory block.

freeze()[source]#: Seal the shared memory to make it visible for other processes.

property name#: Unique name that identifies the shared memory block.

property size#: Size in bytes.

unlink()[source]#: Requests that the underlying shared memory block be destroyed.

class vineyard.shared_memory.ShareableList(vineyard_client, sequence=None, *, name=None)[source]#

Pattern for a mutable list-like object shareable via a shared memory block. It differs from the built-in list type in that these lists can not change their overall length (i.e. no append, insert, etc.)

Because values are packed into a memoryview as bytes, the struct packing format for any storable value must require no more than 8 characters to describe its format.

The ShareableList in vineyard differs slightly with its equivalent in the multiprocessing.shared_memory.ShareableList, as it becomes immutable after obtaining from the vineyard backend.

See also

multiprocessing.shared_memory.ShareableList

freeze()[source]#: Make the shareable list immutable and visible for other vineyard clients.

Deployment#

vineyard.connect(*args, **kwargs)[source]#

Connect to vineyard by specified UNIX-domain socket or TCP endpoint.

The connect() method has various overloading:

connect(socket: str,

username: str = None,

password: str = None) -> IPCClient

Connect to vineyard via UNIX domain socket for IPC service:

client = vineyard.connect('/var/run/vineyard.sock')

Parameters:

socket: str: UNIX domain socket path to setup an IPC connection.
username: str: Username to login, default to None.
password: str: Password to login, default to None.

Returns:

IPCClient: The connected IPC client.

connect(host: str,

port: int or str,

username: str = None,

password: str = None) -> RPCClient

Connect to vineyard via TCP socket.

Parameters:

host: str: Hostname to connect to.
port: int or str: The TCP that listened by vineyard TCP service.
username: str: Username to login, default to None.
password: str: Password to login, default to None.

Returns:

RPCClient: The connected RPC client.

connect(endpoint: (str, int or str),

username: str = None,

password: str = None) -> RPCClient

Connect to vineyard via TCP socket.

Parameters:

endpoint: tuple(str, int or str): Endpoint to connect to. The parameter is a tuple, in which the first element is the host, and the second parameter is the port (can be int a str).
username: str: Username to login, default to None.
password: str: Password to login, default to None.

Returns:

RPCClient: The connected RPC client.

connect(username: str = None,

password: str = None) -> IPCClient or RPCClient

If above all are failed, this method will raise a ConnectionFailed exception.

Parameters:

username: str: Username to login, default to None.
password: str: Password to login, default to None.

Raises:

ConnectionFailed

vineyard.get_current_socket() → str[source]#

Get current vineyard UNIX-domain socket established by vineyard.try_init().

Raises:: ValueError if vineyard is not initialized. –

Launch a local vineyard cluster.

Parameters:

meta – str, optional. Metadata backend, can be “etcd”, “redis” and “local”. Defaults to “etcd”.
etcd_endpoint – str Launching vineyard using specified etcd endpoints. If not specified, vineyard will launch its own etcd instance.
etcd_prefix – str Specify a common prefix to establish a local vineyard cluster.
vineyardd_path – str Location of vineyard server program. If not specified, vineyard will use its own bundled vineyardd binary.
size –
int The memory size limit for vineyard’s shared memory. The memory size can be a plain integer or as a fixed-point number using one of these suffixes:
```
E, P, T, G, M, K.
```
You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi, Mi, Ki. Defaults to “”, means not limited.

For example, the following represent roughly the same value:
```
128974848, 129k, 129M, 123Mi, 1G, 10Gi, ...
```
socket –
str The UNIX domain socket socket path that vineyard server will listen on. Default is None.

When the socket parameter is None, a random path under temporary directory will be generated and used.
rpc_socket_port – int The port that vineyard will use to provided RPC service.
debug – bool Whether print debug logs.

Returns:

Yields a tuple with the subprocess as the first element and the UNIX-domain IPC socket as the second element.

Return type:

(proc, socket)

vineyard.deploy.distributed.start_vineyardd(hosts=None, etcd_endpoints=None, vineyardd_path=None, size='', socket='/var/run/vineyard.sock', rpc_socket_port=9600, debug=False)[source]#

Launch a local vineyard cluster in a distributed fashion.

Parameters:

hosts – list of str A list of machines to launch vineyard server.
etcd_endpoint – str Launching vineyard using specified etcd endpoints. If not specified, vineyard will launch its own etcd instance.
vineyardd_path – str Location of vineyard server program. If not specified, vineyard will use its own bundled vineyardd binary.
size –
int The memory size limit for vineyard’s shared memory. The memory size can be a plain integer or as a fixed-point number using one of these suffixes:
```
E, P, T, G, M, K.
```
You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi, Mi, Ki. Defaults to “”, means not limited.

For example, the following represent roughly the same value:
```
128974848, 129k, 129M, 123Mi, 1G, 10Gi, ...
```
socket – str The UNIX domain socket socket path that vineyard server will listen on.
rpc_socket_port – int The port that vineyard will use to privode RPC service.
debug – bool Whether print debug logs.

vineyard.deploy.kubernetes.start_vineyardd(namespace='vineyard', size='512Mi', socket='/var/run/vineyard.sock', rpc_socket_port=9600, vineyard_image='vineyardcloudnative/vineyardd:latest', vineyard_image_pull_policy='IfNotPresent', vineyard_image_pull_secrets=None, k8s_client=None)[source]#

Launch a vineyard cluster on kubernetes.

Parameters:

namespace – str namespace in kubernetes
size –
int The memory size limit for vineyard’s shared memory. The memory size can be a plain integer or as a fixed-point number using one of these suffixes:
```
E, P, T, G, M, K.
```
You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi, Mi, Ki.

For example, the following represent roughly the same value:
```
128974848, 129k, 129M, 123Mi, 1G, 10Gi, ...
```
socket – str The UNIX domain socket socket path that vineyard server will listen on.
rpc_socket_port – int The port that vineyard will use to provide RPC service.
k8s_client – kubernetes.client.api.ApiClient A kubernetes client. If not specified, vineyard will try to resolve the kubernetes configuration from current context.
vineyard_image – str The docker image of vineyardd to launch the daemonset.
vineyard_image_pull_policy – str The docker image pull policy of vineyardd.
vineyard_image_pull_secrets – str The docker image pull secrets of vineyardd.

Returns:

A list of created kubernetes resources during the deploying process. The resources can be later release using delete_kubernetes_objects().

See also

vineyard.deploy.kubernetes.delete_kubernetes_objects

vineyard.deploy.kubernetes.delete_kubernetes_objects(targets, k8s_client=None, verbose=False, wait=False, timeout_seconds=60, **kwargs)[source]#

Delete the given kubernetes resources.

Parameters:

target – List List of Kubernetes objects
k8s_client – The kubernetes client. If not specified, vineyard will try to resolve the kubernetes configuration from current context.
verbose – bool Whether to print the deletion logs.
wait – bool Whether to wait for the deletion to complete.
timeout_seconds – int The timeout in seconds for waiting for the deletion to complete.

See also

vineyard.deploy.kubernetes.start_vineyardd vineyard.deploy.kubernetes.delete_kubernetes_object

I/O Drivers#

vineyard.io.open(path, *args, mode='r', handlers=None, chunk_hook: Callable | None = None, **kwargs)[source]#

Open a path as a reader or writer, depends on the parameter mode. If mode is r, it will open a stream for read, and open a stream for write when mode is w.

Parameters:

path (str) – Path to open.
mode (char) – Mode about how to open the path, r is for read and w for write.
handlers – A dict that will be filled with a handler that contains the process handler of the underlying read/write process that can be joined using join to capture the possible errors during the I/O proceeding.
chunk_hook (callable, optional) –
If the read/write target is a global dataframe (e.g., csv, orc, parquet, etc.), the hook will be called for each chunk to be read or write (usually a pyarrow.RecordBatch). The hook should return a pyarrow.RecordBatch object and should be stateless as the invoke order is not guaranteed. E.g.,
```
def exchange_column(batch):
    import pyarrow as pa

    columns = batch.columns
    first = columns[0]
    second = columns[1]
    columns = [second, first] + columns[2:]
    return pa.RecordBatch.from_arrays(columns, schema=batch.schema)
```
vineyard_ipc_socket (str) – Vineyard’s IPC socket location.
vineyard_endpoint (str) – Vineyard’s RPC socket address.

See also

vineyard.io.read, vineyard.io.write

vineyard.io.read(path, *args, handlers=None, accumulate=False, chunk_hook: Callable | None = None, **kwargs)[source]#

Open a path and read it as a single stream.

Parameters:

path (str) – Path to read, the last reader registered for the scheme of the path will be used.
handlers (list, optional) – If handlers is not None, launched worker processes will be emplaced into the list for further customized job lifecycle management. Default is None.
accumulate (bool, optional) – If accumulate is True, it will return a data frame, rather than dataframe stream. Default is False.
chunk_hook (callable, optional) –
If the read/write target is a global dataframe (e.g., csv, orc, parquet, etc.), the hook will be called for each chunk to be read or write (usually a pyarrow.RecordBatch). The hook should return a pyarrow.RecordBatch object and should be stateless as the invoke order is not guaranteed. E.g.,
```
def exchange_column(batch):
    import pyarrow as pa

    columns = batch.columns
    first = columns[0]
    second = columns[1]
    columns = [second, first] + columns[2:]
    return pa.RecordBatch.from_arrays(columns, schema=batch.schema)
```
vineyard_ipc_socket (str) – The local or remote vineyard’s IPC socket location that the remote readers will use to establish connections with the vineyard server.
vineyard_endpoint (str, optional) – An optional address of vineyard’s RPC socket, which will be used for retrieving server’s information on the client side. If not provided, the vineyard_ipc_socket will be used, or it will tries to discovery vineyard’s IPC or RPC endpoints from environment variables.

vineyard.io.write(path, stream, *args, handlers=None, chunk_hook: Callable | None = None, **kwargs)[source]#

Write the stream to a given path.

Parameters:

path (str) – Path to write, the last writer registered for the scheme of the path will be used.
stream (vineyard stream) – Stream that produces the data to write.
handlers (list, optional) – If handlers is not None, launched worker processes will be emplaced into the list for further customized job lifecycle management. Default is None.
chunk_hook (callable, optional) –
If the read/write target is a global dataframe (e.g., csv, orc, parquet, etc.), the hook will be called for each chunk to be read or write (usually a pyarrow.RecordBatch). The hook should return a pyarrow.RecordBatch object and should be stateless as the invoke order is not guaranteed. E.g.,
```
def exchange_column(batch):
    import pyarrow as pa

    columns = batch.columns
    first = columns[0]
    second = columns[1]
    columns = [second, first] + columns[2:]
    return pa.RecordBatch.from_arrays(columns, schema=batch.schema)
```
vineyard_ipc_socket (str) – The local or remote vineyard’s IPC socket location that the remote readers will use to establish connections with the vineyard server.
vineyard_endpoint (str, optional) – An optional address of vineyard’s RPC socket, which will be used for retrieving server’s information on the client side. If not provided, the vineyard_ipc_socket will be used, or it will tries to discovery vineyard’s IPC or RPC endpoints from environment variables.

Streams#

class vineyard.io.byte.ByteStream(meta: ObjectMeta, params: Dict | None = None)[source]#

class vineyard.io.dataframe.DataframeStream(meta: ObjectMeta, params: Dict | None = None)[source]#

class vineyard.io.recordbatch.RecordBatchStream(meta: ObjectMeta, params: Dict[str, Any] | None = None)[source]#

Interacting with the CSI Driver#

vineyard.csi.read(path: str)[source]#

Read vineyard object from path, and return python value. Notice, the API is only used for CSI driver.

Parameters:: path – str The path that represents a vineyard object.

>>> arr = vineyard.read('/a/b/c/d/f')
>>> arr
array([0, 1, 2, 3, 4, 5, 6, 7])

Returns:: A python object that return by the resolver, by resolving an vineyard object.

vineyard.csi.write(value: Any, path: str)[source]#

Write python value to vineyard. Notice, the API is only used for CSI driver.

Parameters:: path – str The path that represents a vineyard object.

>>> arr = np.arange(8)
>>> vineyard.write(arr)

Vineyard Cli Tool#

You can also use the Python API to interact with internal Vineyard Cli Tool.

$ python -m vineyard.cli <command> [options]

Vineyard LLM KV Cache#

Before using the KV Cache, you need to install the vineyard-llm package.