Python API Reference
ObjectID and Object
- class vineyard.ObjectID
Opaque type for vineyard’s object id. The object ID is generated by vineyard server, the underlying type of
ObjectID
is a 64-bit unsigned integer. Wrapper utilities are provided to interact with the external python world.>>> id = ObjectID("000043c5c6d5e646") >>> id 000043c5c6d5e646 >>> repr(id) '000043c5c6d5e646' >>> print(id) ObjectID <"000043c5c6d5e646"> >>> int(id) 74516723525190
- class vineyard.Object
Base class for vineyard objects.
- property id
The object id of this object.
- property isglobal
Whether the object is a global object.
- property islocal
Whether the object is a local object.
- property ispersist
Whether the object is a persistent object. The word “persistent” means the object could be seen by clients that connect to other vineyard server instances.
- member()
- member(self, name: str) → Object
Get the member object of this object.
- Parameters
name – str The name of the member object.
- Returns
The member object.
- Return type
See also
ObjectMeta.get, ObjectMeta.__getitem__
- property meta
The metadata of this object.
- property nbytes
The nbytes of this object.
- property signature
The object signature of this object.
- property typename
The typename of this object.
typename
is the string value of the C++ type, e.g.,vineyard::Array<int>
,vineyard::Table
.
- class vineyard.ObjectBuilder
Base class for vineyard object builders.
Metadata of objects
- class vineyard.ObjectMeta
ObjectMeta
is the type for metadata of anObject
. TheObjectMeta
can be treat as a dict-like type. If the the metadata if the metadata obtained from vineyard, the metadata is readonly. Otherwise key-value attributes or object members could be associated with the metadata to construct a new vineyard object.We can inspect the key-value attributes and members of an
ObjectMeta
:>>> meta = client.get_meta(hashmap_id) >>> meta ObjectMeta { "id": "0000347aebe92dd0", "instance_id": "0", ... } >>> meta['num_elements_'] '5' >>> meta['entries'] ObjectMeta { "id": "0000347aebe92dd0", "instance_id": "0", ... }
ObjectMeta
value can be iterated over:>>> list(k for k in meta['entries']) ['transient', 'num_slots_minus_one_', 'max_lookups_', 'num_elements_', 'entries_', 'nbytes', 'typename', 'instance_id', 'id']
- __contains__()
Check if given key exists in the object metadata.
- Parameters
key – str The name to be looked up.
- Returns
True
if the queried key exists in this object metadata, otherwiseFalse
.- Return type
bool
- __getitem__()
- __getitem__(self, key: str) → string or Object
Get meta or member’s meta from metadata.
- Parameters
key – str The name to be looked up.
- Returns
string – If the given key is a key of meta, returns the meta value.
Object – If the given key is a key of member, return the meta of this member.
- __init__()
- __init__(global_: bool = False)
Create an empty metadata, the metadata will be used to create a vineyard object.
- Parameters
global_: bool, if the object meta is for creating a global object.
- __setitem__()
- __setitem__(self, key: str, value) → None
Add a metadata to the ObjectMeta.
- Parameters
key – str The name of the new metadata entry.
value –
str, int, float, bool or list of int The value of the new metadata entry.
When the value is a
str
, it will be convert to string at first.When the value is a list of str, int or float, it will be first dumpped as string using
json.dumps
.
- __setitem__(self, key: str, ObjectID, Object or ObjectMeta) → None
Add a member object.
- Parameters
key – str The name of the member object.
object –
Object
,ObjectID
orObjectMeta
The reference to the member object or the object id of the member object.
- add_member()
- add_member(self, key: str, ObjectID, Object or ObjectMeta) → None
Add a member object.
- Parameters
key – str The name of the member object.
object –
Object
,ObjectID
orObjectMeta
The reference to the member object or the object id of the member object.
- get()
- get(self, key: str, default=None) → string or Object
Get meta or member’s meta from metadata, return default value if the given key is not presented.
- Parameters
key – str The name to be looked up.
- Returns
str – When the given
key
belongs to a metadata pair. Note that the metadata value of type int or float will be returned in string format as well.ObjectMeta – When the given
key
is mapped to a member object.See Also – ObjectMeta.__getitem__
- get_member()
- get_member(self, key: str) → Object
Get member object from metadata, return None if the given key is not presented, and raise exception
RuntimeError
if the given key is associated with a plain metadata, rather than member object.- Parameters
key – str The name to be looked up.
- Raises
RuntimeError – When the given key is associated with a plain metadata, rather than member object.
See also
ObjectMeta.__getitem__, ObjectMeta.get
- property id
The corresponding object ID of this metadata.
- property instance_id
The
"instance_id"
of vineyard instance that the metadata been placed on.
- property isglobal
True if the object is a global object, otherwise a local object or remote object.
- property islocal
True if the object is a local object, otherwise a global object or remote object.
- property memory_usage
Get the total memory usage of buffers in this object meta.
- property nbytes
The
"nbytes"
attribute of this metadata.
- set_global()
Mark the building object as a global object.
- Parameters
global – bool, default is True
- property signature
The corresponding object signature of this metadata.
- property typename
The
"typename"
attribute of this metadata.
Connect to vineyard
- vineyard.connect()
- vineyard.connect(endpoint: str) → IPCClient
Connect to vineyard via UNIX domain socket for IPC service:
client = vineyard.connect('/var/run/vineyard.sock')
- Parameters:
- endpoint: str
UNIX domain socket path to setup an IPC connection.
- Returns:
IPCClient: The connected IPC client.
- vineyard.connect(host: str, port: int or str) → RPCClient
Connect to vineyard via TCP socket.
- Parameters:
- host: str
Hostname to connect to.
- port: int or str
The TCP that listened by vineyard TCP service.
- Returns:
RPCClient: The connected RPC client.
- vineyard.connect(endpoint: (str, int or str)) → RPCClient
Connect to vineyard via TCP socket.
- Parameters:
- endpoint: tuple(str, int or str)
Endpoint to connect to. The parameter is a tuple, in which the first element is the host, and the second parameter, can be int a str, is the port.
- Returns:
RPCClient: The connected RPC client.
- vineyard.connect() → IPCClient or RPCClient
Connect to vineyard via UNIX domain socket or TCP endpoint. This method normally usually no arguments, and will first tries to resolve IPC socket from the environment variable VINEYARD_IPC_SOCKET and connect to it. If it fails to establish a connection with vineyard server, the method will tries to resolve RPC endpoint from the environment variable VINEYARD_RPC_ENDPOINT.
If both tries are failed, this method will raise a
ConnectionFailed
exception.In rare cases, user may be not sure about if the IPC socket or RPC endpoint is available, i.e., the variable might be
None
. In such cases this method can accept a None as arguments, and do resolution as described above.- Raises:
ConnectionFailed
- class vineyard.IPCClient
IPC client that connects to vineyard instance’s UNIX domain socket.
- allocated_size()
- allocated_size(target: Object or ObjectID) → int
Get the allocated size of the given object.
- Parameters
target – Object or ObjectID The given Object.
- Returns
int
- clear()
- clear() → None
Drop all objects that visible to the current instance in the vineyard cluster.
- close()
Close the client.
- property connected
Whether the client instance has been connected to the vineyard server.
- create_blob()
- create_blob(size: int) → Blob
Allocate a blob in vineyard server.
- Parameters
size – int The size of blob that will be allocated on vineyardd.
- Returns
BlobBuilder
- create_empty_blob()
- create_empty_blob() → Blob
Create an empty blob in vineyard server.
- Returns
Blob
- create_metadata()
- create_metadata(metadata: ObjectMeta) → ObjectMeta
Create metadata in vineyardd.
- Parameters
metadata – ObjectMeta The metadata that will be created on vineyardd.
- Returns
The result created metadata.
- delete()
- delete(object_id: ObjectID or List[ObjectID], force: bool = false,
- deep: bool = true) -> None
Delete the specific vineyard object.
- Parameters
object_id – ObjectID or list of ObjectID Objects that will be deleted. The
object_id
can be a singleObjectID
, or a list ofObjectID
.force – bool Forcedly delete an object means the member will be recursively deleted even if the member object is also referred by others. The default value is
True
.deep –
bool Deeply delete an object means we will deleting the members recursively. The default value is
True
.Note that when deleting objects which have direct blob members, the processing on those blobs yields a “deep” behavior.
- delete(object_meta: ObjectMeta, force: bool = false, deep: bool = true)
- -> None
Delete the specific vineyard object.
- Parameters
object_meta – The corresponding object meta to delete.
- delete(object: Object, force: bool = false, deep: bool = true) → None
Delete the specific vineyard object.
- Parameters
object – The corresponding object meta to delete.
- drop_name()
- drop_name(name: str or ObjectName) → None
Remove the association of the given name.
- Parameters
name – str The name that will be removed.
- exists()
- exists(object_id: ObjectID) → bool
Whether the given object exists.
- Parameters
object_id – ObjectID The object id to check if exists.
- Returns
True
when the specified object exists.- Return type
bool
- find_shared_memory(target: ptr) → bool
Find the corresponding blob (if exists) of the given pointer.
- Parameters
target – address, in int format The given address.
- Returns
ObjectID
- get(object_id, resolver=None, **kw)
Get vineyard object as python value.
>>> arr_id = vineyard.ObjectID('00002ec13bc81226') >>> arr = client.get(arr_id) >>> arr array([0, 1, 2, 3, 4, 5, 6, 7])
- Parameters
client – IPCClient or RPCClient The vineyard client to use.
object_id – ObjectID The object id that will be obtained from vineyard.
resolver – When retrieving vineyard object, an optional resolver can be specified. If no resolver given, the default resolver context will be used.
kw – User-specific argument that will be passed to the builder.
- Returns
A python object that return by the resolver, by resolving an vineyard object.
- get_meta()
- get_meta(object_id: ObjectID, sync_remote: bool = False) → ObjectMeta
Get object metadata from vineyard.
- Parameters
object_id – ObjectID The object id to get.
sync_remote – bool If the target object is a remote object,
code_remote=True
will force a meta synchronization on the vineyard server. Default isFalse
.
- Returns
ObjectMeta
- get_metas()
- get_metas(object_ids: List[ObjectID], sync_remote: bool = False)
- -> List[ObjectMeta]
Get metadatas of multiple objects from vineyard.
- Paramters:
- object_ids: List[ObjectID]
The object ids to get.
- sync_remote: bool
If the target object is a remote object,
code_remote=True
will force a meta synchronization on the vineyard server. Default isFalse
.
- Returns
List[ObjectMeta]
- get_name()
- get_name(name: str or ObjectName, wait: bool = False) → ObjectID
Get the associated object id of the given name.
- Parameters
name – str The name that will be queried.
wait – bool Whether to wait util the name appears, if wait, the request will be blocked until the name been registered.
- Returns
The associated object id with the name.
- Return type
- get_object()
-
Get object from vineyard.
- Parameters
object_id – ObjectID The object id to get.
- Returns
Object
- get_objects()
-
Get multiple objects from vineyard.
- Paramters:
object_ids: List[ObjectID]
- Returns
List[Object]
- property instance_id
The instance id of the connected vineyard server.
- property ipc_socket
The UNIX domain socket location of connected vineyard server.
- is_shared_memory(target: ptr) → bool
Check if the address is on the shared memory region.
- Parameters
target – address, in int format The given address.
- Returns
bool
- list_metadatas()
- list_metadatas(pattern: str, regex: bool = False, limit: int = 5,
- nobuffer: bool = False) -> List[Object]
List all objects in current vineyard server.
- Parameters
pattern – str The pattern string that will be matched against the object’s typename.
regex – bool Whether the pattern is a regex expression, otherwise the pattern will be used as wildcard pattern. Default value is False.
limit – int The limit to list. Default value is 5.
nobuffer – bool Whether to fill the buffers in returned object metadatas. Default value is False.
- Returns
List[ObjectMeta]
- list_objects()
- list_objects(pattern: str, regex: bool = False, limit: int = 5)
- -> List[Object]
List all objects in current vineyard server.
- Parameters
pattern – str The pattern string that will be matched against the object’s typename.
regex – bool Whether the pattern is a regex expression, otherwise the pattern will be used as wildcard pattern. Default value is False.
limit – int The limit to list. Default value is 5.
- Returns
List[Object]
- property meta
The metadata information of the vineyard server. The value is a nested dict, the first-level key is the instance id, and the second-level key is the cluster metadata fields.
>>> client.meta { 14: { 'hostid': '54058007061210', 'hostname': '127.0.0.1', 'timestamp': '6882550126788354072' }, 15: { 'hostid': '48843417291806', 'hostname': '127.0.0.1', 'timestamp': '6882568290204737414' } }
- persist()
- persist(object_id: ObjectID) → None
Persist the object of the given object id. After persisting, the object will be visible by clients that connect to other vineyard server instances.
- Parameters
object_id – ObjectID The object that will be persist.
- persist(object_meta: ObjectMeta) → None
Persist the given object.
- Parameters
object_meta – ObjectMeta The object that will be persist.
- persist(object: Object) → None
Persist the given object.
- Parameters
object – Object The object that will be persist.
- put(value, builder=None, **kw)
Put python value to vineyard.
>>> arr = np.arange(8) >>> arr_id = client.put(arr) >>> arr_id 00002ec13bc81226
- Parameters
client – IPCClient The vineyard client to use.
value – The python value that will be put to vineyard. Supported python value types are decided by modules that registered to vineyard. By default, python value can be put to vineyard after serialized as a bytes buffer using pickle.
builder – When putting python value to vineyard, an optional builder can be specified to tell vineyard how to construct the corresponding vineyard
Object
. If not specified, the default builder context will be used to select a proper builder.kw – User-specific argument that will be passed to the builder.
- Returns
The result object id will be returned.
- Return type
- put_name()
- put_name(object: ObjectID or ObjectMeta or Object,
- name: str or ObjectName) -> None
Associate the given object id with a name. An
ObjectID
can be associated with more than one names.- Parameters
object_id – ObjectID
name – str
- reset()
- reset() → None
Alias of :method:`ClientBase.clear`.
- property rpc_endpoint
The RPC endpoint of the connected vineyard server.
- shallow_copy()
-
Create a shallow copy of the given vineyard object.
- Parameters
object_id – ObjectID The vineyard object that is requested to be shallow-copied.
- Returns
The object id of newly shallow-copied vineyard object.
- Return type
Create a shallow copy of the given vineyard object, with extra metadata.
- Parameters
object_id – ObjectID The vineyard object that is requested to be shallow-copied.
extra_metadata – dict Extra metadata to apply to the newly created object. The fields of extra metadata must be primitive types, e.g., string, number, and cannot be array or dict.
- Returns
The object id of newly shallow-copied vineyard object.
- Return type
- property status
The status the of connected vineyard server, returns a
InstanceStatus
.See also
InstanceStatus
- sync_meta()
- sync_meta() → None
Synchronize remote metadata to local immediately.
- property version
The version number string of connected vineyard server, in the format of semver:
MAJOR.MINOR.PATCH
.
- class vineyard.RPCClient
RPC client that connects to vineyard instance’s RPC endpoints.
The RPC client can only access the metadata of objects, any access to the blob payload will trigger a
RuntimeError
exception.- clear()
- clear() → None
Drop all objects that visible to the current instance in the vineyard cluster.
- close()
Close the client.
- property connected
Whether the client instance has been connected to the vineyard server.
- create_metadata()
- create_metadata(metadata: ObjectMeta) → ObjectMeta
Create metadata in vineyardd.
- Parameters
metadata – ObjectMeta The metadata that will be created on vineyardd.
- Returns
The result created metadata.
- delete()
- delete(object_id: ObjectID or List[ObjectID], force: bool = false,
- deep: bool = true) -> None
Delete the specific vineyard object.
- Parameters
object_id – ObjectID or list of ObjectID Objects that will be deleted. The
object_id
can be a singleObjectID
, or a list ofObjectID
.force – bool Forcedly delete an object means the member will be recursively deleted even if the member object is also referred by others. The default value is
True
.deep –
bool Deeply delete an object means we will deleting the members recursively. The default value is
True
.Note that when deleting objects which have direct blob members, the processing on those blobs yields a “deep” behavior.
- delete(object_meta: ObjectMeta, force: bool = false, deep: bool = true)
- -> None
Delete the specific vineyard object.
- Parameters
object_meta – The corresponding object meta to delete.
- delete(object: Object, force: bool = false, deep: bool = true) → None
Delete the specific vineyard object.
- Parameters
object – The corresponding object meta to delete.
- drop_name()
- drop_name(name: str or ObjectName) → None
Remove the association of the given name.
- Parameters
name – str The name that will be removed.
- exists()
- exists(object_id: ObjectID) → bool
Whether the given object exists.
- Parameters
object_id – ObjectID The object id to check if exists.
- Returns
True
when the specified object exists.- Return type
bool
- get(object_id, resolver=None, **kw)
Get vineyard object as python value.
>>> arr_id = vineyard.ObjectID('00002ec13bc81226') >>> arr = client.get(arr_id) >>> arr array([0, 1, 2, 3, 4, 5, 6, 7])
- Parameters
client – IPCClient or RPCClient The vineyard client to use.
object_id – ObjectID The object id that will be obtained from vineyard.
resolver – When retrieving vineyard object, an optional resolver can be specified. If no resolver given, the default resolver context will be used.
kw – User-specific argument that will be passed to the builder.
- Returns
A python object that return by the resolver, by resolving an vineyard object.
- get_meta()
- get_meta(object_id: ObjectID) → ObjectMeta
Get object metadata from vineyard.
- Parameters
object_id – ObjectID The object id to get.
- Returns
ObjectMeta
- get_metas()
- get_metas(object_ids: List[ObjectID] -> List[ObjectMeta]
Get metadatas of multiple objects from vineyard.
- Paramters:
object_ids: List[ObjectID]
- Returns
List[ObjectMeta]
- get_name()
- get_name(name: str or ObjectName, wait: bool = False) → ObjectID
Get the associated object id of the given name.
- Parameters
name – str The name that will be queried.
wait – bool Whether to wait util the name appears, if wait, the request will be blocked until the name been registered.
- Returns
The associated object id with the name.
- Return type
- get_object()
-
Get object from vineyard.
- Parameters
object_id – ObjectID The object id to get.
- Returns
Object
- get_objects()
-
Get multiple objects from vineyard.
- Paramters:
object_ids: List[ObjectID]
- Returns
List[Object]
- property instance_id
The instance id of the connected vineyard server.
- property ipc_socket
The UNIX domain socket location of connected vineyard server.
- list_metadatas()
- list_metadatas(pattern: str, regex: bool = False, limit: int = 5,
- nobuffer: bool = False) -> List[Object]
List all objects in current vineyard server.
- Parameters
pattern – str The pattern string that will be matched against the object’s typename.
regex – bool Whether the pattern is a regex expression, otherwise the pattern will be used as wildcard pattern. Default value is False.
limit – int The limit to list. Default value is 5.
nobuffer – bool Whether to fill the buffers in returned object metadatas. Default value is False.
- Returns
List[ObjectMeta]
- list_objects()
- list_objects(pattern: str, regex: bool = False, limit: int = 5)
- -> List[Object]
List all objects in current vineyard server.
- Parameters
pattern – str The pattern string that will be matched against the object’s typename.
regex – bool Whether the pattern is a regex expression, otherwise the pattern will be used as wildcard pattern. Default value is False.
limit – int The limit to list. Default value is 5.
- Returns
List[Object]
- property meta
The metadata information of the vineyard server. The value is a nested dict, the first-level key is the instance id, and the second-level key is the cluster metadata fields.
>>> client.meta { 14: { 'hostid': '54058007061210', 'hostname': '127.0.0.1', 'timestamp': '6882550126788354072' }, 15: { 'hostid': '48843417291806', 'hostname': '127.0.0.1', 'timestamp': '6882568290204737414' } }
- persist()
- persist(object_id: ObjectID) → None
Persist the object of the given object id. After persisting, the object will be visible by clients that connect to other vineyard server instances.
- Parameters
object_id – ObjectID The object that will be persist.
- persist(object_meta: ObjectMeta) → None
Persist the given object.
- Parameters
object_meta – ObjectMeta The object that will be persist.
- persist(object: Object) → None
Persist the given object.
- Parameters
object – Object The object that will be persist.
- put(value, builder=None, **kw)
Put python value to vineyard.
>>> arr = np.arange(8) >>> arr_id = client.put(arr) >>> arr_id 00002ec13bc81226
- Parameters
client – IPCClient The vineyard client to use.
value – The python value that will be put to vineyard. Supported python value types are decided by modules that registered to vineyard. By default, python value can be put to vineyard after serialized as a bytes buffer using pickle.
builder – When putting python value to vineyard, an optional builder can be specified to tell vineyard how to construct the corresponding vineyard
Object
. If not specified, the default builder context will be used to select a proper builder.kw – User-specific argument that will be passed to the builder.
- Returns
The result object id will be returned.
- Return type
- put_name()
- put_name(object: ObjectID or ObjectMeta or Object,
- name: str or ObjectName) -> None
Associate the given object id with a name. An
ObjectID
can be associated with more than one names.- Parameters
object_id – ObjectID
name – str
- property remote_instance_id
The instance id of the connected remote vineyard server.
- reset()
- reset() → None
Alias of :method:`ClientBase.clear`.
- property rpc_endpoint
The RPC endpoint of the connected vineyard server.
- shallow_copy()
-
Create a shallow copy of the given vineyard object.
- Parameters
object_id – ObjectID The vineyard object that is requested to be shallow-copied.
- Returns
The object id of newly shallow-copied vineyard object.
- Return type
Create a shallow copy of the given vineyard object, with extra metadata.
- Parameters
object_id – ObjectID The vineyard object that is requested to be shallow-copied.
extra_metadata – dict Extra metadata to apply to the newly created object. The fields of extra metadata must be primitive types, e.g., string, number, and cannot be array or dict.
- Returns
The object id of newly shallow-copied vineyard object.
- Return type
- property status
The status the of connected vineyard server, returns a
InstanceStatus
.See also
InstanceStatus
- sync_meta()
- sync_meta() → None
Synchronize remote metadata to local immediately.
- property version
The version number string of connected vineyard server, in the format of semver:
MAJOR.MINOR.PATCH
.
State of server
- class vineyard.InstanceStatus
InstanceStatus
represents the status of connected vineyard instance, including the instance identity, memory statistics and workloads on this instance.>>> status = client.status >>> print(status) InstanceStatus: instance_id: 5 deployment: local memory_usage: 360 memory_limit: 268435456 deferred_requests: 0 ipc_connections: 1 rpc_connections: 0 >>> status.instance_id 5 >>> status.deployment 'local' >>> status.memory_usage 360 >>> status.memory_limit 268435456 >>> status.deferred_requests 0 >>> status.ipc_connections 1 >>> status.rpc_connections 0
- __init__(*args, **kwargs)
Initialize self. See help(type(self)) for accurate signature.
- property deferred_requests
Report number of waiting requests of current vineyardd instance.
- property deployment
The deployment mode of the connected vineyardd cluster, can be
"local"
and"distributed"
.
- property instance_id
Return the instance id of vineyardd that the client is connected to.
- property ipc_connections
Report number of alive IPC connections on the current vineyardd instance.
- property memory_limit
Report memory limit (in bytes) of current vineyardd instance.
- property memory_usage
Report memory usage (in bytes) of current vineyardd instance.
- property rpc_connections
Report number of alive RPC connections on the current vineyardd instance.
Primitives
- class vineyard.Blob
Blob
in vineyard is a consecutive readonly shared memory.- property address
The memory address value of this blob.
- property buffer
The readonly buffer hebind this blob. The result buffer has type
pyarrow::Buffer
.
- static empty()
Whether the blob is an empty blob, i.e., the size of this blob is 0.
- property size
Size of the blob.
- class vineyard.BlobBuilder
BlobBuilder
is the builder for creating a finally immutable blob in vineyard server.A
BlobBuilder
can only be explicitly created using theIPCClient.create_blob()
.See also
IPCClient.create_blob IPCClient.create_empty_blob
- abort()
Abort the blob builder if it is not sealed yet.
- property address
The memory address value of this blob builder.
- property buffer
The writeable buffer hebind this blob builder. The result buffer has type
pyarrow::Buffer
, and it is a mutable one.
- copy()
- copy(self, offset: int, ptr: int, size: int)
Copy the given address to the given offset.
- property id
ObjectID of this blob builder.
- property size
Size of this blob builder.
Resolvers and Builders
- vineyard.core.resolver.resolver_context(resolvers=None, base=None)[source]
Open a new context for register resolvers, without populting outside the global environment.
The
resolver_context
can be useful when users have more than more resolver for a certain type, e.g., thevineyard::Tensor
object can be resolved asnumpy.ndarray
orxgboost::DMatrix
.We could have
def numpy_resolver(obj): ... default_resolver_context.register('vineyard::Tensor', numpy_resolver)
and
def xgboost_resolver(obj): ... default_resolver_context.register('vineyard::Tensor', xgboost_resolver)
Obviously there’s a conflict, and the stackable
resolver_context
could help there,with resolver_context({'vineyard::Tensor', xgboost_resolver}): ...
Assuming the default context resolves
vineyard::Tensor
tonumpy.ndarray
, inside thewith resolver_context
thevineyard::Tensor
will be resolved toxgboost::DMatrix
, and after exiting the context the global environment will be restored back as default.The
with resolver_context
is nestable as well.See also
builder_context driver_context
- class vineyard.core.builder.BuilderContext[source]
- register(type_id, builder)[source]
Register a Python type to the builder context.
- Parameters
type_id (Python type) – Like int, or numpy.ndarray
builder (callable, e.g., a method, callable object) – A builder translates a python object to vineyard, it accepts a Python value as parameter, and returns an vineyard object as result.
- run(client, value, **kw)[source]
Follows the MRO to find the proper builder for given python value.
Here “Follows the MRO” implies:
If the type of python value has been found in the context, the registered builder will be used.
If not, it follows the MRO chain from down to top to find a registered Python type and used the associated builder.
When the traversal reaches the
object
type, since there’s a default builder that serialization the python value, the parameter will be serialized and be put into a blob.
Shared Memory
A memoryview of contents of the shared memory block.
Seal the shared memory to make it visible for other processes.
Unique name that identifies the shared memory block.
Size in bytes.
Requests that the underlying shared memory block be destroyed.
Pattern for a mutable list-like object shareable via a shared memory block. It differs from the built-in list type in that these lists can not change their overall length (i.e. no append, insert, etc.)
Because values are packed into a memoryview as bytes, the struct packing format for any storable value must require no more than 8 characters to describe its format.
The ShareableList in vineyard differs slightly with its equivalent in the multiprocessing.shared_memory.ShareableList, as it becomes immutable after obtaining from the vineyard backend.
See also
multiprocessing.shared_memory.ShareableList
Make the shareable list immutable and visible for other vineyard clients.
Deployment
- vineyard.init(num_instances=1, **kw)[source]
Launching a local vineyardd instance and get a client as easy as possible
In a clean environment, simply use:
vineyard.init()
It will launch a local vineyardd and return a connected client to the vineyardd.
It will also setup the environment variable
VINEYARD_IPC_SOCKET
.For the case to establish a local vineyard cluster consists of multiple vineyardd instances, using the
num_instances
parameter:client1, client2, client3 = vineyard.init(num_instances=3)
In this case, three vineyardd instances will be launched.
The init method can only be called once in a process, to get the established sockets or clients later in the process, use
get_current_socket
orget_current_client
respectively.
- vineyard.get_current_client()[source]
Get current vineyard IPC clients established by
vineyard.init()
.- Raises
ValueError if vineyard is not initialized. –
- vineyard.get_current_socket()[source]
Get current vineyard UNIX-domain socket established by
vineyard.init()
.- Raises
ValueError if vineyard is not initialized. –
- vineyard.deploy.local.start_vineyardd(etcd_endpoints=None, etcd_prefix=None, vineyardd_path=None, size='256M', socket=None, rpc=True, rpc_socket_port=9600, debug=False)[source]
Launch a local vineyard cluster.
- Parameters
etcd_endpoint – str Launching vineyard using specified etcd endpoints. If not specified, vineyard will launch its own etcd instance.
etcd_prefix – str Specify a common prefix to establish a local vineyard cluster.
vineyardd_path – str Location of vineyard server program. If not specified, vineyard will use its own bundled vineyardd binary.
size –
int The memory size limit for vineyard’s shared memory. The memory size can be a plain integer or as a fixed-point number using one of these suffixes:
E, P, T, G, M, K.
You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi, Mi, Ki.
For example, the following represent roughly the same value:
128974848, 129k, 129M, 123Mi, 1G, 10Gi, ...
socket –
str The UNIX domain socket socket path that vineyard server will listen on. Default is None.
When the socket parameter is None, a random path under temporary directory will be generated and used.
rpc_socket_port – int The port that vineyard will use to privode RPC service.
debug – bool Whether print debug logs.
- Returns
Yields a tuple with the subprocess as the first element and the UNIX-domain IPC socket as the second element.
- Return type
(proc, socket)
- vineyard.deploy.distributed.start_vineyardd(hosts=None, etcd_endpoints=None, vineyardd_path=None, size='256M', socket='/var/run/vineyard.sock', rpc_socket_port=9600, debug=False)[source]
Launch a local vineyard cluster in a distributed fashion.
- Parameters
hosts – list of str A list of machines to launch vineyard server.
etcd_endpoint – str Launching vineyard using specified etcd endpoints. If not specified, vineyard will launch its own etcd instance.
vineyardd_path – str Location of vineyard server program. If not specified, vineyard will use its own bundled vineyardd binary.
size –
int The memory size limit for vineyard’s shared memory. The memory size can be a plain integer or as a fixed-point number using one of these suffixes:
E, P, T, G, M, K.
You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi, Mi, Ki.
For example, the following represent roughly the same value:
128974848, 129k, 129M, 123Mi, 1G, 10Gi, ...
socket – str The UNIX domain socket socket path that vineyard server will listen on.
rpc_socket_port – int The port that vineyard will use to privode RPC service.
debug – bool Whether print debug logs.
IO Facilities
- vineyard.io.open(path, *args, mode='r', handlers=None, **kwargs)[source]
Open a path as a reader or writer, depends on the parameter
mode
. Ifmode
isr
, it will open a stream for read, and open a stream for write whenmode
isw
.- Parameters
path (str) – Path to open.
mode (char) – Mode about how to open the path,
r
is for read andw
for write.vineyard_ipc_socket (str) – Vineyard’s IPC socket location.
vineyard_endpoint (str) – Vineyard’s RPC socket address.
handlers – A dict that will be filled with a handler that contains the process handler of the underlying read/write process that can be `join`ed to capture the possible errors during the I/O proceeding.
See also
- vineyard.io.read(path, *args, handlers=None, **kwargs)[source]
Open a path and read it as a single stream.
- Parameters
path (str) – Path to read, the last reader registered for the scheme of the path will be used.
vineyard_ipc_socket (str) – The local or remote vineyard’s IPC socket location that the remote readers will use to establish connections with the vineyard server.
vineyard_endpoint (str, optional) – An optional address of vineyard’s RPC socket, which will be used for retrieving server’s information on the client side. If not provided, the vineyard_ipc_socket will be used, or it will tries to discovery vineyard’s IPC or RPC endpoints from environment variables.
- vineyard.io.write(path, stream, *args, handlers=None, **kwargs)[source]
Write the stream to a given path.
- Parameters
path (str) – Path to write, the last writer registered for the scheme of the path will be used.
stream (vineyard stream) – Stream that produces the data to write.
vineyard_ipc_socket (str) – The local or remote vineyard’s IPC socket location that the remote readers will use to establish connections with the vineyard server.
vineyard_endpoint (str, optional) – An optional address of vineyard’s RPC socket, which will be used for retrieving server’s information on the client side. If not provided, the vineyard_ipc_socket will be used, or it will tries to discovery vineyard’s IPC or RPC endpoints from environment variables.