Use vineyard operator#

Vineyard operator has been designed to manage vineyard components within Kubernetes. This tutorial provides a step-by-step guide on how to effectively utilize the vineyard operator. For more details, please refer to Vineyard Operator.

Step 0: (optional) Initialize Kubernetes Cluster#

If you don’t have a Kubernetes cluster readily available, we highly recommend using kind to create one. Before setting up the Kubernetes cluster, please ensure you have the following tools installed:

  • kubectl: version >= 1.19.2

  • kind: version >= 0.14.0

  • docker: version >= 0.19.0

Utilize kind (v0.14.0) to create a Kubernetes cluster consisting of 4 nodes (1 master node and 3 worker nodes):

$ cat > kind-config.yaml <<EOF
# four node (three workers) cluster config
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker
- role: worker
EOF
$ kind create cluster --config kind-config.yaml

Expected output

Creating cluster "kind" ...
  Ensuring node image (kindest/node:v1.24.0) 🖼
  Preparing nodes 📦
  Writing configuration 📜
  Starting control-plane 🕹️
  Installing CNI 🔌
  Installing StorageClass 💾
 Set kubectl context to "kind-kind"
 You can now use your cluster with:

 kubectl cluster-info --context kind-kind

 Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community 🙂

Note

The kind cluster’s config file is stored in ~/.kube/config, so you can use the kubectl directly as it’s the default config path.

Check all kubernetes pods.

$ kubectl get pod -A

Expected output

NAMESPACE            NAME                                         READY   STATUS    RESTARTS   AGE
kube-system          coredns-6d4b75cb6d-k2sk2                     1/1     Running   0          38s
kube-system          coredns-6d4b75cb6d-xm4dt                     1/1     Running   0          38s
kube-system          etcd-kind-control-plane                      1/1     Running   0          52s
kube-system          kindnet-fp24b                                1/1     Running   0          19s
kube-system          kindnet-h6swp                                1/1     Running   0          39s
kube-system          kindnet-mtkd4                                1/1     Running   0          19s
kube-system          kindnet-zxxpd                                1/1     Running   0          19s
kube-system          kube-apiserver-kind-control-plane            1/1     Running   0          52s
kube-system          kube-controller-manager-kind-control-plane   1/1     Running   0          53s
kube-system          kube-proxy-6zgq2                             1/1     Running   0          19s
kube-system          kube-proxy-8vghn                             1/1     Running   0          39s
kube-system          kube-proxy-c7vz5                             1/1     Running   0          19s
kube-system          kube-proxy-kd4zz                             1/1     Running   0          19s
kube-system          kube-scheduler-kind-control-plane            1/1     Running   0          52s
local-path-storage   local-path-provisioner-9cd9bd544-2vrtq       1/1     Running   0          38s

Check all kubernetes nodes.

$ kubectl get nodes -A

Expected output

NAME                 STATUS   ROLES           AGE     VERSION
kind-control-plane   Ready    control-plane   2m30s   v1.24.0
kind-worker          Ready    <none>          114s    v1.24.0
kind-worker2         Ready    <none>          114s    v1.24.0
kind-worker3         Ready    <none>          114s    v1.24.0

Step 1: Deploy the Vineyard Operator#

Create a dedicated namespace for the Vineyard Operator.

$ kubectl create namespace vineyard-system

Expected output

namespace/vineyard-system created

The operator needs a certificate created by cert-manager for webhook(https), so we have to install the cert-manager(v1.9.1) at first.

$ kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.9.1/cert-manager.yaml

Expected output

namespace/cert-manager created
customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io created
...
validatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created

Check whether all cert-manager pods are running.

$ kubectl get pod -n cert-manager

Expected output

NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-5dd59d9d9b-cwp8n              1/1     Running   0          58s
cert-manager-cainjector-8696fc9f89-tvftj   1/1     Running   0          58s
cert-manager-webhook-7d4b5b8c56-htchs      1/1     Running   0          58s

Vineyard CRDs、Controllers、Webhooks and Scheduler are packaged by helm, you could deploy all resources as follows.

$ helm repo add vineyard https://vineyard.oss-ap-southeast-1.aliyuncs.com/charts/

Expected output

"vineyard" has been added to your repositories

Update the vineyard operator chart to the newest one.

$ helm repo update

Expected output

Hang tight while we grab the latest from your chart repositories...
 ...Successfully got an update from the "vineyard" chart repository
Update Complete. ⎈Happy Helming!⎈

Deploy the vineyard operator 0.11.4 in the namespace vineyard-system.

$ helm install vineyard-operator vineyard/vineyard-operator -n vineyard-system --version 0.11.4

Expected output

NAME: vineyard-operator
LAST DEPLOYED: Wed Jan  4 16:41:45 2023
NAMESPACE: vineyard-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Thanks for installing VINEYARD-OPERATOR:0.11.4, release at namespace: vineyard-system, name: vineyard-operator.

To learn more about the release, try:

$ helm status vineyard-operator -n vineyard-system   # get status of running vineyard operator
$ helm get all vineyard-operator -n vineyard-system  # get all deployment yaml of vineyard operator

To uninstall the release, try:

$ helm uninstall vineyard-operator -n vineyard-system

You could get all details about vineyard operator in the doc Vineyard Operator, just have fun with vineyard operator!

Check the status of all vineyard resources created by helm:

$ kubectl get all -n vineyard-system

Expected output

NAME                                   READY   STATUS    RESTARTS   AGE
pod/vineyard-operator-cbcd58cb-5zs84   2/2     Running   0          4m56s

NAME                                        TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
service/vineyard-operator-metrics-service   ClusterIP   10.96.23.137   <none>        8443/TCP   4m56s
service/vineyard-operator-webhook-service   ClusterIP   10.96.215.18   <none>        443/TCP    4m56s

NAME                                READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/vineyard-operator   1/1     1            1           4m56s

NAME                                         DESIRED   CURRENT   READY   AGE
replicaset.apps/vineyard-operator-cbcd58cb   1         1         1       4m56s

Step 2: Deploy a Vineyard Cluster#

After successfully installing the Vineyard operator as described in the previous step, you can now proceed to deploy a Vineyard cluster. To create a cluster with two Vineyard instances, simply create a Vineyardd Custom Resource (CR) as shown below.

$ cat <<EOF | kubectl apply -f -
apiVersion: k8s.v6d.io/v1alpha1
kind: Vineyardd
metadata:
name: vineyardd-sample
namespace: vineyard-system
spec:
# vineyard instances
replicas: 2
EOF

Expected output

vineyardd.k8s.v6d.io/vineyardd-sample created

Check the status of all relevant resources managed by the vineyardd-sample cr.

$ kubectl get all -l app.kubernetes.io/instance=vineyardd -n vineyard-system

Expected output

NAME                                   READY   STATUS    RESTARTS   AGE
pod/vineyardd-sample-879798cb6-qpvtw   1/1     Running   0          2m59s
pod/vineyardd-sample-879798cb6-x4m2x   1/1     Running   0          2m59s

NAME                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/vineyardd-sample   2/2     2            2           2m59s

NAME                                         DESIRED   CURRENT   READY   AGE
replicaset.apps/vineyardd-sample-879798cb6   2         2         2       2m59s

Step 3: Connect to Vineyard#

Vineyard currently supports clients in various languages, including mature support for C++ and Python, as well as experimental support for Java, Golang, and Rust. In this tutorial, we will demonstrate how to connect to a Vineyard cluster using the Python client. Vineyard provides two connection methods: IPC and RPC. In the following sections, we will explore both methods.

First, let’s deploy the Python client on two Vineyard nodes as follows.

$ cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: vineyard-python-client
namespace:  vineyard-system
spec:
selector:
    matchLabels:
    app: vineyard-python-client
replicas: 2
template:
    metadata:
    annotations:
        scheduling.k8s.v6d.io/required: none
    labels:
        app: vineyard-python-client
        # related to which vineyard cluster
        scheduling.k8s.v6d.io/vineyardd-namespace: vineyard-system
        scheduling.k8s.v6d.io/vineyardd: vineyardd-sample
        scheduling.k8s.v6d.io/job: v6d-workflow-demo-job1
    spec:
    # use the vineyard scheduler to deploy the pod on the vineyard cluster.
    schedulerName: vineyard-scheduler
    containers:
    - name: vineyard-python
        imagePullPolicy: IfNotPresent
        image: vineyardcloudnative/vineyard-python:v0.11.4
        command:
        - /bin/bash
        - -c
        - sleep infinity
        volumeMounts:
        - mountPath: /var/run
            name: vineyard-sock
    volumes:
    - name: vineyard-sock
        hostPath:
        path: /var/run/vineyard-kubernetes/vineyard-system/vineyardd-sample
EOF

Expected output

pod/vineyard-python-client created

Wait for the vineyard python client pod ready.

$ kubectl get pod -l app=vineyard-python-client -n vineyard-system

Expected output

NAME                                      READY   STATUS    RESTARTS   AGE
vineyard-python-client-6fd8c47c98-7btkv   1/1     Running   0          93s

Use the kubectl exec command to enter the first vineyard python client pod.

$ kubectl exec -it $(kubectl get pod -l app=vineyard-python-client -n vineyard-system -oname | head -n 1 | awk -F '/' '{print $2}') -n vineyard-system /bin/bash

After entering the shell, you can connect to the vineyard cluster,

In [1]: import numpy as np
In [2]: import vineyard

In [3]: client = vineyard.connect('/var/run/vineyard.sock')

In [4]: objid = client.put(np.zeros(8))

In [5]: # persist the object to make it visible to form the global object
In [6]: client.persist(objid)

In [7]: objid
Out[7]: o001027d7c86a49f0

In [8]: # get meta info
In [9]: meta = client.get_meta(objid)
In [10]: meta
Out[10]:
{
    "buffer_": {
        "id": "o801027d7c85c472e",
        "instance_id": 1,
        "length": 0,
        "nbytes": 0,
        "transient": true,
        "typename": "vineyard::Blob"
    },
    "global": false,
    "id": "o001027d7c86a49f0",
    "instance_id": 1,
    "nbytes": 64,
    "order_": "\"C\"",
    "partition_index_": "[]",
    "shape_": "[8]",
    "signature": 4547407361228035,
    "transient": false,
    "typename": "vineyard::Tensor<double>",
    "value_type_": "float64",
    "value_type_meta_": "<f8"
}

Open another terminal and enter the second vineyard python client pod.

$ kubectl exec -it $(kubectl get pod -l app=vineyard-python-client -n vineyard-system -oname | tail -n 1 | awk -F '/' '{print $2}') -n vineyard-system /bin/bash

Expected output

kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.

Also, you can connect to the vineyard cluster by RPC and get the metadata of above object as follows.

In [1]: import vineyard

In [2]: rpc_client = vineyard.connect('vineyardd-sample-rpc.vineyard-system',9600)

In [3]: # use the object id created by another vineyard instance here
In [4]: meta = rpc_client.get_meta(vineyard._C.ObjectID('o001027d7c86a49f0'))
In [5]: meta
Out[5]:
{
    "buffer_": {
        "id": "o801027d7c85c472e",
        "instance_id": 1,
        "length": 0,
        "nbytes": 0,
        "transient": true,
        "typename": "vineyard::Blob"
    },
    "global": false,
    "id": "o001027d7c86a49f0",
    "instance_id": 1,
    "nbytes": 64,
    "order_": "\"C\"",
    "partition_index_": "[]",
    "shape_": "[8]",
    "signature": 4547407361228035,
    "transient": false,
    "typename": "vineyard::Tensor<double>",
    "value_type_": "float64",
    "value_type_meta_": "<f8"
}

For more examples, please refer the vineyard data accessing.

Step 4: Cleanup#

  • Destroy the vineyard operator via helm:

$ helm uninstall vineyard-operator -n vineyard-system

Expected output

release "vineyard-operator" uninstalled
  • Delete the namespace:

$ kubectl delete namespace vineyard-system

Expected output

namespace "vineyard-system" deleted