Use vineyard operator

Vineyard operator has been designed to manage vineyard components within Kubernetes. This tutorial provides a step-by-step guide on how to effectively utilize the vineyard operator. For more details, please refer to Vineyard Operator.

Step 0: (optional) Initialize Kubernetes Cluster

If you don’t have a Kubernetes cluster readily available, we highly recommend using kind to create one. Before setting up the Kubernetes cluster, please ensure you have the following tools installed:

  • kubectl: version >= 1.19.2

  • kind: version >= 0.14.0

  • docker: version >= 0.19.0

Utilize kind (v0.14.0) to create a Kubernetes cluster consisting of 4 nodes (1 master node and 3 worker nodes):

$ cat > kind-config.yaml <<EOF
# four node (three workers) cluster config
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker
- role: worker
EOF
$ kind create cluster --config kind-config.yaml

Expected output

Creating cluster "kind" ...
  Ensuring node image (kindest/node:v1.24.0) 🖼
  Preparing nodes 📦
  Writing configuration 📜
  Starting control-plane 🕹️
  Installing CNI 🔌
  Installing StorageClass 💾
 Set kubectl context to "kind-kind"
 You can now use your cluster with:

 kubectl cluster-info --context kind-kind

 Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community 🙂

Note

The kind cluster’s config file is stored in ~/.kube/config, so you can use the kubectl directly as it’s the default config path.

Check all kubernetes pods.

$ kubectl get pod -A

Expected output

NAMESPACE            NAME                                         READY   STATUS    RESTARTS   AGE
kube-system          coredns-6d4b75cb6d-k2sk2                     1/1     Running   0          38s
kube-system          coredns-6d4b75cb6d-xm4dt                     1/1     Running   0          38s
kube-system          etcd-kind-control-plane                      1/1     Running   0          52s
kube-system          kindnet-fp24b                                1/1     Running   0          19s
kube-system          kindnet-h6swp                                1/1     Running   0          39s
kube-system          kindnet-mtkd4                                1/1     Running   0          19s
kube-system          kindnet-zxxpd                                1/1     Running   0          19s
kube-system          kube-apiserver-kind-control-plane            1/1     Running   0          52s
kube-system          kube-controller-manager-kind-control-plane   1/1     Running   0          53s
kube-system          kube-proxy-6zgq2                             1/1     Running   0          19s
kube-system          kube-proxy-8vghn                             1/1     Running   0          39s
kube-system          kube-proxy-c7vz5                             1/1     Running   0          19s
kube-system          kube-proxy-kd4zz                             1/1     Running   0          19s
kube-system          kube-scheduler-kind-control-plane            1/1     Running   0          52s
local-path-storage   local-path-provisioner-9cd9bd544-2vrtq       1/1     Running   0          38s

Check all kubernetes nodes.

$ kubectl get nodes -A

Expected output

NAME                 STATUS   ROLES           AGE     VERSION
kind-control-plane   Ready    control-plane   2m30s   v1.24.0
kind-worker          Ready    <none>          114s    v1.24.0
kind-worker2         Ready    <none>          114s    v1.24.0
kind-worker3         Ready    <none>          114s    v1.24.0

Step 1: Deploy the Vineyard Operator

Create a dedicated namespace for the Vineyard Operator.

$ kubectl create namespace vineyard-system

Expected output

namespace/vineyard-system created

The Vineyard CRDs、Controllers、Webhooks and Scheduler are packaged by helm, you could deploy all resources as follows.

Note

The vineyard operator needs permission to create several CRDs and kubernetes resources, before deploying the vineyard operator, please ensure you can create the clusterrole.

$ helm repo add vineyard https://vineyard.oss-ap-southeast-1.aliyuncs.com/charts/

Expected output

"vineyard" has been added to your repositories

Update the vineyard operator chart to the newest one.

$ helm repo update

Expected output

Hang tight while we grab the latest from your chart repositories...
 ...Successfully got an update from the "vineyard" chart repository
Update Complete. ⎈Happy Helming!⎈

Deploy the vineyard operator in the namespace vineyard-system.

$ helm install vineyard-operator vineyard/vineyard-operator -n vineyard-system

Expected output

NAME: vineyard-operator
LAST DEPLOYED: Wed Jan  4 16:41:45 2023
NAMESPACE: vineyard-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Thanks for installing VINEYARD-OPERATOR, release at namespace: vineyard-system, name: vineyard-operator.

To learn more about the release, try:

$ helm status vineyard-operator -n vineyard-system   # get status of running vineyard operator
$ helm get all vineyard-operator -n vineyard-system  # get all deployment yaml of vineyard operator

To uninstall the release, try:

$ helm uninstall vineyard-operator -n vineyard-system

You could get all details about vineyard operator in the doc Vineyard Operator, just have fun with vineyard operator!

Check the status of all vineyard resources created by helm:

$ kubectl get all -n vineyard-system

Expected output

NAME                                                            READY   STATUS    RESTARTS   AGE
pod/vineyard-operator-controller-manager-5bcbb75fb6-cfdpk       2/2     Running   0          2m30s

NAME                                                           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/vineyard-operator-controller-manager-metrics-service   ClusterIP   10.96.153.134   <none>        8443/TCP   2m30s
service/vineyard-operator-webhook-service                      ClusterIP   10.96.9.101     <none>        443/TCP    2m30s

NAME                                                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/vineyard-operator-controller-manager        1/1     1            1           2m30s

NAME                                                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/vineyard-operator-controller-manager-5bcbb75fb6       1         1         1       2m30s

Step 2: Deploy a Vineyard Cluster

After successfully installing the Vineyard operator as described in the previous step, you can now proceed to deploy a Vineyard cluster. To create a cluster with two Vineyard instances, simply create a Vineyardd Custom Resource (CR) as shown below.

$ cat <<EOF | kubectl apply -f -
apiVersion: k8s.v6d.io/v1alpha1
kind: Vineyardd
metadata:
  name: vineyardd-sample
  namespace: vineyard-system
spec:
  # vineyard instances
  replicas: 2
EOF

Expected output

vineyardd.k8s.v6d.io/vineyardd-sample created

Check the status of all relevant resources managed by the vineyardd-sample cr.

$ kubectl get all -l app.kubernetes.io/instance=vineyard-system-vineyardd-sample -n vineyard-system

Expected output

NAME                                   READY   STATUS    RESTARTS   AGE
pod/vineyardd-sample-879798cb6-qpvtw   1/1     Running   0          2m59s
pod/vineyardd-sample-879798cb6-x4m2x   1/1     Running   0          2m59s

NAME                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/vineyardd-sample   2/2     2            2           2m59s

NAME                                         DESIRED   CURRENT   READY   AGE
replicaset.apps/vineyardd-sample-879798cb6   2         2         2       2m59s

Step 3: Connect to Vineyard

Vineyard currently supports clients in various languages, including mature support for C++ and Python, as well as experimental support for Java, Golang, and Rust. In this tutorial, we will demonstrate how to connect to a Vineyard cluster using the Python client. Vineyard provides two connection methods: IPC and RPC. In the following sections, we will explore both methods.

First, let’s deploy the Python client on two Vineyard nodes as follows.

$ cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vineyard-python-client
  namespace:  vineyard-system
spec:
  selector:
    matchLabels:
      app: vineyard-python-client
  replicas: 2
  template:
    metadata:
      labels:
        app: vineyard-python-client
        # related to which vineyard cluster
        scheduling.k8s.v6d.io/vineyardd-namespace: vineyard-system
        scheduling.k8s.v6d.io/vineyardd: vineyardd-sample
        scheduling.k8s.v6d.io/job: v6d-workflow-demo-job1
    spec:
      # use the vineyard scheduler to deploy the pod on the vineyard cluster.
      schedulerName: vineyard-scheduler
      containers:
      - name: vineyard-python
        imagePullPolicy: IfNotPresent
        image: python:3.10
        command:
        - /bin/bash
        - -c
        - pip3 install vineyard && sleep infinity
        volumeMounts:
        - mountPath: /var/run
          name: vineyard-sock
      volumes:
      - name: vineyard-sock
        hostPath:
          path: /var/run/vineyard-kubernetes/vineyard-system/vineyardd-sample
EOF

Expected output

pod/vineyard-python-client created

Wait for the vineyard python client pod ready.

$ kubectl get pod -l app=vineyard-python-client -n vineyard-system

Expected output

NAME                                      READY   STATUS    RESTARTS   AGE
vineyard-python-client-6fd84bc897-27glp   1/1     Running   0          93s
vineyard-python-client-6fd84bc897-tlb22   1/1     Running   0          93s

Use the kubectl exec command to enter the first vineyard python client pod.

$ kubectl exec -it $(kubectl get pod -l app=vineyard-python-client -n vineyard-system -oname | head -n 1 | awk -F '/' '{print $2}') -n vineyard-system /bin/bash

After entering the shell, you can connect to the vineyard cluster,

In [1]: import numpy as np
In [2]: import vineyard

In [3]: client = vineyard.connect('/var/run/vineyard.sock')

In [4]: objid = client.put(np.zeros(8))

In [5]: # persist the object to make it visible to form the global object
In [6]: client.persist(objid)

In [7]: objid
Out[7]: o001027d7c86a49f0

In [8]: # get meta info
In [9]: meta = client.get_meta(objid)
In [10]: meta
Out[10]:
{
    "buffer_": {
        "id": "o801027d7c85c472e",
        "instance_id": 1,
        "length": 0,
        "nbytes": 0,
        "transient": true,
        "typename": "vineyard::Blob"
    },
    "global": false,
    "id": "o001027d7c86a49f0",
    "instance_id": 1,
    "nbytes": 64,
    "order_": "\"C\"",
    "partition_index_": "[]",
    "shape_": "[8]",
    "signature": 4547407361228035,
    "transient": false,
    "typename": "vineyard::Tensor<double>",
    "value_type_": "float64",
    "value_type_meta_": "<f8"
}

Open another terminal and enter the second vineyard python client pod.

$ kubectl exec -it $(kubectl get pod -l app=vineyard-python-client -n vineyard-system -oname | tail -n 1 | awk -F '/' '{print $2}') -n vineyard-system /bin/bash

Expected output

kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.

Also, you can connect to the vineyard cluster by RPC and get the metadata of above object as follows.

In [1]: import vineyard

In [2]: rpc_client = vineyard.connect('vineyardd-sample-rpc.vineyard-system',9600)

In [3]: # use the object id created by another vineyard instance here
In [4]: meta = rpc_client.get_meta(vineyard._C.ObjectID('o001027d7c86a49f0'))
In [5]: meta
Out[5]:
{
    "buffer_": {
        "id": "o801027d7c85c472e",
        "instance_id": 1,
        "length": 0,
        "nbytes": 0,
        "transient": true,
        "typename": "vineyard::Blob"
    },
    "global": false,
    "id": "o001027d7c86a49f0",
    "instance_id": 1,
    "nbytes": 64,
    "order_": "\"C\"",
    "partition_index_": "[]",
    "shape_": "[8]",
    "signature": 4547407361228035,
    "transient": false,
    "typename": "vineyard::Tensor<double>",
    "value_type_": "float64",
    "value_type_meta_": "<f8"
}

For more examples, please refer the vineyard data accessing.

Step 4: Cleanup

  • Destroy the vineyard operator via helm:

$ helm uninstall vineyard-operator -n vineyard-system

Expected output

release "vineyard-operator" uninstalled
  • Delete the namespace:

$ kubectl delete namespace vineyard-system

Expected output

namespace "vineyard-system" deleted