How do you install mayastor for openebs with microk8s to use as PV/SC?

Question

I have a 3 node microk8s cluster running on virtualbox Ubuntu vms. And I am trying to get mayastor for openebs working to use with PVCs. I have followed the steps in this guide:

1. https://mayastor.gitbook.io/introduction/quickstart/preparing-the-cluster
2. https://mayastor.gitbook.io/introduction/quickstart/deploy-mayastor
3. https://mayastor.gitbook.io/introduction/quickstart/configure-mayastor

An example of my MayastorPool from step 3 looks like this:

apiVersion: "openebs.io/v1alpha1"
kind: MayastorPool
metadata:
  name: pool-on-node1-n2
  namespace: mayastor
spec:
  node: node1
  disks: [ "/dev/nvme0n2" ]

And my StorageClass looks like this:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: mayastor-3
provisioner: io.openebs.csi-mayastor
parameters:
  repl: '3'
  protocol: 'nvmf'
  ioTimeout: '60'
  local: 'true'
volumeBindingMode: WaitForFirstConsumer

All the checks seem fine according to guide, but when I try to creating a PVC and using it according to this https://mayastor.gitbook.io/introduction/quickstart/deploy-a-test-application the the test application fio pod doesn't come up. When I look at it with describe I see the following: $ kubectl describe pods fio -n mayastor

Name:         fio
Namespace:    mayastor
Priority:     0
Node:         node2/192.168.40.12
Start Time:   Wed, 02 Jun 2021 22:56:03 +0000
Labels:       <none>
Annotations:  <none>
Status:       Pending
IP:           
IPs:          <none>
Containers:
  fio:
    Container ID:  
    Image:         nixery.dev/shell/fio
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Args:
      sleep
      1000000
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-l6cdf (ro)
      /volume from ms-volume (rw)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  ms-volume:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  ms-volume-claim
    ReadOnly:   false
  kube-api-access-l6cdf:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              openebs.io/engine=mayastor
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason                  Age                  From                     Message
  ----     ------                  ----                 ----                     -------
  Normal   Scheduled               44m                  default-scheduler        Successfully assigned mayastor/fio to node2
  Normal   SuccessfulAttachVolume  44m                  attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-ec6ce101-fb3e-4a5a-8d61-1d228f8f8199"
  Warning  FailedMount             24m (x4 over 40m)    kubelet                  Unable to attach or mount volumes: unmounted volumes=[ms-volume], unattached volumes=[kube-api-access-l6cdf ms-volume]: timed out waiting for the condition
  Warning  FailedMount             13m (x23 over 44m)   kubelet                  MountVolume.SetUp failed for volume "pvc-ec6ce101-fb3e-4a5a-8d61-1d228f8f8199" : rpc error: code = Internal desc = Failed to find parent dir for mountpoint /var/snap/microk8s/common/var/lib/kubelet/pods/b1166af6-1ade-4a3a-9b1d-653151418695/volumes/kubernetes.io~csi/pvc-ec6ce101-fb3e-4a5a-8d61-1d228f8f8199/mount, volume ec6ce101-fb3e-4a5a-8d61-1d228f8f8199
  Warning  FailedMount             4m3s (x13 over 42m)  kubelet                  Unable to attach or mount volumes: unmounted volumes=[ms-volume], unattached volumes=[ms-volume kube-api-access-l6cdf]: timed out waiting for the condition

Any ideas where to look or what to do to get mayastor working with microk8s? Happy to post more information.

Answer 1

Thanks to Kiran Mova's comments and Niladri from the openebs slack channel:

Replace the step: https://mayastor.gitbook.io/introduction/quickstart/deploy-mayastor#csi-node-plugin

kubectl apply -f https://raw.githubusercontent.com/openebs/Mayastor/master/deploy/csi-daemonset.yaml

with

curl -fSs https://raw.githubusercontent.com/openebs/Mayastor/master/deploy/csi-daemonset.yaml | sed "s|/var/lib/kubelet|/var/snap/microk8s/common/var/lib/kubelet|g" - | kubectl apply -f -

So replace the path with the microk8s installation specific path. Even though there is a symlink things don't seem to work out right without this change.