Failed to Create volume: This IO profile requires a minimum replication factor of 2 nodes

Question

volume creation failed while installing the statefulset.

Events:

9s          Warning   ProvisioningFailed   persistentvolumeclaim/kafka-vol-kafka-0                     Failed to provision volume with StorageClass "hello-sc": rpc error: code = Internal desc = Failed to create volume: This IO profile requires a minimum replication factor of 2 nodes

PVC

kubectl get pvc -n hello
NAME                STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
hello-vol-hello-0     Pending                                      hell-sc     44m

Storage class definition:

Name:            hello-sc
IsDefaultClass:  Yes
Annotations:     kubectl.kubernetes.io/last-applied-configuration={"allowVolumeExpansion":true,"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"},"name":"hello-sc"},"parameters":{"cow_ondemand":"true","disable_io_profile_protection":"true","fs":"ext4","io_profile":"db_remote","priority_io":"high","repl":"1"},"provisioner":"kubernetes.io/portworx-volume","reclaimPolicy":"Delete","volumeBindingMode":"Immediate"}
,storageclass.kubernetes.io/is-default-class=true
Provisioner:           kubernetes.io/portworx-volume
Parameters:            cow_ondemand=true,disable_io_profile_protection=true,fs=ext4,io_profile=db_remote,priority_io=high,repl=1
AllowVolumeExpansion:  True
MountOptions:          <none>
ReclaimPolicy:         Delete
VolumeBindingMode:     Immediate
Events:                <none>

Installing the statefulset with PVC. The volume should be created, even with replication factor 1.

Answer 1

The error Failed to create volume: This IO profile requires a minimum replication factor of 2 nodes suggesting that to create volume IO profile requires replication factor of at least 2 nodes.

As mentioned in the document on Best practises for Apache kafka running on kubernetes with portworx :

Kafka, being designed as a distributed and horizontally scalable streaming system, relies on the replication factor of topic-partitions to achieve higher availability of the data if a broker has to fail. As such for production readiness, you should never run Kafka with the default replication factor of 1 even if you are running the application on Kubernetes with Portworx. The general recommendation is to use either 2 or 3 for the replication factor which allows either 1 or 2 broker failures and still have access to the data.

Portworx enables high availability of the application data through replication at the storage level. Portworx allows you to set the replication between 1 and 3 where 3 offers the highest level of data protection and availability.

You can also refer to this github link and this document for more information.

Answer 2

explicitly specifying the io_profile is causing the issue.

Problematic definition:

allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"allowVolumeExpansion":true,"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{"storagecla
ss.kubernetes.io/is-default-class":"true"},"name":"hello-sc"},"parameters":{"cow_ondemand":"true","disable_io_profile_protecti
on":"true","fs":"ext4","io_profile":"db_remote","priority_io":"high","repl":"1"},"provisioner":"kubernetes.io/portworx-volume","
reclaimPolicy":"Delete","volumeBindingMode":"Immediate"}
    storageclass.kubernetes.io/is-default-class: "true"
  creationTimestamp: "2022-12-07T08:15:45Z"
  name: hello-sc
  resourceVersion: "256593028"
  uid: 557e55db-3440-4c2b-a5ee-842d09c1e7d2
parameters:
  cow_ondemand: "true"
  disable_io_profile_protection: "true"
  fs: ext4
  io_profile: db_remote
  priority_io: high
  repl: "1"
provisioner: kubernetes.io/portworx-volume
reclaimPolicy: Delete
volumeBindingMode: Immediate

Working defintion:

allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"allowVolumeExpansion":true,"apiVersion":"storage.k8s.io/v1beta1","kind":"StorageClass","metadata":{"annotations":{"stora
geclass.kubernetes.io/is-default-class":"true"},"name":"hello-sc"},"parameters":{"fs":"ext4","repl":"1"},"provisioner":"kubern
etes.io/portworx-volume"}
    storageclass.kubernetes.io/is-default-class: "true"
  creationTimestamp: "2022-04-08T11:32:23Z"
  name: hello-sc
  resourceVersion: "36880"
  uid: db05d65c-33ae-4feb-986d-b7e92b3c077a
parameters:
  fs: ext4
  repl: "1"
provisioner: kubernetes.io/portworx-volume
reclaimPolicy: Delete
volumeBindingMode: Immediate