why apache Kafka documentation not gives the ability to use Kafka over NFS as Netapp

77 Views Asked by At

We read the latest apache documentation as https://kafka.apache.org/35/documentation.html

And what is interesting is that , documentation not mentioned the option to use storage over NFS As Netapp or ONTAP

A little background - we are supporting On-Prem Kafka cluster with 34 machines and Kafka used internal SAS disks

kafka brokers setup (machine spec & disks):

34 kafka brokers, Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz, 16 cores.

each broker has sdb device mounted to /var/kafka, in size 44.6T.

the sdb device is combined of 16 SAS disks ~1TB in RAID-10. which means 8 disks are used as parity.

now per customer request , customer want to check the option move On Prem storage as Netapp storage instead of physical disks

honestly , I am little confused because we saw some documentation that not like the idea to use Kafka cluster over NFS as the following

1)https://access.redhat.com/documentation/ru-ru/red_hat_amq_streams/2.1/html/configuring_amq_streams_on_openshift/assembly-deployment-configuration-str#considerations-for-data-storage-str

2)https://strimzi.io/docs/operators/latest/configuring.html#considerations-for-data-storage-str

3)https://docs.confluent.io/platform/current/kafka/deployment.html#disks

4)https://sbg.technology/2018/07/10/kafka-nfs/

5)Kafka doesn't work with external NFS Volume

and other documentation that support the idea to use Netapp or Ontap as the following

https://github.com/NetApp/trident/issues/808

or

https://docs.netapp.com/us-en/netapp-solutions/data-analytics/kafka-nfs-why-netapp-nfs-for-kafka-workloads.html#architectural-setup

above link say that:

With ONTAP 9.12.1 and higher, NFSv4.1, and Linux changes that are in RHEL 8.7 or 9.1 and higher, there are fixes to support running Kafka over NFS. There are some details about this at https://www.netapp.com/blog/simplify-apache-kafka-confluent/. In order to enable this functionality in ONTAP, there is a new volume setting in 9.12.1 called "-is-preserve-unlink-enabled", which must be set to "true". The ask is for Trident to provide a way for this setting to be enabled so that PVCs for Kafka can be created using the ontap-nas or ontap-nas-economy drivers.

So can we say that only ONTAP 9.12.1 and higher, and Linux RHEL 8.7 or 9.1 and higher, can support Kafka cluster ?

0

There are 0 best solutions below