We want to create a pipeline for each customers that can be new topic for kafka.
But its unclear most of the places especially on MSK how many topics we can create on lets say m7g.xlarge instance.
Would be helpful to know. how many topics can be created and if topics count exceed 10K do we start to see any lags.
Do these high number of topics affect the kafka connectors ingestion and throughput.
Thanks
MSK follows the best practices of Apache Kafka in regards to the total number of partitions (including followers) that can be defined on a single Kafka cluster, regardless of the cluster size. There is a also a recommended partition count per broker.
MSK defines recommended max partition count per cluster at 200,000. A count per broker depends on a broker type, but an absolute max is 4,000 partitions per broker.
These restrictions could be changed in the future, so please follow the MSK documentation.