Multiple sparkOperators on same EKS cluster?

20 Views Asked by At

I'm new to sparkOperator set up on EKS and EMR on EKS. There is an sparkOperator already set up on 1 EKS cluster, can I set up another sparkOperator on the same EKS using sparkOperator on EMR on EKS ? Some background: Current pipelines are deployed with Argo on 1 EKS cluster, spark workload is created by Argo using SparkApplication kind, which is submitted to the sparkOperator installed. Now, I would like to compare the performance and cost using EMR on EKS, and thinking the easiest way is to set up EMR on the same EKS and install another sparkOperator on EMR on EKS with potentially a different name/namespace, then SparkApplication can submit to the it by specifying the sparkOperator name (not sure if that can be done yet). I assume SparkApplication submit to either sparkOperator can utilize same config (except the image), appreciate your advice.

I have Googled and checked openAI, can not find explicit answers to that.

0

There are 0 best solutions below