I'm trying to put some records into a Kafka cluster that is kerberized, from a Spark Process which is running in another Kerberized cluster (different domains). I think the issue is with trying to read a dataset in spark record by record and put them in kafka (as a result of which an active connection needs to be maintained with both clusters). Please help.
I have tried setting the VM property -Djava.security.auth.login.config with a jaas file with the principal and keytab corresponding to the kafka cluster during run time using System.setProperty(), but with no luck.
Before submitting the spark process using spark2-submit, I need to kinit using the keytab corresponding to the Hadoop Cluster in order for the spark process to run.