Timeout exception occurs while connecting confluent cloud from Azure databricks notebook

Question

The below code throws below exception. Topic present in the kafka cluster. No network connectivity relate issue.

Exception:

Job aborted due to stage failure: Topic spark_poc_topic not present in metadata after 60000 ms. Caused by: TimeoutException: Topic spark_poc_topic not present in metadata after 60000 ms.

 df2.selectExpr("key", "value")
    .write
    .format("kafka")
    .option("kafka.bootstrap.servers", "xxxx.azure.confluent.cloud:9092")
    .option("kafka.security.protocol", "SASL_PLAINTEXT")
    .option("kafka.sasl.jaas.config", "kafkashaded.org.apache.kafka.common.security.plain.PlainLoginModule required username='{}' password='{}';".format("INxxxxxxxI", "n/LeO+aEJbxxxxx"))
    .option("kafka.ssl.endpoint.identification.algorithm", "https")
    .option("kafka.sasl.mechanism", "PLAIN")
    .option("topic", "topic123")
    .save()

Problem: .option("kafka.sasl.jaas.config", "kafkashaded.org.apache.kafka.common.security.plain.PlainLoginModule required username='{}' password='{}';".format("INxxxxxxxI", "n/LeO+aEJbxxxxx"))

Solution: .option("kafka.sasl.jaas.config", "kafkashaded.org.apache.kafka.common.security.plain.PlainLoginModule required username="" + confluentApiKey + ""password="" + confluentSecret + "";")