I have issues using the newest version of databricks-connect (13.3.0). I would like to access the sparkContext and tried it as it worked for databricks-connect<13.0:
from databricks.connect import DatabricksSession
spark = DatabricksSession.builder.getOrCreate()
spark.sparkContext
However, now I get the error:
pyspark.errors.exceptions.base.PySparkNotImplementedError: [NOT_IMPLEMENTED] sparkContext() is not implemented.
Can someone help?
Databricks connect in versions 13+ is based on Spark Connect that doesn't support RDD APIs together with related objects like SparkContext. It's really documented as known limitation.