Get Command number in databricks

Question

How can i get the current command number dynamically from a cell in Azure databricks notebook? Is there any dbutils command to retrieve it?

For below cell, it should return 1.

Answer 1

There is no direct built-in function in dbutils or Spark to retrieve the current command number dynamically from a cell in an Azure Databricks notebook.

I have tried the following approach:

spark._jsparkSession.sparkContext().getLocalProperty("spark.databricks.notebook.id")
result = spark.sql("SELECT * FROM default.my_table")
print(f"Current command number: {command_number}")

Result:

Current command number: 2585995787002146

spark._jsparkSession.sparkContext().getLocalProperty("spark.databricks.notebook.id"): The command above retrieves the number of the current cell in the Databricks notebook.

• It uses the SparkSession (spark) to access the underlying Java SparkSession, then gets the local property "spark.databricks.notebook.id" which holds the command number.

• The code uses the Spark context (spark._jsparkSession.sparkContext()) to access a property that represents the current command number associated with the notebook.

• This property, spark.databricks.notebook.id, uniquely identifies each command execution within the notebook. The getLocalProperty method is then used to retrieve the value of this property, which corresponds to the command number.