azure, databricks, spark - can't connect with SAS token

Question

I am trying to get access to my azure storage container json file to read json files (data). I have config my spark like this:

spark.conf.set(f"fs.azure.account.auth.type.{STORAGE_ACCOUNT_NAME}.dfs.core.windows.net", "SAS")
        spark.conf.set(f"fs.azure.sas.token.provider.type.{STORAGE_ACCOUNT_NAME}.dfs.core.windows.net", "org.apache.hadoop.fs.azurebfs.sas.FixedSASTokenProvider")
        spark.conf.set(f"fs.azure.sas.fixed.token.{STORAGE_ACCOUNT_NAME}.dfs.core.windows.net", SAS_TOKEN)
        logging.info('spark was configured')

When I try to connect I get the error: "Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature." (403).

data = get_data_from_file(BLOB_PATH)

/* The SAS token is correct, the file path is correct.

** Firewall is enabled in the networking setting in the container (Disabling it doesn't help).

*** If I try to read from a different storage it works, but why, I didn't do something in the settings.

Answer 1

"Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature." (403)

There are multiple reasons for this issue:

• If you are using blob storage, then you have to use wasbs and not abfss. I have tried using below code with my SAS token and got the same error with my blob storage.

** Firewall is enabled in the networking setting in the container (Disabling it doesn't help).**

To access files from azure blob storage where the firewall settings are only from selected networks,

• You need to configure VNet for the Databricks workspace.
• Now add the same virtual network to your storage account as well.
• I have also selected service endpoints and subnet delegation as following:
• Now run the code to get the data from ADLS File.

Also check this SO answer for reference