how to make Azure Databricks RunsApi run faster

58 Views Asked by At

In a local python file, I am using the databricks RunsApi to call a notebook in databricks and then return the response as a Json. When I run the notebook in databricks directly, it only takes about 3 seconds to run, but using the api is taking 25 seconds. Is there a way I can make this runtime shorter? This is the code I am using. Thanks

import os
import json
import time
from databricks_cli.sdk.api_client import ApiClient
from databricks_cli.runs.api import RunsApi

api_client = ApiClient(host=os.getenv('DATABRICKS_HOST'), token=os.getenv('DATABRICKS_TOKEN'))

runJson = {
            "name ": "api_test",
            "max_concurrent_runs" : 1,
            "tasks" : [
                {
                "task_key" : "api_test",
                "description" : "test",
                "notebook_task" :
                    {           
                    "notebook_path" : "path"
                    },
                "existing_cluster_id" : "id",
                "timeout_seconds" : 3600,
                "max_retries" : 3,
                "retry_on_timeout" : True
                }
            ]
            }

runs_api = RunsApi(api_client)
run_id = runs_api.submit_run(runJson)
runOutput = runs_api.get_run_output(run_id['run_id'])
0

There are 0 best solutions below