GCP Vertex AI Endpoint returning empty prediction array

Question

KFP pipeline job executes successfully, but upon hitting the endpoint, am getting an empty predictions array ([]). I suspect the issue is in the model upload, where the model is not registered correctly somehow. Any tips are appreciated.

Code to upload model deploy task:

    #Import a model programmatically
    model_upload = aiplatform.Model.upload(
        display_name = DISPLAY_NAME, 
        serving_container_image_uri = serving_container_image_uri,
        serving_container_health_route="/health_check",
        serving_container_predict_route="/predict",
        serving_container_ports=[8080],
        serving_container_environment_variables={
            "MODEL_NAME": MODEL_NAME,
        },       
    )

Code to get predictions:

response = endpoint.predict({"user_id": 150})
# response = endpoint.predict({"instances":{"user_id": 150}})
response

Response:

Prediction(predictions=[], deployed_model_id='4656867150235959296', explanations=None)

Answer 1

TLDR:

Check your handler output is returning a correctly formatted string without extra slashes (\) in the response which meets the requirements.

Or use raw predict instead of predict call. (guide)

---

Debugging steps:

I also had this issue where I was using the Vertex AI custom serving container approach and calling endpoint.predict(instances=[{...}]).predictions was only returning an empty list: [].

Even though, when using raw predict from the Python SDK or via REST endpoint, it returned a valid response for the data JSON key:

Python SDK response:

"{\"predictions\": \"[{\\\"key1\\\": \\\"string1\\\", \\\"key2\\\": [\\\"string2\\\"], \\\"key3\\\": 0.0}]\"}"

REST API response:

{"predictions": "[{\"key1\": \"string1\", \"key2\": [\"string2\"], \"key3\": 0.0}]"}

However, in both cases you can see that there were extra slashes (\) in the responses. This means that it was a formatting issue.

---

Solution:

It turned out that in the post-processing step of my handler, I had done the following:

prediction_result = {
            "key1": value1,
            "key2": value2,
            "key3": value3,
        }
array = np.array([prediction_result])

return json.dumps({"predictions": array.tolist()})

Changing it to the following fixed the problem for me:

prediction_result = {
            "key1": value1,
            "key2": value2,
            "key3": value3,
        }

return json.dumps({"predictions": [prediction_result]})

And then calling endpoint.predict(instances=[{...}]).predictions returned the following list for me:

[{'key1': 'string1',
  'key2': ['string2'],
  'key3': 0.0}]

After the fix, the responses from raw predict no longer contained the extra slashes (\).

Answer 2

Had the similar issue when postprocess returned dictionary with str value

def postprocess(self, prediction_results: str) -> Dict:
   ...
   return {"predictions": prediction_results}

Change return {"predictions": prediction_results} to return {"predictions": [prediction_results]} to start receive non-empty predictions