gRPC Health Checks on AWS ALB

Question

No matter which health check path I provide, even if it's completely arbitrary text, the health check will only pass as long if the matcher code is set to 0.

I have one service that's just using the default path given by the AWS example - /AWS.ALB/healthcheck. It's passing with a 0.

Another service has implemented a specific health check path that returns 1 on success, but the check continues to fail unless the matcher code is set to 0. It's worth noting that this service has implemented server reflection.

Am I missing a piece here? I more or less followed the setup in the official example.

Answer 1

Per the official GRPC documentation on response codes, a response code of 0 indicates

Not an error; returned on success.

This would be the correct return value for a health check that has passed.

I'm not sure where the blog post you linked to got the idea that a response code of "12" (UNIMPLEMENTED) is indicative of a health check passing, but this certainly shouldn't be the case.

You should definitely be aiming to have your GRPC services return a code of 0 for success as this is the standard mechanism for indicating a healthy service.

I have embedded an image taken from the linked website in the event it is not available for future viewers.

Answer 2

AWS Target Groups with the protocol version set to gRPC will default to the following:

• HealthCheckPath: /AWS.ALB/healthcheck
• Matcher (expected status code): 12 (Unimplemented)

This is directly from Amazon's documentation: https://docs.aws.amazon.com/elasticloadbalancing/latest/application/target-group-health-checks.html.

You can change these values, but your gRPC services are not going to have the precise RPC that exactly matches the default /AWS.ALB/healthcheck (package = AWS, service = ALB, unary rpc = healthcheck) and therefore Amazon's default expected status code of 12 (Unimplemented) makes sense.

That said, if you choose to implement your own realistic health check as an RPC on your service, it is recommended to use a more idiomatic response code to indicate success (0 OK). For example, a /com.mypackage/ServiceA.healthcheck should return a 0 OK and have its target group's health check matcher set to 0 OK as well.