On AWS ALB a gRPC stream CloseSend() without a prior Send() is not received by server

Question

I am using the official example here https://grpc.io/docs/languages/go/basics/#bidirectional-streaming-rpc-1

I only skip calling Send() on client to emulate a possible case in https://github.com/grpc/grpc-go/blob/master/examples/route_guide/client/client.go#L140

@@ -137,8 +137,8 @@ func runRouteChat(client pb.RouteGuideClient) {
                        log.Printf("Got message %s at point(%d, %d)", in.Message, in.Location.Latitude, in.Location.Longitude)
                }
        }()
        for _, note := range notes {
+               continue
                if err := stream.Send(note); err != nil {
                        log.Fatalf("client.RouteChat: stream.Send(%v) failed: %v", note, err)
                }

Running the above client/server via an AWS ALB, in client:

# ./client -tls -ca_file /etc/ssl/certs/Amazon_Root_CA_1.pem -server_host_override some-alb-address.app -addr some-alb-address.app:443
2024/02/02 09:06:38 http2: Framer 0xc0001f4460: read SETTINGS len=18, settings: MAX_CONCURRENT_STREAMS=128, INITIAL_WINDOW_SIZE=65536, MAX_FRAME_SIZE=16777215
2024/02/02 09:06:38 http2: Framer 0xc0001f4460: read WINDOW_UPDATE len=4 (conn) incr=2147418112
2024/02/02 09:06:38 http2: Framer 0xc0001f4460: wrote SETTINGS len=0
2024/02/02 09:06:38 http2: Framer 0xc0001f4460: wrote SETTINGS flags=ACK len=0
2024/02/02 09:06:38 http2: Framer 0xc0001f4460: wrote HEADERS flags=END_HEADERS stream=1 len=114
2024/02/02 09:06:38 http2: Framer 0xc0001f4460: wrote DATA flags=END_STREAM stream=1 len=0 data=""
2024/02/02 09:06:38 http2: Framer 0xc0001f4460: read SETTINGS flags=ACK len=0
2024/02/02 09:07:38 http2: Framer 0xc0001f4460: read HEADERS flags=END_STREAM|END_HEADERS stream=1 len=87
2024/02/02 09:07:38 http2: decoded hpack field header field ":status" = "504"
2024/02/02 09:07:38 http2: decoded hpack field header field "server" = "awselb/2.0"
2024/02/02 09:07:38 http2: decoded hpack field header field "date" = "Fri, 02 Feb 2024 09:07:38 GMT"
2024/02/02 09:07:38 http2: decoded hpack field header field "content-type" = "application/grpc"
2024/02/02 09:07:38 http2: decoded hpack field header field "content-length" = "0"
2024/02/02 09:07:38 http2: decoded hpack field header field "grpc-status" = "14"
2024/02/02 09:07:38 http2: decoded hpack field header field "grpc-message" = "unavailable"
2024/02/02 09:07:38 http2: Framer 0xc0001f4460: wrote RST_STREAM stream=1 len=4 ErrCode=PROTOCOL_ERROR
2024/02/02 09:07:38 client.RouteChat failed: rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 504 (Gateway Timeout)

In server

2024/02/02 09:06:38 http2: Framer 0xc0001bb180: wrote SETTINGS len=6, settings: MAX_FRAME_SIZE=16384
2024/02/02 09:06:38 http2: Framer 0xc0001bb180: read SETTINGS len=18, settings: HEADER_TABLE_SIZE=0, ENABLE_PUSH=0, INITIAL_WINDOW_SIZE=2147483647
2024/02/02 09:06:38 http2: Framer 0xc0001bb180: read WINDOW_UPDATE len=4 (conn) incr=2147418112
2024/02/02 09:06:38 http2: Framer 0xc0001bb180: read HEADERS flags=END_STREAM|END_HEADERS stream=1 len=240
2024/02/02 09:06:38 http2: Framer 0xc0001bb180: wrote SETTINGS flags=ACK len=0
2024/02/02 09:06:38 http2: decoded hpack field header field ":method" = "POST"
2024/02/02 09:06:38 http2: decoded hpack field header field ":scheme" = "http"
2024/02/02 09:06:38 http2: decoded hpack field header field ":path" = "/routeguide.RouteGuide/RouteChat"
2024/02/02 09:06:38 http2: decoded hpack field header field "x-forwarded-for" = "10.12.210.230"
2024/02/02 09:06:38 http2: decoded hpack field header field "x-forwarded-proto" = "https"
2024/02/02 09:06:38 http2: decoded hpack field header field "x-forwarded-port" = "443"
2024/02/02 09:06:38 http2: decoded hpack field header field "host" = "some-alb-address.app"
2024/02/02 09:06:38 http2: decoded hpack field header field "x-amzn-trace-id" = "Root=1-xxx-xxx"
2024/02/02 09:06:38 http2: decoded hpack field header field "te" = "trailers"
2024/02/02 09:06:38 http2: decoded hpack field header field "content-type" = "application/grpc"
2024/02/02 09:06:38 http2: decoded hpack field header field "user-agent" = "grpc-go/1.59.0"
2024/02/02 09:06:38 http2: decoded hpack field header field "grpc-timeout" = "199990m"
2024/02/02 09:06:38 http2: Framer 0xc0001bb180: read SETTINGS flags=ACK len=0
2024/02/02 09:06:42 http2: Framer 0xc0001bb340: wrote SETTINGS len=6, settings: MAX_FRAME_SIZE=16384
2024/02/02 09:06:42 http2: Framer 0xc0001bb340: read SETTINGS len=18, settings: HEADER_TABLE_SIZE=0, ENABLE_PUSH=0, INITIAL_WINDOW_SIZE=2147483647

I simply cannot see END_STREAM for DATA on server side.

If I call Send() at least once than this line appears

2024/02/02 09:14:53 http2: Framer 0xc00010e620: read DATA flags=END_STREAM stream=1 len=0 data=""

When not using an AWS ALB an EOF (END_STREAM on DATA) is received on server as above.

AWS support insist that this is a problem with the server code. I would be surprised if there was. Maybe there is a logic error like Send() is expected to be always called for some reason? Or, is AWS ALB simply doing something that breaks the above case?

An END_STREAM for DATA on server side.