YARP forwarding to gRPC endpoint fails with 'Incomplete message' error

Question

I am trying to expose one of my gRPC services (protobuf-net) more easily by using YARP to forward HTTP 1.1 calls to HTTP2.0 gRPC endpoints.

My main issue is that I am getting an error like this whenever I attempt to forward such a call.

2023-10-09 21:58:57 info: Grpc.AspNetCore.Server.ServerCallHandler[14]
2023-10-09 21:58:57       Error reading message.
2023-10-09 21:58:57       Grpc.Core.RpcException: Status(StatusCode="Internal", Detail="Incomplete message.")
2023-10-09 21:58:57          at Grpc.AspNetCore.Server.Internal.PipeExtensions.ReadSingleMessageAsync[T](PipeReader input, HttpContextServerCallContext serverCallContext, Func`2 deserializer)
2023-10-09 21:58:57 info: Grpc.AspNetCore.Server.ServerCallHandler[7]
2023-10-09 21:58:57       Error status code 'Internal' with detail 'Incomplete message.' raised.

This is what I can see on the logs of my .NET 7 WebAPI that implements the gPRC service.

The BFF service (Duende.BFF), however is able to forward my call properly, or at least it looks like.

2023-10-09 21:58:57 [18:58:57 Information] Yarp.ReverseProxy.Forwarder.HttpForwarder
2023-10-09 21:58:57 Proxying to http://host.docker.internal:64868/<redactedNamespace>.LocationGrpcService/GetLocations HTTP/2 RequestVersionExact no-streaming

This gRPC service does work whenever I do grpc-style calls to it, either from C# or from Postman, so the implementation of the service shouldn't be the issue.

  using var channel = GrpcChannel.ForAddress("http://host.docker.internal:64868");
  var locationService = channel.CreateGrpcService<ILocationGrpcService>();
  var locations = await locationService.GetLocationsAsync();

But, if I am doing my request from my Angular HttpClient then it will 'fail' with the upper error, while I am getting these response headers.

I have attempted to investigate with an interceptor what's going on by doing this:

   services.AddCodeFirstGrpc(config =>
   {
       config.ResponseCompressionLevel = System.IO.Compression.CompressionLevel.Optimal;
       config.EnableDetailedErrors = true;
       config.Interceptors.Add<ServerLoggerInterceptor>();
   });

The problem is, I can catch breakpoints only for my valid gRPC calls (postman, gRPC to gRPC, gRPC health-checks, etc) but not for my YARP call.

This is my interceptor code

 public class ServerLoggerInterceptor : Interceptor
 {
     private readonly ILogger<ServerLoggerInterceptor> _logger;

     public ServerLoggerInterceptor(ILogger<ServerLoggerInterceptor> logger)
     {
         _logger = logger;
     }

     public override async Task<TResponse> UnaryServerHandler<TRequest, TResponse>(
         TRequest request,
         ServerCallContext context,
         UnaryServerMethod<TRequest, TResponse> continuation)
     {
         //LogCall<TRequest, TResponse>(MethodType.Unary, context);

         try
         {
             return await continuation(request, context);
         }
         catch (Exception ex)
         {
             // Note: The gRPC framework also logs exceptions thrown by handlers to .NET Core logging.
             _logger.LogError(ex, $"Error thrown by {context.Method}.");

             throw;
         }
     }

Looking at the documentation I can see that my code is this one:

 **INTERNAL 13  Internal errors. This means that some invariants expected by the underlying system have been broken. This error code is reserved for serious errors.**

So since my gRPC implementation works, and only my YARP calls are failing, I can safely assume my YARP is somehow configured wrong, let me share it here.

  "ReverseProxy": {
    "Routes": {
      "video-provider": {
        "Transforms": [
          { "PathRemovePrefix": "/bff" }
        ],
        "ClusterId": "video-provider-rest",
        "Match": {
          "Path": "/bff/video-provider/{**catch-all}"
        },
        "Metadata": {
          "Duende.Bff.Yarp.TokenType": "User",
          "Duende.Bff.Yarp.AntiforgeryCheck": "true"
        }
      },
      "video-provider-grpc": {
        "Transforms": [
          { "PathRemovePrefix": "/bff/grpc/video-provider" },
          {
            "RequestHeader": "content-type",
            "Set": "application/grpc"
          }
        ],
      
        "ClusterId": "video-provider-grpc",
        "Match": {
          "Path": "/bff/grpc/video-provider/{*any}"
        },
        "Metadata": {
          "Duende.Bff.Yarp.TokenType": "User",
          "Duende.Bff.Yarp.AntiforgeryCheck": "true"
        }
      }
    },
    "Clusters": {
      // This is used for proxying data to VP thru HTTP1.1
      "video-provider-rest": {
        "Destinations": {
          "video-provider/destination1": {
            "Address": "http://host.docker.internal:64869/"
          }
        }
      },
      // This can be used for proxying HTTP1.1 to HTTP2 calls
      "video-provider-grpc": {
        "HttpRequest": {
          "Version": "2",
          "VersionPolicy": "RequestVersionExact"
        },
        "Destinations": {
          "video-provider/destination1": {
            "Address": "http://host.docker.internal:64868"
          }
        }
      }
    }
  },

I am setting the destination to use the HTTP 2.0 protocol, I do set up any additional stuff like explicitly setting the request header's content-type to be of type application/grpc   I am not sure what I am doing wrong.

I have checked out Duende's BFF Yarp implementation and couldn't find anything of use. I have also reviewed MS's documentation in regard to troubleshooting gRPC calls but it feels like I'm dancing around some sort of misconfiguration of YARP from my end.

Edit: I'm running on .NET Core 7 WebAPI Duende.BFF

I'm running on latest stable packages.