Grpc.Core.RpcException: Status(StatusCode="PermissionDenied", Detail="Bad gRPC response. HTTP status code: 403")

Question

When ever I deployed my grpc on azure app service its fine but I want to deploy it on kubernetes I got above error .Is there anything special for kubernetes deployment?

gotting permissiondenied error

program.cs

using ChannelMicroservice.Database;
using ChannelMicroservice.Grpc.Service;
using ChannelMicroservice.Repository;
using ChannelMicroservice.Repository.Interface;
using Microsoft.AspNetCore.Server.Kestrel.Core;
using Microsoft.OpenApi.Models;

var builder = WebApplication.CreateBuilder(args);


// Add services to the container.
builder.Services.AddGrpc(options=>options.EnableDetailedErrors=true);
builder.Services.AddControllers();
builder.Services.AddHttpClient(); // Register IHttpClientFactory
// Learn more about configuring Swagger/OpenAPI at https://aka.ms/aspnetcore/swashbuckle
builder.Services.AddEndpointsApiExplorer();


ConfigureDI(builder.Services);
builder.Services.AddCors(options =>
{
    options.AddPolicy("CorsPolicy", builder => builder.AllowAnyMethod()
.AllowAnyHeader().AllowAnyOrigin());
});
// Automapper
builder.Services.AddAutoMapper(AppDomain.CurrentDomain.GetAssemblies());
builder.Services.Configure<KestrelServerOptions>(options =>
{
    //options.ListenAnyIP(8080);
    //options.ListenAnyIP(8585, listenOptions =>
    //{
    //    listenOptions.Protocols = Microsoft.AspNetCore.Server.Kestrel.Core.HttpProtocols.Http2;
    //});
    options.Limits.MaxConcurrentConnections = 100000;
    options.Limits.MaxConcurrentUpgradedConnections = 100000;
});
var app = builder.Build();
app.UseCors("CorsPolicy");
app.UseHttpsRedirection();
app.UseAuthorization();
app.MapControllers();
app.MapGrpcService<ChannelService>();
app.MapGet("/", () => "Communication with gRPC endpoints must be made through a gRPC client. To learn how to create a client, visit: https://go.microsoft.com/fwlink/?linkid=2086909");

app.Run();
void ConfigureDI(IServiceCollection services)
{
    builder.Services.AddDbContext<MavenXDBContext>(ServiceLifetime.Transient);
    builder.Services.AddScoped<IChannelsRepository, ChannelsRepository>();
    //builder.Services.AddScoped<IJobServce, JobService>();
    //builder.Services.AddScoped<IJobRepository, JobRepository>();
}

backend logs:

Grpc.Core.RpcException: Status(StatusCode="PermissionDenied", Detail="Bad gRPC response. HTTP status code: 403") at MavenxV2.GrpcServiceClient.ChannelGrpcClient.GetTopChannelsBasedOnDuration(GetTopRecommendedChannelsRequest request) in /src/MavenxV2.GrpcServiceClient/ChannelGrpcClient.cs:line 27 at MavenXV2.Gateway.Controllers.ChannelController.getTopChannelsBasedOnDurationGrpc(String userGuidId) in /src/MavenXV2.Gateway/Controllers/ChannelController.cs:line 298 at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(ActionContext actionContext, IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope) at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext) at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext) at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider) at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context) at IdentityServer4.Hosting.IdentityServerMiddleware.Invoke(HttpContext context, IEndpointRouter router, IUserSession session, IEventService events, IBackChannelLogoutService backChannelLogoutService) at IdentityServer4.Hosting.MutualTlsEndpointMiddleware.Invoke(HttpContext context, IAuthenticationSchemeProvider schemes) at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context) at IdentityServer4.Hosting.BaseUrlMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddlewareImpl.Invoke(HttpContext context)

HEADERS

Accept: / Host: dev-api-gateway.mavenx.gg User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Accept-Encoding: gzip, br Accept-Language: en-GB,en-US;q=0.9,en;q=0.8 Cookie: _ga=GA1.1.1813384287.1689857648; _ga_PJJ7ETHBF1=GS1.1.1691593660.4.0.1691593660.0.0.0 Referer: https://dev-api-gateway.mavenx.gg/swagger/index.html X-Request-ID: 4ada8ca76c9208f2fdba85a068113700 X-Real-IP: 192.168.62.194 X-Forwarded-For: 192.168.62.194 X-Forwarded-Host: dev-api-gateway.mavenx.gg X-Forwarded-Port: 443 X-Forwarded-Proto: https X-Forwarded-Scheme: https X-Scheme: https X-Original-Forwarded-For: 139.135.41.231 cdn-loop: cloudflare cf-ipcountry: PK cf-ray: 868743872d6a73b7-MRS cf-visitor: {"scheme":"https"} sec-ch-ua: "Chromium";v="122", "Not(A:Brand";v="24", "Google Chrome";v="122" DNT: 1 sec-ch-ua-mobile: ?0 sec-ch-ua-platform: "Windows" sec-fetch-site: same-origin sec-fetch-mode: cors sec-fetch-dest: empty priority: u=1, i cf-connecting-ip: 139.135.41.231

Here is Ingress.yaml

apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: grpc-ingress annotations: nginx.ingress.kubernetes.io/backend-protocol: "HTTP2" nginx.ingress.kubernetes.io/enable-cors: "true" nginx.ingress.kubernetes.io/cors-allow-origin: "*" nginx.ingress.kubernetes.io/cors-allow-credentials: "true" nginx.ingress.kubernetes.io/cors-allow-methods: "GET, PUT, POST, DELETE, OPTIONS" nginx.ingress.kubernetes.io/cors-allow-headers: "Authorization, Origin, X- Requested-With, Content-Type, Accept" kubernetes.io/ingress.class: nginx nginx.ingress.kubernetes.io/ssl-redirect: "true" nginx.ingress.kubernetes.io/use-regex: "true" nginx.ingress.kubernetes.io/rewrite-target: /$1 nginx.ingress.kubernetes.io/proxy-body-size: "0" nginx.ingress.kubernetes.io/proxy-connect-timeout: "7200" nginx.ingress.kubernetes.io/proxy_read_timeout: "9200" nginx.ingress.kubernetes.io/client-max-body-size: "5000m" nginx.ingress.kubernetes.io/client-body-buffer-size: "1000M"

spec: tls: - hosts: - xyz rules:

• host: xyz http: paths:
  • path: /(.*) pathType: Prefix backend: service: name: xyz port: number: 8080