Scenario
I am using helm to facilitate creation of deployments and services for jaeger. Specifically https://github.com/jaegertracing/helm-charts
I have an application running on the same k8s that pushed traces to
http://jaeger-collector.observability.svc.cluster.local:4317 and these work fine. (observability is the namespace where jaeger is installed)
Now i also have an external application which I want to forward traces to jaeger aswell.
Helm chart values below
...ommitted...
collector:
enabled: true
basePath: /
service:
otlp:
grpc:
name: otlp-grpc
port: 4317
http:
name: otlp-http
port: 4318
ingress:
enabled: true
ingressClassName: nginx
hosts:
host: collector.example.com
...ommitted...
Ingress Problem and workaround
However there is a problem in the above helm chart, it seems to only let me create an ingress for port 14268 only. I have worked around that, and for testing purposes I have updated the generated yml file directly on k8s, and replaced the port as below
spec:
ingressClassName: nginx
rules:
- host: collector.example.com
http:
paths:
- path: /
pathType: ImplementationSpecific
backend:
service:
name: jaeger-collector
port:
number: 4317 #was 14268
My expectations were that nginx (ingress controller) will be listetning on port 80 and 443 and will forward all requests to jaeger-collector.
Analysis
This still doesnt work. My collector service is working correctly 100% as applications running on k8's manage to push data. So the problem seems to be something with the ingress or something before that.
From what i can tell, the (modified) ingress is correct. My application code is .Net C#.
Application code
public static IServiceCollection AddAppTelemetry(this IServiceCollection services, IConfiguration configuration, IWebHostEnvironment webHostEnvironment)
{
services.AddOpenTelemetry()
.ConfigureResource(r => r.AddService(serviceName: webHostEnvironment.ApplicationName))
.WithTracing(builder =>
{
builder
.AddAspNetCoreInstrumentation()
.AddHttpClientInstrumentation()
.AddNpgsql()
.AddRedisInstrumentation()
.AddOtlpExporter();
});
services.Configure<OtlpExporterOptions>(configuration.GetSection("OpenTelemetry:otlp"));
return services;
}
and my appsettings options
"OpenTelemetry": {
"otlp": {
"Endpoint": "http://collector.example.com:80"
}
},
Possible issues and solution attempts
- I am running collector.example.com behind cloudflare proxy, I disabled this to no avail.
- Curl and telnet to collector.example.com works fine.
- Tried creating my own ingress instead of the helm one, no luck
- My own service that listens on port 80/443 and redirects to 4317
At this point I'm out of ideas, I only just learned Helm and jaeger since yesterday, and im quite fresh with k8s aswell. Any help is really appreciated.