I have injected an OpenTelemetry JAVA agent in a Spring application in OpenShift via kubernetes operator, but most instrumentations are disabled and not generating traces.
OpenTelemetry Agent image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:1.28.0
JAVA version: openjdk 11.0.15
Kubernetes version: v1.25.10+3fe2906
After setting OTEL_JAVAAGENT_DEBUG to true, I see most instrumentations are disabled:
...
[otel.javaagent 2023-07-26 11:36:50:133 -0300] [main] DEBUG io.opentelemetry.javaagent.tooling.instrumentation.InstrumentationLoader - Loading instrumentation executors [class io.opentelemetry.javaagent.instrumentation.executors.ExecutorsInstrumentationModule]
[otel.javaagent 2023-07-26 11:36:50:133 -0300] [main] DEBUG io.opentelemetry.javaagent.extension.instrumentation.InstrumentationModule - Instrumentation executors is disabled
[otel.javaagent 2023-07-26 11:36:50:133 -0300] [main] DEBUG io.opentelemetry.javaagent.tooling.instrumentation.InstrumentationLoader - Loading instrumentation internal-application-logging [class io.opentelemetry.javaagent.instrumentation.internal.logging.ApplicationLoggingInstrumentationModule]
[otel.javaagent 2023-07-26 11:36:50:133 -0300] [main] DEBUG io.opentelemetry.javaagent.extension.instrumentation.InstrumentationModule - Instrumentation internal-application-logging is disabled
...
Lettuce instrumentation works as intended, generating traces which show up in our OpenTelemetry collector:
...
[otel.javaagent 2023-07-26 11:36:50:838 -0300] [main] DEBUG io.opentelemetry.javaagent.tooling.instrumentation.InstrumentationLoader - Loading instrumentation lettuce [class io.opentelemetry.javaagent.instrumentation.lettuce.v4_0.LettuceInstrumentationModule]
[otel.javaagent 2023-07-26 11:36:50:842 -0300] [main] DEBUG io.opentelemetry.javaagent.tooling.instrumentation.InstrumentationLoader - Loading instrumentation lettuce [class io.opentelemetry.javaagent.instrumentation.lettuce.v5_0.LettuceInstrumentationModule]
[otel.javaagent 2023-07-26 11:36:50:908 -0300] [main] DEBUG io.opentelemetry.javaagent.tooling.instrumentation.InstrumentationLoader - Loading instrumentation lettuce [class io.opentelemetry.javaagent.instrumentation.lettuce.v5_1.LettuceInstrumentationModule]
...
[otel.javaagent 2023-07-26 11:37:07:128 -0300] [main] DEBUG io.opentelemetry.javaagent.extension.instrumentation.InstrumentationModule - Applying instrumentation: lettuce [class io.opentelemetry.javaagent.instrumentation.lettuce.v5_1.LettuceInstrumentationModule] on org.springframework.boot.loader.LaunchedURLClassLoader@64db4967
[otel.javaagent 2023-07-26 11:37:07:128 -0300] [main] DEBUG io.opentelemetry.javaagent.tooling.HelperInjector - Injecting classes onto class loader org.springframework.boot.loader.LaunchedURLClassLoader@64db4967 -> [io.opentelemetry.javaagent.instrumentation.lettuce.v5_1.TracingHolder, io.opentelemetry.javaagent.shaded.instrumentation.lettuce.v5_1.LettuceTelemetry, io.opentelemetry.javaagent.shaded.instrumentation.lettuce.v5_1.OpenTelemetryTracing, io.opentelemetry.javaagent.shaded.instrumentation.lettuce.v5_1.LettuceTelemetryBuilder, io.opentelemetry.javaagent.shaded.instrumentation.lettuce.v5_1.OpenTelemetryTracing$1, io.opentelemetry.javaagent.shaded.instrumentation.lettuce.v5_1.LettuceNetAttributesGetter, io.opentelemetry.javaagent.shaded.instrumentation.lettuce.v5_1.OpenTelemetryTracing$OpenTelemetryEndpoint, io.opentelemetry.javaagent.shaded.instrumentation.lettuce.v5_1.OpenTelemetryTracing$OpenTelemetryTraceContextProvider, io.opentelemetry.javaagent.shaded.instrumentation.lettuce.v5_1.OpenTelemetryTracing$OpenTelemetryTracerProvider, io.opentelemetry.javaagent.shaded.instrumentation.lettuce.v5_1.OpenTelemetryTracing$OpenTelemetryTracer, io.opentelemetry.javaagent.shaded.instrumentation.lettuce.v5_1.OpenTelemetryTracing$OpenTelemetryTraceContext, io.opentelemetry.javaagent.shaded.instrumentation.lettuce.v5_1.OpenTelemetryTracing$OpenTelemetrySpan, io.lettuce.core.protocol.OtelCommandArgsUtil, io.opentelemetry.javaagent.shaded.instrumentation.lettuce.common.LettuceArgSplitter]
[otel.javaagent 2023-07-26 11:37:07:128 -0300] [main] DEBUG io.opentelemetry.javaagent.tooling.HelperInjector - Injecting classes onto class loader <bootstrap> -> [io.opentelemetry.javaagent.bootstrap.field.VirtualFieldAccessor$io$lettuce$core$protocol$AsyncCommand$io$opentelemetry$javaagent$shaded$io$opentelemetry$context$Context]
[otel.javaagent 2023-07-26 11:37:07:129 -0300] [main] DEBUG io.opentelemetry.javaagent.tooling.HelperInjector - Injecting classes onto class loader <bootstrap> -> [io.opentelemetry.javaagent.bootstrap.field.VirtualFieldImpl$io$lettuce$core$protocol$AsyncCommand$io$opentelemetry$javaagent$shaded$io$opentelemetry$context$Context]
[otel.javaagent 2023-07-26 11:37:07:132 -0300] [main] DEBUG io.opentelemetry.javaagent.tooling.AgentInstaller$TransformLoggingListener - Transformed io.lettuce.core.resource.DefaultClientResources -- org.springframework.boot.loader.LaunchedURLClassLoader@64db4967
...
I have tried setting both default and individual instrumentation flags to 'true':
OTEL_INSTRUMENTATION_COMMON_DEFAULT_ENABLED=true
OTEL_INSTRUMENTATION_TOMCAT_ENABLED=true
But nothing changed. Most instrumentations were still disabled with only a couple working properly.
Just to be sure, I tried disabling lettuce instrumentation:
OTEL_INSTRUMENTATION_LETTUCE_ENABLED=false
And it worked. It showed as disabled and stopped generating traces, so I don't think it has to do with enabling/disabling instrumentations. Moreover, we have working java instrumentation agents on both Azure and AWS kubernetes, which leads me to believe it has something to do with our OpenShift cluster.
Has anyone experienced this issue before?
The OneAgent does not disable any opentelemetry-modules. The spans exported by the OpenTelemetry agent will look exactly like the OneAgent isn't present at all. The OneAgent just won't integrate spans from certain otel-agent-modules into its own distributed traces structure.
So, we don't suppress the OTel Java agent from creating all the spans it needs to and then sending them to whatever exporters are configured. That will work and we don't touch the spans generated/created.
Which means that in e.g. Jaeger, you will see all the spans as if the OneAgent was not present.
However, if you have the Dynatrace OTel code module integration enabled, the spans that the OA ingests via the code module (NOT by the OTel exporter) will be missing certain instrumentation libraries that we know the OA is already instrumenting well - such as HTTP).
So, we do suppress some spans, but this will only be visible inside Dynatrace, when we use the code module ingestion. Whatever exporter you have configured on the OTel Java agent is unaffected by this.
This is basically to avoid double tracing and a lot of duplicated spans in your Distributed Trace view.