How to send Next.js trace data to Datadog using OpentelemetryCollector

Question

I would like to send trace data from a Next.js application I am developing in TypeScript to Datadog, but I can't figure it out after reading the documentation and have not been able to do that. I have specified Datadog as the exporter in collector.yaml and it shows graphs for CPU, memory, etc., but it does not show Traces data for GET and POST requests to the application.

Looking at the app log of the Docker container, it seems that traces are being made, but they are not being sent to Datadog.

How can I use OpentelemetryCollector to send application trace data to Datadog?

I am running in a local environment on a mac and don't know how to reference the logs in the Docker container, so I have commented out the filelog, is this the cause?

Here is what I have set up and the graph in question.

English may be unnatural due to the use of translation software. Sorry.

Translated with DeepL.com (free version)

・collector.yaml

receivers:
  otlp:
    protocols:
      http:
        endpoint: "localhost:4318"
      grpc:
        endpoint: "localhost:4317"
  hostmetrics:
    collection_interval: 10s
    scrapers:
      paging:
        metrics:
          system.paging.utilization:
            enabled: true
      cpu:
        metrics:
          system.cpu.utilization:
            enabled: true
      disk:
      filesystem:
        metrics:
          system.filesystem.utilization:
            enabled: true
      load:
      memory:
      network:
      processes:
  prometheus:
    config:
      scrape_configs:
      - job_name: 'otelcol'
        scrape_interval: 10s
        static_configs:
        - targets: ['0.0.0.0:8888']

  # filelog:
  #   include_file_path: true
  #   poll_interval: 500ms
  #   include:
  #     - /var/log/**/*example*/*.log

processors:
  batch:
    send_batch_max_size: 1000
    send_batch_size: 100
    timeout: 10s

exporters:
  datadog:
    api:
      site: datadoghq.com
      key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

service:
  pipelines:
    metrics:
      receivers: [hostmetrics, prometheus, otlp]
      processors: [batch]
      exporters: [datadog]
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [datadog]
    logs:
      # receivers: [otlp, filelog]
      receivers: [otlp]
      processors: [batch]
      exporters: [datadog]

・instrumentation.ts

/*instrumentation.ts*/
import { NodeSDK } from '@opentelemetry/sdk-node';
import { ConsoleSpanExporter } from '@opentelemetry/sdk-trace-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import {
  PeriodicExportingMetricReader,
  ConsoleMetricExporter,
} from '@opentelemetry/sdk-metrics';

const sdk = new NodeSDK({
  traceExporter: new ConsoleSpanExporter(),
  metricReader: new PeriodicExportingMetricReader({
    exporter: new ConsoleMetricExporter(),
  }),
  instrumentations: [getNodeAutoInstrumentations()],
});

sdk.start();

enter image description here

Answer 1

The issue is that the instrumentation is not actually exporting the data anywhere.

You are currently using ConsoleSpanExporter and ConsoleMetricExporter which exports Spans and Metrics to the console.

You need to change your instrumentation to use the OTLP exporters.

import * as opentelemetry from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-proto';
import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-proto';
import { PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics';

const sdk = new opentelemetry.NodeSDK({
  traceExporter: new OTLPTraceExporter({
    // optional - default url is http://localhost:4318/v1/traces
    url: '<your-collector-endpoint>/v1/traces',
  }),
  metricReader: new PeriodicExportingMetricReader({
    exporter: new OTLPMetricExporter({
      // optional - default is http://localhost:4318/v1/metrics
      url: '<your-collector-endpoint>/v1/metrics',
    }),
  }),
  instrumentations: [getNodeAutoInstrumentations()],
});
sdk.start();

If your collector is exposing ports 4317 and 4318 to your localhost, you can remove the url from OTLPTraceExporter and OTLPMetricExporter.

If you prefer, you can also set the following environment variable in your application Dockerfile or Docker compose file to point the OTLP data to the collector:

OTEL_EXPORTER_OTLP_ENDPOINT=http://<otel-collector-endpoint>:<otel-collector-port>

Another thing that I've noticed is that if you are using the OTel Collector version 0.95.0+, you will also need the datadog/connector component in your collector config:

receivers:
  otlp:
    protocols:
      http:
        endpoint: "localhost:4318"
      grpc:
        endpoint: "localhost:4317"
  hostmetrics:
    collection_interval: 10s
    scrapers:
      paging:
        metrics:
          system.paging.utilization:
            enabled: true
      cpu:
        metrics:
          system.cpu.utilization:
            enabled: true
      disk:
      filesystem:
        metrics:
          system.filesystem.utilization:
            enabled: true
      load:
      memory:
      network:
      processes:
  prometheus:
    config:
      scrape_configs:
      - job_name: 'otelcol'
        scrape_interval: 10s
        static_configs:
        - targets: ['0.0.0.0:8888']

  # filelog:
  #   include_file_path: true
  #   poll_interval: 500ms
  #   include:
  #     - /var/log/**/*example*/*.log

connectors:
    datadog/connector:

processors:
  batch:
    send_batch_max_size: 1000
    send_batch_size: 100
    timeout: 10s

exporters:
  datadog:
    api:
      site: datadoghq.com
      key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

service:
  pipelines:
    metrics:
      receivers: [hostmetrics, prometheus, otlp, datadog/connector]
      processors: [batch]
      exporters: [datadog]
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [datadog, datadog/connector]
    logs:
      # receivers: [otlp, filelog]
      receivers: [otlp]
      processors: [batch]
      exporters: [datadog]

Answer 2

I changed the receiver URL from the default localhost to opentelemetry-collector and changed the endpoint to the following and the trace data is displayed!

・collector.yaml

receivers:
  otlp:
    protocols:
      http:
        endpoint: "0.0.0.0:4318"
      grpc:
        endpoint: "0.0.0.0:4317"
  hostmetrics:
    collection_interval: 10s
    scrapers:
      paging:
        metrics:
          system.paging.utilization:
            enabled: true
      cpu:
        metrics:
          system.cpu.utilization:
            enabled: true
      disk:
      filesystem:
        metrics:
          system.filesystem.utilization:
            enabled: true
      load:
      memory:
      network:
      processes:
  prometheus:
    config:
      scrape_configs:
      - job_name: 'otelcol'
        scrape_interval: 10s
        static_configs:
        - targets: ['0.0.0.0:8888']

  # filelog:
  #   include_file_path: true
  #   poll_interval: 500ms
  #   include:
  #     - /var/log/**/*example*/*.log

processors:
  batch:
    send_batch_max_size: 1000
    send_batch_size: 100
    timeout: 10s

exporters:
  datadog:
    api:
      site: datadoghq.com
      key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

service:
  pipelines:
    metrics:
      receivers: [hostmetrics, prometheus, otlp]
      processors: [batch]
      exporters: [datadog]
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [datadog]
    logs:
      # receivers: [otlp, filelog]
      receivers: [otlp]
      processors: [batch]
      exporters: [datadog]

・instrumentation.ts

import * as opentelemetry from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-proto';
import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-proto';
import { PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics';

const sdk = new opentelemetry.NodeSDK({
  traceExporter: new OTLPTraceExporter({
    url: 'http://opentelemetry-collector:4318/v1/traces',
  }),
  metricReader: new PeriodicExportingMetricReader({
    exporter: new OTLPMetricExporter({
      url: 'http://opentelemetry-collector:4318/v1/metrics',
    }),
  }),
  instrumentations: [getNodeAutoInstrumentations()],
});
sdk.start();