Elastic APM responding with 503 (not publish ready)

Question

I have a locally running elasticsearch and kibana stack. I'm looking to use it to track APM for a nodejs application I also have running locally.

I've installed elastic agent in an ubuntu docker container as a fleet server. On a second ubuntu docker container I have my APM elastic agent running, which enrolled through the fleet server. My APM settings in kibana are very basic, with only changes to the Host and URL fields:

I've configured my node app with the following snippet (at the top of the main file):

// Add this to the very top of the first file loaded in your app
var apm = require('elastic-apm-node').start({
    serviceName: 'my-service-name',
    secretToken: '',
    serverUrl: 'http://apm.mydomain.com:8200',
    environment: 'my-environment'
});

I receive the following error when running the app:

{"log.level":"error","@timestamp":"2023-12-30T20:36:03.605Z","log.logger":"elastic-apm-node","ecs.version":"8.10.0","message":"APM Server transport error (503): Unexpected APM Server response when polling config\nAgent configuration infrastructure is not ready. Please retry later."}

When I query the apm agent directly (GET http://apm.mydomain.com:8200/) I receive the following response:

{
    "build_date": "2023-12-07T11:11:14-05:00",
    "build_sha": "cba21e88bd0d376ec77dc77fccccfdc0c27206ac",
    "publish_ready": false,
    "version": "8.11.3"
}

Does anyone know why my agent would not be accepting APM data?

Answer 1

For anyone that finds this in the future, the issue was despite my elastic agents being marked as healthy in kibana, they were unable to post logs to my elastic cluster due to failed certificate validation (I'm using self-signed certs).

The agents themselves did not report any errors indicating a failed certificate verification. I was able to discover the issue in the elastic cluster logs.

I defined my CA cert everywhere possible, but eventually had to add the following to the docker-compose configuration the agents are running in, which solved the problem:

environment:
      - ELASTICSEARCH_SSL_CERTIFICATE_VERIFICATION=false