How to use the elastic-agent container to collect service logs from another container?

Question

My target container contains NGINX logs which I wanted to collect from Elastic Fleet's NGINX Integration.

I followed every step, even successfully hosting the fleet server and the agent in two separate containers, what confuses me, is how can I configure my Agent which has the NGINX integration setup on its policy, to collect logs from the service container?

I have mostly encountered examples using the elastic-agent as a package installer directly on the target container.

I've attached three snippets of my docker-compose setup, that I follow for the Fleet, Agent and App containers.

FLEET SERVER

  fleet:
    image: docker.elastic.co/beats/elastic-agent:$ELASTIC_VERSION
    healthcheck:
      test: "curl -f http://127.0.0.1:8220/api/status | grep HEALTHY 2>&1 >/dev/null"
      retries: 12
      interval: 5s
    hostname: fleet
    container_name: fleet
    restart: always
    user: root
    environment:
    - FLEET_SERVER_ENABLE=1
    - "FLEET_SERVER_ELASTICSEARCH_HOST=https://elasticsearch:9200"
    - FLEET_SERVER_ELASTICSEARCH_USERNAME=elastic
    - FLEET_SERVER_ELASTICSEARCH_PASSWORD=REPLACE1
    - FLEET_SERVER_ELASTICSEARCH_CA=$CERTS_DIR/ca/ca.crt
    - FLEET_SERVER_INSECURE_HTTP=1
    - KIBANA_FLEET_SETUP=1
    - "KIBANA_FLEET_HOST=https://kibana:5601" 
    - KIBANA_FLEET_USERNAME=elastic
    - KIBANA_FLEET_PASSWORD=REPLACE1
    - KIBANA_FLEET_CA=$CERTS_DIR/ca/ca.crt
    - FLEET_ENROLL=1
    ports:
      - 8220:8220
    networks:
      - elastic
    volumes:
       - certs:$CERTS_DIR

Elastic Agent

agent:
    image: docker.elastic.co/beats/elastic-agent:$ELASTIC_VERSION
    container_name: agent
    hostname: agent
    restart: always
    user: root 
    healthcheck:
      test: "elastic-agent status"
      retries: 90
      interval: 1s
    environment:
      - FLEET_ENROLLMENT_TOKEN=REPLACE2
      - FLEET_ENROLL=1
      - FLEET_URL=http://fleet:8220
      - FLEET_INSECURE=1
      - ELASTICSEARCH_HOSTS='["https://elasticsearch:9200"]'
      - ELASTICSEARCH_USERNAME=elastic
      - ELASTICSEARCH_PASSWORD=REPLACE1
      - ELASTICSEARCH_CA=$CERTS_DIR/ca/ca.crt
      - "STATE_PATH=/usr/share/elastic-agent"
    networks:
      - elastic
    volumes:
       - certs:$CERTS_DIR

App Container (NGINX logs)

  demo-app:
    image: ubuntu:bionic
    container_name: demo-app
    build:
      context: ./docker/
      dockerfile: Dockerfile
    volumes:
      - ./app:/var/www/html/app
      - ./docker/nginx.conf:/etc/nginx/nginx.conf
    ports:
      - target: 90
        published: 9090
        protocol: tcp
        mode: host
    networks:
      - elastic

The ELK stack currently runs on version 7.17.0. If anyone could provide any info on what next needs to be done , It'll be very much helpful, thanks!

Answer 1

you could share nginx log files through volume mount. mount a directory to nginx log directory, and mount that to a directory in your elastic agent container. then youre good to harvest the nginx log in elastic agent container from there.

there might be directory read write permission problem, feel free to ask below.

kinda like:

nginx compose:

   demo-app:
     ...
     volumes:
       - ./app:/var/www/html/app
       - ./docker/nginx.conf:/etc/nginx/nginx.conf
+      - /home/user/nginx-log:/var/log/nginx/access.log
     ...

elastic agent compose:

 services:
   agent:
     ...
     volumes:
        - certs:$CERTS_DIR
+       - /home/user/nginx-log:/usr/share/elastic-agent/nginx-log