How do I add s3 bucket object with Thanos and prometheus

Question

Thanos, Prometheus, and Grafana were each placed in the same ec2 as containers. At this time, ec2 is in the public subnet, and eks is in the private subnet. I want to use Thanos to store Prometheus' metrics in an AWS S3 bucket, but there seems to be a problem connecting Prometheus and Thanos. In the current situation, even if all containers are running, objects are not being created in the AWS S3 bucket. The error log below is an error in the Prometheus container, and below it is the Docker-Compose file. Could you please let me know what the problem is?

ts=2023-09-xxxxx:42:44.---- caller=dedupe.go:112 component=remote level=warn remote_name=****da url=http://192.xxx.xx.x:xxxxx/api/v1/write msg="Failed to send batch, retrying" err="Post \"http://192.xxx.xx.x:xxxxx/api/v1/write\": write tcp 192.xxx.xx.4:35630->192.xxx.xx.5:xxxxx: write: connection reset by peer"

I tried changing the endpoint and modifying the Docker compose file in various ways, but I couldn't solve it.

Answer 1

The error message write tcp 192.xxx.xx.4:35630->192.xxx.xx.5:xxxxx: write: connection reset by peer indicates that the Prometheus container is trying to send metrics to the Thanos container on port xxxxx, but the connection is being reset by the peer. This can happen for a few reasons, such as:

• The Thanos container is not listening on port xxxxx.
• There is a firewall or other network device blocking traffic between the Prometheus and Thanos containers.
• The Prometheus and Thanos containers are on different subnets and cannot communicate with each other.

In your case, it is most likely that the Prometheus and Thanos containers are on different subnets and cannot communicate with each other. This is because you mentioned that the EC2 instance is in the public subnet and the EKS cluster is in the private subnet.

To fix this problem, you need to create a VPC peering connection between the public subnet and the private subnet. This will allow the Prometheus and Thanos containers to communicate with each other.

Once you have created a VPC peering connection, you need to update the Prometheus configuration to point to the Thanos endpoint on the private subnet. You can do this by setting the remote_write.targets parameter in the Prometheus configuration file to the Thanos endpoint.

For example, if the Thanos endpoint is http://192.168.1.10:9090, you would set the remote_write.targets parameter to ["http://192.168.1.10:9090"].

Once you have updated the Prometheus configuration, you need to restart the Prometheus container.

Here is an example of a Docker Compose file that you could use to run Prometheus and Thanos in different subnets:

version: "3.7"

services:
  prometheus:
    image: prom/prometheus:latest
    networks:
      - public
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"

  thanos:
    image: thanosio/thanos:latest
    networks:
      - private
    ports:
      - "9090:9090"

networks:
  public:
    name: public
    type: public

  private:
    name: private
    type: private

To run the Docker Compose file, you can use the following command:

docker-compose up -d

This will start the Prometheus and Thanos containers in different subnets.

Once the Prometheus and Thanos containers are running, you should be able to see objects being created in the AWS S3 bucket.

If you are still having problems, please provide more information about your environment, such as the output of the following commands:

docker ps
docker network inspect public
docker network inspect private