Postgres inside Docker container, how to log JSON to Loki via Promtail?

Question

I have Postgres16.1 running in a container in Docker Swarm, alongside inter alia Promtail, Loki and Grafana. I want to direct Postgres logs in JSON format towards Promtail for easy processing in Promtail pipeline stages (especially of multiline logs). The setting Postgres config log_destination=jsonlog will create JSON logs in log/postgresql-<DATE>.json (see documentation here). But Docker stdout/stderr logs (which Promtail reads) will remain in default config.

docker-compose.yaml:

(...)
  db:
    container_name: postgres
    image: <from my local registry>
    command: ["postgres", "-c", "log_statement=all", "-c", "logging_collector=on", "-c", "log_destination=jsonlog"]
    labels:
      logging: "promtail"
(...)

How can I configure Postgres to log to stderr in JSON format? Also, is this a good idea/good practice in the first place?

Not having stderr as log_destination also does not seem to influence the stderr logging behaviour of the container.

Answer 1

Found a straightforward solution that does not require volume mounting or other workarounds here.

After having set logging_collector=onand log_destination=jsonlog, configure postgres log_filename to something static (exluding date/time as per default config) like postgresql.json (see doc here).

Then run the container with ln -sf /dev/stdout /var/lib/postgresql/data/log/postgresql.json.

This way, you shouldn't even have to worry about postgres log rotation, as the link will reroute to stdout directly and thus to docker log management.

Answer 2

Seems log_destination=jsonlog is not about stdout/err.

Probably makes sense to add a sidecar container with one more Promtail looking for a mounted folder with Postgres JSON logs. Alternatively, but not recommended, you can use supervisord with 2 processes: one for postgress and the second for a bash script that tail -f logs. In this case, you'd need to turn off default stdout logs and rely on only tail -f. Also, you have to care about log rotation, which is challenging but seems doable.

One more extreme and less recommended way is to use mkfifo pipes to send logs to and simultaneously read them with tail -f. In this case, you don't need to care about log rotation.