Promtail -- reload at runtime

Question

Accoarding to the loki configuration page it is possible to reload promtail at runtime. link-loki_configuration_info

But whenever I try to reload, I get following error:

panic: duplicate metrics collector registration attempted

goroutine 200 [running]:
github.com/prometheus/client_golang/prometheus.(*Registry).MustRegister(0x3e7294d?, {0xc00194b0e0?, 0x1, 0xb?})
        /drone/src/vendor/github.com/prometheus/client_golang/prometheus/registry.go:405 +0x78
github.com/grafana/loki/clients/pkg/promtail/wal.NewWatcherMetrics({0x478b7b0, 0x648ec00})
        /drone/src/clients/pkg/promtail/wal/watcher_metrics.go:73 +0xa79
github.com/grafana/loki/clients/pkg/promtail/client.NewManager(0x0?, {0x4770b20, 0xc0006ba9b0}, {0x40c3880000000000, 0x2710, 0x0, 0x1, 0x0, 0x0, 0x0}, ...)
        /drone/src/clients/pkg/promtail/client/manager.go:61 +0x85
github.com/grafana/loki/clients/pkg/promtail.(*Promtail).reloadConfig(0xc0009481e0, 0xc0016ce000)
        /drone/src/clients/pkg/promtail/promtail.go:170 +0x88c
github.com/grafana/loki/clients/pkg/promtail.(*Promtail).reload(0xc0009481e0)
        /drone/src/clients/pkg/promtail/promtail.go:286 +0xa5
github.com/grafana/loki/clients/pkg/promtail.(*Promtail).watchConfig(0xc0009481e0)
        /drone/src/clients/pkg/promtail/promtail.go:271 +0x3d1
created by github.com/grafana/loki/clients/pkg/promtail.(*Promtail).Run in goroutine 1
        /drone/src/clients/pkg/promtail/promtail.go:214 +0xcd

Loki version:

loki, version 2.9.3 (branch: HEAD, revision: 2535f9bede)
  build user:       root@998f10a08814
  build date:       2023-12-11T19:17:52Z
  go version:       go1.21.3
  platform:         windows/amd64
  tags:             netgo

Promtail version:

promtail, version 2.9.3 (branch: HEAD, revision: 2535f9bede)
  build user:       root@998f10a08814
  build date:       2023-12-11T19:17:52Z
  go version:       go1.21.3
  platform:         windows/amd64
  tags:             netgo

Any ideas how to fix this? Thanks.

I expect that after sending a HTTP POST request to the /reload endpoint promtail will recognize the updated config file and continue running.

Answer 1

Your problem is described in this issue. Fix for it is merged into main, but I don't see it in the release branch.

I don't familiar with release process for loki, but you could probably cherry-pick solution into separate branch based on release_2.9.x and proposed a PR for maintainers to review.

Otherwise, I don't think you have much options except waiting for 2.10 or compiling promtail with included fix yourself.