We use GKE Autopilot, and configure everything with helm and terraform. We are seeing heaps of error logs due to what looks like incorrect error formatting coming through to stackdriver. The (mostly) info logs are coming through are just textPayload with level=info in them and some, like node-cache seems to output a log line by line and then the entire contents of Corefile, like this
I'm not sure there's any config I can actually do here, as from reading it the docs it would seem the default behaviour is to send logs to stackdriver, so I would imagine this to just work?
I raised a bug issue for the managed prometheus service https://issuetracker.google.com/issues/270593811
But there's many others and considering the sheer number of error logs, it seems odd that no one else seems to be complaining about it. Leading me to believe we have some incorrect config
From looking at the config, there appears to only be logging_service and logging_config, neither of which seem to be what I need and so are default.
Genuinely out of ideas unfortunately!