Cloud Logging doesn't work on a Container-optimized OS running a container

Question

I'm running a Docker container on a GCE instance under the Container-optimizes OS. I've followed this manual https://cloud.google.com/container-optimized-os/docs/how-to/logging and added the google-logging-enabled metadata value to enable sending containers' logs to Cloud Logging:

instance metadata screenshot

But it doesn't work. I can't see containers' logs in the Cloud Logging:

cloud logging screenshot

While I have a lot of container logs in reality:

container logs screenshot

How to make it working and really send containers' logs to Cloud Logging?

Update The log filter is the default one: (resource.type="gce_instance" AND resource.labels.instance_id="***") OR (resource.type="global" AND jsonPayload.instance.id="***") that means that all types of logs from the current instance should be shown.

Answer 1

My issue was that my service account associated with the instance had no the Logs Writer permission. I found that by running the sudo systemctl status stackdriver-logging command which shown the following:

● stackdriver-logging.service - Fluentd container for Stackdriver Logging
   Loaded: loaded (/usr/lib/systemd/system/stackdriver-logging.service; static; vendor preset: disabled)
   Active: active (running) since Fri 2021-03-05 01:18:55 UTC; 1h 12min ago
 Main PID: 385 (docker)
    Tasks: 9 (limit: 2382)
   Memory: 31.9M
      CPU: 751ms
   CGroup: /system.slice/stackdriver-logging.service
           └─385 /usr/bin/docker run --rm --name=stackdriver-logging-agent -v /etc/stackdriver/logging.config.d/:/etc/google-fluentd/config.d/ -v /var/log:/var/log -v /var/lib/docker/containers/:/v

Mar 05 02:30:11 jwp-gitlab-runner-m3l0 docker[385]: 2021-03-05 02:30:11 +0000 [warn]: #0 Dropping 1 log message(s) error="User unauthorized to access *** for resource ***

By googling the error message I found the answer in the GitHub issue https://github.com/GoogleCloudPlatform/fluent-plugin-google-cloud/issues/295

The following answer also helped me to diagnose the problem: https://stackoverflow.com/a/65768214/8016720

Now I have all the logs I want in Cloud Logging.