Cannot write Micrometer metrics to Google Stackdriver Monitoring: Permission Denied

Question

I am receiving

io.grpc.StatusRuntimeException: PERMISSION_DENIED: Permission monitoring.metricDescriptors.list denied (or the resource may not exist).
    at io.grpc.Status.asRuntimeException(Status.java:539)
    ... 14 common frames omitted
Wrapped by: com.google.api.gax.rpc.PermissionDeniedException: io.grpc.StatusRuntimeException: PERMISSION_DENIED: Permission monitoring.metricDescriptors.list denied (or the resource may not exist).

from io.micrometer.stackdriver.StackdriverMeterRegistry on GCP.

My GKE cluster is configured like so:

resource "google_container_cluster" "primary-cluster" {
  provider = google-beta

  project  = var.project_id
  name     = "${var.project_id}-autopilot-cluster"
  location = var.region

  node_locations = toset(var.k8s_node_zones)

  ip_allocation_policy {
  }

  network    = google_compute_network.vpc.name
  subnetwork = google_compute_subnetwork.vpc-subnet.name

  min_master_version = var.k8s_min_cluster_version

  release_channel {
    channel = var.k8s_release_channel
  }

  enable_autopilot = true

  cluster_autoscaling {
    auto_provisioning_defaults {
      service_account = google_service_account.gke-service-account.email
      image_type      = "COS_CONTAINERD"
      disk_size       = 10
      oauth_scopes = [
        "https://www.googleapis.com/auth/cloud-platform",
        "https://www.googleapis.com/auth/devstorage.full_control",
        "https://www.googleapis.com/auth/logging.write",
        "https://www.googleapis.com/auth/monitoring",
        "https://www.googleapis.com/auth/monitoring.read",
        "https://www.googleapis.com/auth/monitoring.write"
      ]
    }
  }

}

resource "google_project_iam_member" "logging_writer" {
  project = var.project_id
  role    = "roles/logging.logWriter"
  member  = "serviceAccount:${google_service_account.gke-service-account.email}"
}

resource "google_project_iam_member" "metric_writer" {
  project = var.project_id
  role    = "roles/monitoring.metricWriter"
  member  = "serviceAccount:${google_service_account.gke-service-account.email}"
}

and the gke-service-account has got the roles/monitoring.metricWriter role. I do see the OOTB GKE metrics as well as logging works, too.

Autopilot clusters in GCP have forced Workload Identity. Could it have something to do with that? It would be confusing as I can see logs in Stackdriver Logging as well as Error Reporting. Also, one of my PODs is reaching Cloud Storage using com.google.cloud.storage.Storage without problems, too.

Answer 1

Workload Identity will allow you to associate a Kubernetes service account with a Google service account. It enables more secure and fine-grained access control when your applications running on GKE need to interact with various GCP services.

To configure Workload Identity on your GKE clusters, see this document.