Terraform Pipeline Key Vault secret deployment with Private Endpoint

Question

We are deploying Azure Key Vaults with secrets in DevOps pipeline with terraform. In key vault module, we also add private endpoint. The service connection has Key Vault Admin IAM access on the subscription. The pipeline runs on a self-hosted agent. This agent has access to the newly deployed virtual network that private endpoint uses.

We get 403 error on the first run when the pipeline tries to deploy the secrets, without any change, the 2nd run would succeed. It's like the first run just doesn't include the IAM into account, but once it does on the 2nd run, it successfully creates secrets.

is there a way for the pipeline to "recognize" the IAM role when it was deployed the first time?

Answer 1

Key Vault secret deployment with Private Endpoint

The challenges encountered with IAM (Identity and Access Management) roles and permissions arise due to the time required for their propagation across the system. This delay can lead to the failure of operations reliant on newly assigned permissions if executed immediately after applying IAM roles or policies. In Azure, there is a brief delay before role assignments become active, which may cause initial attempts to fail while later ones succeed.

One strategy is to implement a delay or retry mechanism in the pipeline, particularly after assigning IAM roles and before initiating operations that need those permissions. Regrettably, Terraform doesn't inherently support a wait for IAM role propagation. Nonetheless, you can circumvent this constraint by utilizing Terraforms null_resource combined with a local-exec provisioner or external data sources, to either introduce a delay or execute a check that ensures the permissions have propagated.

I tried a demo version of terraform configuration how it works.

My terraform configuration:

provider "azurerm" {
  features {}
}


data "azurerm_client_config" "current" {}

data "azurerm_resource_group" "example" {
  name     = "rg-name"
}

resource "azurerm_virtual_network" "example" {
  name                = "testvk-vnet"
  address_space       = ["10.0.0.0/16"]
  location            = data.azurerm_resource_group.example.location
  resource_group_name = data.azurerm_resource_group.example.name
}

resource "azurerm_subnet" "example" {
  name                 = "testvk-subnet"
  resource_group_name  = data.azurerm_resource_group.example.name
  virtual_network_name = azurerm_virtual_network.example.name
  address_prefixes     = ["10.0.2.0/24"]

  service_endpoints = ["Microsoft.KeyVault"]
}

resource "azurerm_key_vault" "example" {
  name                        = "testvksskv"
  location                    = data.azurerm_resource_group.example.location
  resource_group_name         = data.azurerm_resource_group.example.name
  tenant_id                   = data.azurerm_client_config.current.tenant_id
  sku_name                    = "standard"

  network_acls {
    default_action = "Deny"
    bypass         = "AzureServices"
    ip_rules                   = ["yourIP"]
    virtual_network_subnet_ids = [azurerm_subnet.example.id]
  }
}

resource "azurerm_private_endpoint" "example" {
  name                = "testvk-endpoint"
  location            = data.azurerm_resource_group.example.location
  resource_group_name = data.azurerm_resource_group.example.name
  subnet_id           = azurerm_subnet.example.id

  private_service_connection {
    name                           = "example-connection"
    private_connection_resource_id = azurerm_key_vault.example.id
    is_manual_connection           = false
    subresource_names              = ["vault"]
  }
}

  resource "azurerm_key_vault_access_policy" "example" {
   key_vault_id = azurerm_key_vault.example.id
   tenant_id = data.azurerm_client_config.current.tenant_id
   object_id = data.azurerm_client_config.current.object_id

   key_permissions = ["Get", "List", "Update", "Create", "Import", "Delete", "Recover", "Backup", "Restore", "Decrypt", "Encrypt", "UnwrapKey", "WrapKey", "Verify", "Sign", "Purge", "Release", "Rotate", "GetRotationPolicy", "SetRotationPolicy"]

   secret_permissions = ["Get", "List", "Set", "Delete", "Recover", "Backup", "Restore", "Purge"]

   certificate_permissions = ["Get", "List", "Delete", "Recover", "Backup", "Restore", "Purge"]
}

resource "null_resource" "delay" {
  depends_on = [azurerm_key_vault.example]

  provisioner "local-exec" {
    interpreter = ["pwsh", "-Command"]
    command     = "Start-Sleep -Seconds 60"
  }
}


resource "azurerm_key_vault_secret" "example" {
  name         = "testvk-secret"
  value        = "s3cr3t"
  key_vault_id = azurerm_key_vault.example.id

  depends_on = [null_resource.delay]
}

deployment succeeded:

Answer 2

Terraform has a built-in resource, which simplifies these scenarios. It's called time_sleep.

In your scenario, the code would look like this:

resource "time_sleep" "wait_1_min" {
  depends_on = [azurerm_key_vault.example]

  create_duration = "1m"
}

resource "azurerm_key_vault_secret" "example" {
  name         = "testvk-secret"
  value        = "s3cr3t"
  key_vault_id = azurerm_key_vault.example.id

  depends_on = [time_sleep.wait_1_min]
}