Create notifications when GCP secret nears expiry

Question

Whenever a GCP secret nears it's expiration date, it creates expiration logs in Secret Manager secret resource as mentioned here. It states one can create Log-based metrics and use them to create alerts for upcoming expirations. But the log based metric do not provide a filter for this. How can it be accomplished?

Answer 1

To view logs of Secret’s Expiration, use the following log query in Logs Explorer,

severity=NOTICE
jsonPayload.type: "EXPIRE"

We then can create an User-defined metric on this filter, follow the below steps to create counter-type logs-based metrics:

1. Go to the Logging > Logs-based Metrics page

2. Click Create Metric, select Metric Type as “Counter” and fill Name, Description and Units(can be left blank).

3. In Build filter column use the following query,

   severity=NOTICE  
   jsonPayload.type: "EXPIRE"
   

4. Click Create Metric. Refer to Creating counter metrics for more information.

After the User-defined metric is created, we now create an alert on the above metric depending on our desired Violation rules, follow the below steps to create an alerting policy on a logs-based counter metric:

1. Go to the Logging > Logs-based Metrics page.
2. Find the metric you want to explore and select Create alert from metric in the metric's More menu. The Conditions pane opens
3. In Target dialog the Metric field is populated automatically with user-defined metric and then we can make our desired modifications across each field and click Save and Click Next.
4. In Notification Channels, we are provided with many channels like Mail, SMS, Cloud Pub/Sub..etc from which we can select a channel of our preference.
5. Next, give the policy a name in the Alert Name column, Click Save. Refer to Creating an alerting policy on a counter metric for more information.

I implemented the above steps with my Mail ID as a Notification Channel and was able to view the alerts without any problem.

Answer 2

The easiest way to create a logs-based metric is from the Logs Explorer page. You can use a query like the following to find all log messages where the secret will expire in 1 hour:

resource.type="secretmanager.googleapis.com/Secret"
jsonPayload.type="EXPIRES_IN_1_HOUR"

As noted in the Secret Manager expiration documentation, there are other event types you could subscribe to including:

• EXPIRES_IN_30_DAYS
• EXPIRES_IN_7_DAYS
• EXPIRES_IN_1_DAY
• EXPIRES_IN_6_HOURS
• EXPIRES_IN_1_HOUR
• EXPIRED

Then, from the Actions menu, choose "Create metric":

The metric type should be "Counter". You can provide whatever name and description you'd like. The units should be unitless (1).

You can optionally add a label. For example, if you wanted to label the metric with the secret name, you could add a label named "secret" targeting the jsonPayload.name field. If you wanted to get extra fancy, you could strip off the leading projects/.../secrets prefix using a regular expression extractor like projects/.+/secrets/(.+).

---

After you've created the metric, you can create an alert under Monitoring -> Alerting -> Create Policy. Choose the name of your metric (it will be logging.googleapis.com/user/NAME_YOU_CHOSE).

Set the aligner to count and the aggregator to sum, and then set the condition to be above 1.

You will need to configure notification channels to actually get alerted, but you can watch the alert fire in the web UI too.

Answer 3

In the drop down menu of the screenshot shared by @sethvargo there is Create log alert that can be used now to create an alert policy directly without going through the Monitoring -> Alerting -> Create Policy

Example of such resulting alerting policy in the screenshot below: