Temporarily disable kubernetes liveness probes for a pod

Question

I have a NodeJS app running inside a k8s pod and I want to take a heap dump of it.

In NodeJS taking a heap dump is time-consuming and blocks the main thread, so the pod is not able to respond to k8s liveness probes and is occasionally SIGKILLed.

Is there a way to prevent such behavior? F.e. stop liveness probes for a pod in runtime for let's say 10 minutes until a dump is ready. Or are there any known practices to handle cases similar to mine?

Answer 1

There is the open issue in k8s with the request similar to mine.

In the end I replaced the http probe with the exec probe to conditionally check some temporary file for existence:

#!/bin/sh

[ -f "/tmp/liveness-status" ] || curl -f http://localhost:8081/status >/dev/null 2>&1

So to run some long-running task (such as taking a dump) on a pod I should at first create some file:

1. kubectl exec <pod> – touch /tmp/liveness-status
2. take an action
3. kubectl exec <pod> – rm /tmp/liveness-status

Hope it helps somebody.

Answer 2

It's not possible to "pause" livenessProbe. Therefore, the options to consider are:

1. Adapt livenessProbe parameters to handle this 10 min down scenario; e.g. increase failureThreshold and periodSeconds
2. Solve it at the application / container level; e.g. you can have a second process running which replies to /health check
3. Solve it at the sidecar level; e.g. you can have a sidecar container which will handle /health check