Kubernetes - find pods without memory limits (or whose memory limits are too high above requests)

Question

On my kubernetes cluster, I have 15% (~4GB) extra GB of memory taken by my pods compared to my memory requests. I suspect it's been the reason some of my nodes have been crashing lately. How can I easily find the misconfigured pods and add the missing limits (ie find pods without memory requests, or whose memory limits are too high compared to requests ?

Answer 1

The easiest option is to use:

kubectl describe node your_node

This command gives you a lot of useful informations about your node and the list of pods running on it. This list includes CPU Requests, CPU Limits, Mem Requests, Mem Limits, etc.

This is fine if you have just a few nodes. But if you have a lot, is not optimal.

Another good option is to use k9s. With k9s cli, you have a nice overview of the running pods in your cluster and if you use the "wide" view (ctrl-w), you can also see all your pods limits and requests.

Answer 2

You can use custom-columns as output format for a get request.

The query syntax is jsonpath, https://kubernetes.io/docs/reference/kubectl/jsonpath/.

For example

#!/bin/bash

ns='NAMESPACE:.metadata.namespace'
pod="POD:.metadata.name"
cont='CONTAINER:.spec.containers[*].name'
mreq='MEM_REQ:.spec.containers[*].resources.requests.memory'
mlim='MEM_LIM:.spec.containers[*].resources.limits.memory'
creq='CPU_REQ:.spec.containers[*].resources.requests.cpu'
clim='CPU_LIM:.spec.containers[*].resources.limits.cpu'

kubectl get pod -A -o custom-columns="$ns,$pod,$cont,$mreq,$mlim,$creq,$clim"