I have 5 m5.xlarge nodes. When I create a new pod that cannot be scheduled on these nodes due to CPU/memory constraints, the cluster-autoscaler/karpenter triggers the creation of a new node, and the pod is then scheduled to this new node. I'm looking to use Prometheus/Grafana to measure the time taken for:
Node creation, Pod scheduling on the new node, and The pod reaching a 'Ready' status.
I attempted to calculate this using the formula timestamp(kube_pod_status_unschedulable{pod="podName"}) - timestamp(kube_pod_status_scheduled{pod="podName"}), but it hasn't worked as expected.
I except to see something like: Pod1: 3m to be scheduled Pod2: 2m24s to be scheduled etc