I try to migrate rails server that running directly on EC2 instance to EKS pod.
The server runs with unicorn_rails that has 20+ worker, also it has nginx as frontend that receive requests and communicate with unicorn_rails via unix domain socket.
When it run directly on EC2 isntance and ALB with target group of these instances, latency always < 500ms and fluctuation range was only ±10 ms
But when it runs as pod on EKS, latency increases nearly a hundreds ms on average and fluctuation range become ±100 ms, even if it receives only a few request per second per pod. (for EKS nodepool, using same instance type of ec2 version)
Is it common for the latency of a request sent from alb to an eks pod to be this large or unstable compared to a request sent to an ec2 instances, even if request flow is nearly same?
belows are details of migration.
To migrate this server to eks pod, I use following 2 containers
- unicorn_rails with single worker of it
- nginx as sidecar container of 1, receives requests from alb
container 1 and 2 communicate with unix domain socket like EC2 instance version of the server does.
I believe request flow nearly same between ec2 version and eks version, so cannot figure out why latency behavior is so different.
old: alb =(network)=> EC2 (nginx) =(unix domain socket)=> EC2 (unicorn rails worker)
new: alb =(network)=> EKS pod (nginx) =(unix domain socket)=> EKS pod (unicorn rails worker)
reducing unicorn_rails worker is critical? or kube-proxy adds overhead?
any thoughts?