Long running backend process giving 504 gateway timeout error. SpringBoot backend, AWS EKS Ingress

Question

I am facing a problem where my backend application is doing some complex calculation and due to which its taking more more than 2 minutes to complete and I am constantly getting 504 gateway timeout.

When I check the logs using pod logs watch command, I can see that the process is running smoothly and its finished its task without any error.

Please help to fix.

Answer 1

Here is my solution:

My backend application (SprintBoot app) was performing some complex calculation and I was getting 504 Gateway Timeout, while the calculation was performing its work at the backend.

After investing the issue, I found out that this problem is related to idle timeout duration annotation in the kubernetes ingress configuration. My timeout was set to 120 seconds which wasn't enough for calculation and the calculation was taking more time to complete.

To resolve this issue, I updated the following annotation in the ingress.
alb.ingress.kubernetes.io/load-balancer-attributes: idle_timeout.timeout_seconds=300.

To update in Windows, I performed the following steps:

1. kubectl get ingress (Get the current ingress)

PS C:\aks> kubectl get ingress -n my-namespace
NAME       CLASS  HOSTS ADDRESS        PORTS  AGE
my-ingress <none>   *    52.152.91.154  80    10m

2. Edit the ingress configuration using edit command
   kubectl edit ingress my-ingress -n my-namespace (It will open a new file in a notepad)

PS C:\aks> kubectl edit ingress my-ingress -n my-namespace
ingress.networking.k8s.io/my-ingress edited

In the notepad , you can go-ahead and modify the service name and save it.
Once you see this message ingress.networking.k8s.io/my-ingress edited, this means the changes have been applied.

I took help from following articles and wanna share here for the readers

• https://repost.aws/knowledge-center/eks-http-504-errors
• https://learn.microsoft.com/en-us/answers/questions/1031687/how-to-edit-an-existing-ingress-nginx-rule-in-azur ( This is Azure example but its applicable for EKS and GKE also)