AWS ECS Fargate service is getting scaled up despite no autoscaling policies being defined

Question

I have an ECS cluster and service running in AWS. CPU usage is typically around 5-10% but once a month or so it may briefly (for a minute or two) peak to 70%+. I've noticed that when this happens, another instance of the service gets spun up and then the old instance is shut down a couple minutes later. This has caused problems because the service was never designed to run more than 1 instance, and the two concurrent instances compete for resources and cause race conditions while working on the same data. I'm trying to understand why it's scaling and how to configure it not to.

In the console the service says it has 1 desired task and No Auto Scaling resources configured for this service. The cluster has no default capacity provider strategy and no capacity providers specified. I've been going through AWS documentation to try and find where else autoscaling could be defined for a Fargate service, but can't find anything. What am I missing?

Answer 1

Even without autoscaling configured, ECS will replace your task if it becomes unresponsive. It sounds like something is causing the CPU to spike in your application, at which point it becomes unresponsive, leading ECS to think that your application has crashed.

By default, when ECS replaces your task it will first start up another instance of the task and wait for it to become healthy, before terminating the unhealthy task. To change this behavior you need to change the maximumPercent value in the ECS deployment setting to 100, and the minimumHealthyPercent to 0, which will tell ECS that at most you want 100% of your desired task count of 1 running during a deployment. To satisfy this configuration, ECS will first shut down the unresponsive task before starting up the new one.