AWS ECS Task fails to provision due to EMPTY CAPACITY PROVIDER

Question

I'm trying to run an ECS task in AWS using the EC2 launch type. This task has been running for almost a year without problems, but a few weeks ago it stopped working.

When I create a new task, it remains in "provisioning" until it fails. The error I'm getting is "TaskFailedToStart: EMPTY CAPACITY PROVIDER". I have been looking for answers but so far I have found nothing useful. Can anyone help with some ideas of what may be happening?

This is the autoscalling group configuration: autoscalling config

Answer 1

I faced the same issue. In my case the ACL rules attached to the related subnets were too restrictive and somehow EC2, ASG and ECS could not communicated with each other.

Please try - just for debugging - to open the inbound & outbound rules for all. See https://docs.aws.amazon.com/vpc/latest/userguide/vpc-network-acls.html#default-network-acl

If it works with wide open ACL rules, then you know the root cause and can work on this issue.

Answer 2

When you create ECS cluster, AWS creates a Cloudformation stack with a lot of resources:

• cluster itself (AWS::ECS::Cluster)
• capacity provider (AWS::ECS::CapacityProvider)
• cluster-to-capacity-provider-association (AWS::ECS::ClusterCapacityProviderAssociations)
• autoscaling group
• launch template and so on... then when you delete/recreate something it stops working (or you'll find that when you create a new capacity provider inside cluster, it is created (visible thru AWS CLI), but does not appear in the list. And then you cannot create a new capacity provider because the name is already taken. And you cannot find it anywhere in AWS Console as there is no view to list capacity providers (looks like they are shown inside cluster only). BUGGY BUGGY architecture.

Ideally delete the cloudformation stack & create everth yourself:

1. create launch template(s)
2. create autoscaling group(s) (ASG) - if you run on EC2 and not Fargate
3. finally create cluster

you would be able to switch ASGs and launch templates inside ASG without much hassle. do not know why AWS in some cases uses Cloudformation stacks and in some does not, and does not inform user about it. Then after some time you discover all those abandoned stacks and have no clue why they were created.