AWS How to place task on new ASG instance first try, without hitting "No Container Instances were found in your cluster."

50 Views Asked by At

I'm trying to write a boto3 script to spin up a instance, and place a task on it. The problem is if you place the task too soon, you get this error in the console:

service <my-service> was unable to place a task because no container instance met all of its requirements. Reason: No Container Instances were found in your cluster. For more information, see the Troubleshooting section of the Amazon ECS Developer Guide.

It will try again ~1min later and will succeed then, but I'm trying to get it to spin up ASAP. I don't want to wait this minute.
The weird part is if you look at the autoscaling client docs, there's a asg_client.get_waiter method. Looking at the base docs though, no waiters are listed. I'm guessing none actually exist, and it inherited that method from a base class it extends?

Here is a WIP of what I'm trying to accomplish:

import os

import boto3


# Hard coded for now, will pass in as an env var eventually:
ECS_CLUSTER_NAME = "my-ecs-cluster"
ECS_CLUSTER_SERVICE = "my-ecs-service"
ASG_NAME = "my-asg"

# Boto3 Clients:
asg_client = boto3.client('autoscaling')
ecs_client = boto3.client('ecs', region_name=os.environ['AWS_REGION'])

def update_ecs_container(spin_up_container: bool) -> None:
    # If spinning up, first spin up the container, then the service
    # If spinning down, first spin down the service, then the container
    if spin_up_container:
        update_asg(desired_count=1)
        update_ecs_service(desired_count=1)
    else:
        update_ecs_service(desired_count=0)
        update_asg(desired_count=0)



def update_asg(desired_count: int) -> dict:
    # https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/autoscaling.html#AutoScaling.Client.update_auto_scaling_group
    response = asg_client.update_auto_scaling_group(
        AutoScalingGroupName=ASG_NAME,
        DesiredCapacity=desired_count,
    )
    # HERE <----------------
    # How do I wait for the instance to finish spinning up, and be ready for a task to be placed?
    # There's no waiters, like with the ECS Service below
    return response

def update_ecs_service(desired_count: int) -> dict:
    response = ecs_client.update_service(
        cluster=ECS_CLUSTER_NAME,
        service=ECS_CLUSTER_SERVICE,
        desiredCount=desired_count,
    )
    # Now wait for it to be done updating:
    # https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ecs/waiter/ServicesStable.html
    waiter = ecs_client.get_waiter('services_stable')
    waiter.wait(
        cluster=ECS_CLUSTER_NAME,
        services=[ECS_CLUSTER_SERVICE],
        WaiterConfig={
            "Delay": 1,
            "MaxAttempts": 60,
        },
    )
    return response

if __name__ == "__main__":
    update_ecs_container(spin_up_container=True)

I tried using asg_client.describe_auto_scaling_instances() so I could grab the instance ID and wait for that with a ec2 waiter, but it takes a bit to appear. I'd have to write my own waiter to watch for it, and at that point I should just write my own for asg. Is there something I'm missing?

Edit: I just tried using EventBridge, to trigger lambda when the ASG instance sends "EC2 Instance Launch Successful", and that lambda spins up the service. This also hit the same problem of not placing the task the first try. This makes me think it's either impossible, or ASG isn't the right place to be waiting to spin up the task?

0

There are 0 best solutions below