EC2 instance being terminated after setting up lifecycle hooks

30 Views Asked by At

I'm using the AWS CDK to add lifecycle hooks to my ALB in order to be notified by email when an instance gets terminated or when a new one is started. This is what my code looks like:

topic = sns.Topic(my_stack, "fleet_changes")
topic_hook = autoscaling_hooktargets.TopicHook(topic)

asg.add_lifecycle_hook(
    'instance-terminating-hook',
    lifecycle_transition=autoscaling.LifecycleTransition.INSTANCE_TERMINATING,
    lifecycle_hook_name="instance-terminating",
    notification_target=topic_hook,
    notification_metadata="INFO: An instance has been terminated"
)

asg.add_lifecycle_hook(
    'instance-launching-hook',
    lifecycle_transition=autoscaling.LifecycleTransition.INSTANCE_LAUNCHING,
    lifecycle_hook_name="instance-launching",
    notification_target=topic_hook,
    notification_metadata="INFO: A new instance has been launched"
)

This works fine as I'm indeed getting notifications when an instance is terminated or launched. The problem is that the instances are terminated every hour, and this stops happening when I remove the lifecycle hooks. This is what I see in the ALB events when the hooks are set:

At 2024-03-22T17:11:34Z an instance was taken out of service in response to a launch failure.

And right after:

At 2024-03-22T17:12:35Z an instance was started in response to a difference between desired and actual capacity, increasing the capacity from 1 to 2.   

I've checked the application logs in Cloudwatch and haven't found anything that would indicate a bug, and there is currently little to no activity on these instances.

Any idea what could be the culprit?

Thank you.

0

There are 0 best solutions below