I am currently running sidekiq as a container in ECS.
What I have currently is that I upload a custom metric to cloudwatch which indicates the number of jobs in the queue (which I collect through a lambda function periodically) and then I've set a target tracking policy with a target value of 100 jobs to track the custom metric.
While the target tracking is working great to scale up and down the nr of ECS tasks, I can't make the ECS service have a minimum number of tasks to 0 since if i do that there will be jobs stuck in queue which will not be processed until there are more than 100 jobs in the queue.
The current target tracking policy is as follows:
AutoScalingDefaultWorkerTarget:
Type: AWS::ApplicationAutoScaling::ScalableTarget
Properties:
MinCapacity: 1
MaxCapacity: 8
ResourceId: !Join
- "/"
- - "service"
- !Ref ECSCluster
- !GetAtt DefaultWorkerService.Name
ScalableDimension: ecs:service:DesiredCount
ServiceNamespace: ecs
RoleARN:
Fn::Sub: "arn:aws:iam::${AWS::AccountId}:role/aws-service-role/ecs.application-autoscaling.amazonaws.com/AWSServiceRoleForApplicationAutoScaling_ECSService"
AutoScalingDefaultWorkerTargetTrackingPolicy:
Type: AWS::ApplicationAutoScaling::ScalingPolicy
Properties:
PolicyName: default-worker-target-tracking-scaling-policy
PolicyType: TargetTrackingScaling
ScalingTargetId: !Ref AutoScalingDefaultWorkerTarget
TargetTrackingScalingPolicyConfiguration:
TargetValue: 50
ScaleInCooldown: 10
ScaleOutCooldown: 150
CustomizedMetricSpecification:
MetricName: default-worker-nr-jobs-avg
Namespace: App-Queues
Statistic: Average
Is there a way to modify the target tracking policy so that even if 1 job is in the queue and the ECS service currently has 0 tasks running it will scale up?
I don't think I can use AWS Batch since i believe that won't be compatible with sidekiq since sidekiq is by nature a long running process and not something that finishes.