How to limit max concurrency for SQS queue receivers based on each type of message?

Question

I lambda function that returns a bunch of requests that I want to send to an external service. This external service has an api with the following rate limit:

Paramters: 
type: enum
content: list of n elements

Rate limit: 
For each type, there can be no more than 3 concurrent slots. Each slot will be will need to wait n milliseconds between requests. 

I want to throttle the requests being sent. My deign is to use sqs to queue the request payloads and set an autoscaling lambda to consume them. The sqs configuration includes a max_concurrency, but I dont see a way to configure that based on the message content.

Is there a way to implement the rate limit throttling described above using SQS? Or should I use another solution like Amazon MSK?

Example:

For below queue, I want a lambda function to be able to concurrently process 3 messages for type 1, 3 messages for type 2, and so on.

queue: 
type=1, payload
type=1, payload
type=1, payload
type=1, payload
type=2, payload
type=2, payload
type=2, payload
type=3, payload

Answer 1

If you want to process messages differently, you would need to:

• Send the messages to an Amazon SNS Topic
• Subscribe multiple AWS Lambda functions to the SNS Topic
• Add Amazon SNS subscription filter policies - Amazon Simple Notification Service to the subscriptions such that each subscribed Lambda function only receives one 'type' of message
  OR
• Use Lambda event filtering - AWS Lambda to filter the messages in the Lambda function's event source definition

Answer 2

You would need to:

• Configure the AWS Lambda function to have Reserved Concurrency = 3
• Add a sleep command to the end of your Lambda function to wait the required time

The Reserved Concurrency - AWS Lambda will ensure that a maximum of 3 functions can run at the same time.

The sleep will delay the start of the next function. Unfortunately, it means that the Lambda function will be running for a longer time and you will be paying for the Lambda function to effectively do nothing.

Answer 3

You probably could build a eventbridge pipe where you pass the SQS records into a stepfunction that redirects each type to a lambda task. The task always uses the same lambda. When you build a map task around it (if you have batches of the same type) you can process more by type in parallel. Its kinda overkill for the example but maybe you need to reconsider your use case and why you need that strict isolated parallelism. It feels what you need is more or less something stateful and with lambda this does not work so well with you parallelism requirements.

Note: use a express stepfunction otherwise it might be expensive.