How is batching done in FIFO SQS queues?

Question

Let's say I have a FIFO SQS, and a lambda which consumes a batch of messages from the FIFO SQS. The max limit on the size of this batch is 10, as mentioned here. I was wondering how this would work.

Let's say we have some message group ID G, because of which we have 5 messages groups - G1, G2, ..., G5.

Let's say we have set the batch size as 10. So that would mean at a time, more than one message would be required to be picked up from each group. So let's say we have two messages M1 and M2 inside G1. M1 came first, and M2 came afterwards.

But since we are getting both of these messages in the same batch, they might get processed out of order.

Is this the correct expectation? Or would the FIFO queue only put 5 messages in the batch, and send that to the lambda?

Answer 1

The batch will be supplied to the AWS Lambda function in order.

Let's say that A/B/C represents the Group and the number represents the message number.

Let's assume that the messages were sent in this order: A1 B1 A2 C1 A3 D1 C2 B2 A4 D2 C3 B3

If the batch size is set to 10, then the Lambda function would receive the first 10 messages:

A1 B1 A2 C1 A3 D1 C2 B2 A4 D2

They would be provided in the batch in that order so your code should process the messages in the order given.

While the batch contains multiple entries for Group A, they remain in the order that the messages were sent.

Even though there are extra messages waiting, no other Lambda functions will be triggered because there are unprocessed messages in each Group. However, if a message E1 was sent, then a Lambda function would be triggered with E1 since it can be processed at any time without waiting for earlier messages to be processed.

Once the initial Lambda function has finished running, the function will again be invoked with C3 B3 since the earlier messages in those groups have now been processed.

Answer 2

When there are batch sizes greater than 1 and multiple consumers, SQS will avoid creating a race-condition by holding back queued messages from the message groups that have messages in flight.

This means the following:

If Group A and Group B have messages queued,
And Group A and Group B have messages in flight,
And Consumer A receives a batch of 10 messages from Group A and Group B,
And Consumer B receives a batch of 10 messages from Group B,
And Consumer A finishes its batch of messages from Group A and Group B,
And Consumer B is still processing its batch of messages from Consumer B,
Then then the next batch of messages that Consumer A receives will not 
contain messages from Group B.

Explained in detail here: https://aws.amazon.com/blogs/compute/solving-complex-ordering-challenges-with-amazon-sqs-fifo-queues/

Side note: Setting batch size to 1 would eventually create a bottleneck.