Kinesis KCL not pulling records from all shards

147 Views Asked by At

I've configured the Kinesis stream with 4 shards and implemented the KCL library version-1 as a consumer. The consumer pulling records from one shard only (shardId-000000000000). I've also noticed if I restart the consumer, It pulls a few records(stale records) from all shards for some time but mostly from single shards.

1st Sep records count(Note - I restarted consumer)
 228830 shardId-000000000000
  98697 shardId-000000000001
    579 shardId-000000000002
    306 shardId-000000000003

2nd Sep records count

163170 Shard: shardId-000000000000

Note- InitialPositionInStream.LATEST configured

1

There are 1 best solutions below

0
ankit.vishen On

I didn't get the exact answer but I think it happens when the KCL-managed dyanamoDB table goes out of sync. I changed the Kinesis client app name(It creates a new table with a new app name) and restarted the consumer; it works fine now.

Please answer if anybody can give more specific details.