In the Dynamo table that holds the leases and checkpoint information, I can see that workers have an uneven distribution of shards. In my case for example - for 256 shards and 41 workers (pods in Kubernetes), the majority have ~6-8 shards assigned, but I also see ~5 workers with only 1-2 shards.
How can I make the distribution more evenly? is there some configuration I can change for it?
This causes workers to not fully utilize their resources (the workers with the 1-2 shards barely use any CPU and I would rather have another worker with ~6-8 shards than 5 workers with 1-2 shards each).
It is worth mentioning I am using the amazon-kinesis-client-python library.