I have a Google Task queue set up with the following configuration:
Max rate: 500/s
Max concurrent: 1000
Max burst size: 100
So theoretically it should be able to handle 1000 tasks concurrently. The task that is being run takes about ~10 seconds on average and uses about ~400mb of memory in cloud functions.
My queue generally fills up to about ~300 tasks to process, and I would expect them to process the 300 concurrently, however the queue handles at most 25 at a time, and it builds up to that number. Generally only adding 1-2 per second to that processing tasks list.
What do I seem to be missing in terms of concurrency? Am I misunderstanding the capabilities of google tasks, or is there more configurations that could improve the concurrency capability of my queue?
Google Cloud Tasks is just an orchestrator for work items, doing things like putting limits into place in order to avoid overloading the resources that actually do the work, and performing retries on work that doesn't complete successfully. On its own, it doesn't do much other than delegate the work items as it's configured.
It sounds like the thing doing the actual work for you is Cloud Functions, which has its own configurations for the maximum load it can handle. You should look into that first to understand where things might be limited. For example, the max number of instances could put a cap on how much work can go through the system at any given time.