I'm working on a project where I receive messages from various pieces(300+) of equipment. I'm aiming to establish a system that allows me to create a task queue using these messages (each message results in a single task), with multiple workers processing tasks from this queue, with these limitations • tasks should be picked up FIFO • tasks from the same equipment can’t run simultaneously • tasks that are not finished in time (e.g. worker crashes) should remain on the queue without violating FIFO, to be picked up by another worker • no correlation between tasks from different equipment
I’ve looked at Redis and/or RabbitMQ for locking/queueing functionality, but can’t figure out how to use those to implement this complete behavior (I stored tasks in a list for FIFO logic, so new messages come in from the right and workers start processing from the left. I implemented basic locking mechanisms to prevent multiple workers from processing tasks for the same equipment simultaneously using a lock 'key' with a TTL (in case a worker should crash). While Redis worked to a certain extent, maintaining chronological order for tasks of the same equipment became more complex than I initially anticipated). Before I dive into coding a custom solution, I'm wondering if anyone in the community has tackled a similar challenge. I'm particularly interested in exploring elegant solutions that make use of existing application functionality rather than building a custom message queuing system. Has anyone faced a comparable scenario? Any advice, insights, or recommended tools/approaches would be greatly appreciated. Thank you in advance for your help