How do I make different instances of an application to read different files to process?

Question

I have an application that reads a file from a specific location and processes it. Now my requirement is to increase the no.of instances of application. How do I make sure that each instance does not pick the same file? Is there a way to make each instance to pick different file from the same file location? Please suggest.

Answer 1

Depends on few things:

1. How your application is triggered? if you are using something like spring batch scheduler then it's cause of worry as all instances get triggered and multiple instances can pick-up same files. Thus you need to have some logic within app to trigger only 1 instance like make an entry in the table and lock the table when processing begins so another trigger wont run until it finds the table available. Ensure to update and lock the table for one instance only to avoid multiple triggers. There are few other ways you can manage.

2. If you application is triggered by some other process then go-router will trigger only 1 instance at one time on round robin basis however if process takes some time and another instance gets triggered, then it might try picking up the same file. So may be you can move the file to some temp location when you pick up the file for processing and this way another trigger won't find the same file. Once processed successfully - it can be moved to 'done' folder..

3. How soon files gets uploaded to shared location? If its very frequent, best to move in unique temp folders as soon as app instance triggers and keep the staging folder with unpicked files only