I'm working on a project with a FastAPI backend that interacts with Azure Form Recognizer. Using sqlite and planning using postgres in the future. The project's primary purpose is to use the results from the service to do some analysis and calculations.
When a client creates a job, the server adds it to the job table in the database, along with its status, which can be 'Pending,' 'Processing,' 'Complete,' or 'Abort.'
For the job consumer component, I've wrote a separate program that continuously checks the database to see whether there are any pending job. However, I'm cannot come up with a good approach for the job stopping mechanism.
My concept is that when a client sends an abort request, the status should be set to 'Abort,' and the job consumer should terminate the current processing job. However, continuous database polling approach is resource-intensive, and checking every 5 seconds doesn't seem to be a suitable solution either.
Any suggestions? Or is there anything I can improve for my system?