I am trying to communicate between two processes. process 1 : collects artifacts links that I have to download by querying a database every 15 mins. process 2 : should take these links and then download each of them in some specific folder. However, to download the artifacts, it takes a lot of time so the process 1 might be collecting a lot of links per hour and this may download only few.
How should I share the information between these two processes? Until now , only thing I can think of is to maintain a file and lock it using fcntl.flock. The process 2(downloading process) can take each link and download it and repeat the process.
Do you have any better solutions ? please let me know!