Python project to download files from network devices via TELNET and FTP with multi-tasking

Question

im starting a python project and i need some guidance.

My project consists in periodicly access several devices on my network and download some files from it.

I need to know, how i can accomplish this with this characteristcs:

1. I need to run the program in multi tasking, because i have to access a lot of devices (~300 devices), so i want to do a amount of them in parallel.
2. What i need to download uses Telnet and FTP protocols, thinking in use the "telnetlib" and "ftplib" libraries to accomplish that.
3. I also need someway to do this tasks periodicly, thinking in use "timeloop" library (https://medium.com/greedygame-engineering/an-elegant-way-to-run-periodic-tasks-in-python-61b7c477b679)
4. Would be nice if i can choose which frequency i want to each device, example: device A i want to access it each 1 hour and get the most recent files from it and device B i want to access every 24 hours and do the same.

I was looking and i saw some differences between multi threading and multi processing, dont know if it makes difference to my application.

My question is: Which libraries do you reccomend to accomplish that?

Second question, python allows me to accept external commands, from a website for example and force to start a job on a specific device?

Answer 1

Your questions are not very on-topic for StackOverflow (too broad, or asking for recommendations), but I tried to answer anyway.

The difference between multi-threading and multi-processing is that (in a very broad sense, not to take as a general rule) multi-processing is better for heavy computations (CPU-bound) while multi-threading is simpler for blocking operations (IO-bound).
In your case, both would do.

On Linux, cron is the standard way to schedule tasks. You could create some scripts that take parameters from the command line (sys.argv), and create cron rules to invoke them with the required parameters. That way, all downloads will happen in different processes, which means that the OS will take care of the parallelization. That way, as long as the computer/server is running, the tasks will be run.

If you are dealing with FTP and TelNet, so yeah try to use these ones, check that they do what you want. If your requirement is simply "i want to download a file", I think any FTP lib covers that.

As for your second question, yes you can, in many different ways. It depends on how you intend to send your command. You could SSH into the server running the scripts, and run a specific script there. You could add an HTTP Server that waits for a specific request, then run something ([see for example])(Run python script on my computer after Http Request). Your question is again very broad, so it is difficult to answer precisely.