Python - ThreadPoolExecutor Multithreading works like Single Treading

23 Views Asked by At

I'm working on developing a python script that sends multiple URL requests concurrently. I use concurrent.futures ThreadPoolExecutor class to multithread.

However, seems like my computer doesn't allow sending concurrent URL requests as the requests worked like are sent back-to-back and not sent concurrently.

Believe the root cause may be due to some system settings that restrict my computer from sending multiple URL requests at the same time. Please assist is you've come to this issue before.

Thank you.

Here's my simple code depicted from GFG [https://www.geeksforgeeks.org/how-to-use-threadpoolexecutor-in-python3/]

import requests
import time
import concurrent.futures

img_urls = [
    'https://media.geeksforgeeks.org/wp-content/uploads/20190623210949/download21.jpg',
    'https://media.geeksforgeeks.org/wp-content/uploads/20190623211125/d11.jpg',
    'https://media.geeksforgeeks.org/wp-content/uploads/20190623211655/d31.jpg',
    'https://media.geeksforgeeks.org/wp-content/uploads/20190623212213/d4.jpg',
    'https://media.geeksforgeeks.org/wp-content/uploads/20190623212607/d5.jpg',
    'https://media.geeksforgeeks.org/wp-content/uploads/20190623235904/d6.jpg',
]

t1 = time.perf_counter()


def download_image(img_url):
    img_bytes = requests.get(img_url).content
    print("Downloading..")


# Download images 1 by 1 => slow
for img in img_urls:
    download_image(img)
t2 = time.perf_counter()
print(f'Single Threaded Code Took :{t2 - t1} seconds')

print('*' * 50)

t1 = time.perf_counter()


def download_image(img_url):
    img_bytes = requests.get(img_url).content
    print("Downloading..")


# Fetching images concurrently thus speeds up the download.
with concurrent.futures.ThreadPoolExecutor(8) as executor:
    executor.map(download_image, img_urls)

t2 = time.perf_counter()
print(f'MultiThreaded Code Took:{t2 - t1} seconds')

Here is the output result I get:

Downloading..
Downloading..
Downloading..
Downloading..
Downloading..
Downloading..
Single Threaded Code Took :26.32503480000014 seconds
**************************************************
Downloading..
Downloading..
Downloading..
Downloading..
Downloading..
Downloading..
MultiThreaded Code Took:24.53087719999894 seconds

If multithreaded correctly, it should take around 26/6=5 seconds to download all 6 images.

In other case, I've tested out using ThreadPoolExecutor to run a local simple task that doesn't require sending URL request. That test showed the multithreading was working as expected. The problem is that URLs are not sent concurrently.

0

There are 0 best solutions below