Python AIOHTTP gets slow after a number of requests

46 Views Asked by At

I am using aiohttp library in Python to make async requests. I use the following code:

async def get_response(session, url):
    async with session.get(url) as resp:
        response_json = await resp.json()
        return response_json


async with aiohttp.ClientSession() as session:
    for round in num_rounds:
        tasks = []
        for i in range(1000):
            url = "some_url[i]"
            tasks.append(asyncio.ensure_future(get_response(session, url)))

        output = await asyncio.gather(*tasks)

So for num_rounds I am doing pretty the same stuff: making 1000 async requests.

Problem: after 10-15 rounds aiohttp gets SUPER slow. For example rounds 1,2,3,... were completed in 10-15 seconds per round, when rounds after 15th take more than 200 seconds per round... What could be the reason of this behavior and what could be the fixes?

I was trying several things:

  1. It is not the network issue, since I was trying to run this code within different networks and the behavior is the same everywhere.
  2. It is not likely to be the server issue (i.e., slow down from the server side), because if I will hard stop the code and immediately run it again the timings will be the same (fast at the beginning, super slow further)
  3. I tried to move async with aiohttp.ClientSession() as session: line inside the for round in num_rounds: loop, but surprisingly the behavior is the same (?!)

Would appreciate any help on how to fix this slowdowns...

0

There are 0 best solutions below