I am using aiohttp library in Python to make async requests. I use the following code:
async def get_response(session, url):
async with session.get(url) as resp:
response_json = await resp.json()
return response_json
async with aiohttp.ClientSession() as session:
for round in num_rounds:
tasks = []
for i in range(1000):
url = "some_url[i]"
tasks.append(asyncio.ensure_future(get_response(session, url)))
output = await asyncio.gather(*tasks)
So for num_rounds I am doing pretty the same stuff: making 1000 async requests.
Problem: after 10-15 rounds aiohttp gets SUPER slow. For example rounds 1,2,3,... were completed in 10-15 seconds per round, when rounds after 15th take more than 200 seconds per round... What could be the reason of this behavior and what could be the fixes?
I was trying several things:
- It is not the network issue, since I was trying to run this code within different networks and the behavior is the same everywhere.
- It is not likely to be the server issue (i.e., slow down from the server side), because if I will hard stop the code and immediately run it again the timings will be the same (fast at the beginning, super slow further)
- I tried to move
async with aiohttp.ClientSession() as session:line inside thefor round in num_rounds:loop, but surprisingly the behavior is the same (?!)
Would appreciate any help on how to fix this slowdowns...