How to make asynchronous requests using reqwest?

Question

I am attempting to reimplement the following Python code using Rust's tokio and reqwest crates.

#!/usr/bin/env python3

import asyncio
import httpx
import time

async def async_req(url, client):
    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36 Edg/111.0.1661.54",
        "Cookie": ""
    }
    response = await client.get(url, headers=headers, timeout=180)
    print(response.status_code)
    print(response.text)

async def main():
    urls = ["http://localhost:5000/hello", "http://localhost:5000/world"]
    start = time.time()
    async with httpx.AsyncClient() as client:
        tasks = [async_req(url, client) for url in urls]
        await asyncio.gather(*tasks)
    print(time.time()-start)

if __name__ == "__main__":
    asyncio.run(main())

The code implemented in Rust is as follows:

async fn async_req(
    url: &str,
    client: reqwest::Client,
) -> Result<reqwest::Response, Box<dyn std::error::Error>> {
    let response = client
        .get(url)
        .timeout(std::time::Duration::from_secs(180))
        .send()
        .await?;

    Ok(response)
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let urls = vec![
        "http://localhost:5000/hello",
        "http://localhost:5000/world",
    ];

    let mut handles = vec![];

    let client = reqwest::Client::new();
    for url in urls {
        let handle = {
            let client = client.clone();
            async move { async_req(url, client).await }
        };
        handles.push(handle);
    }

    let mut responses = vec![];
    for handle in handles {
        responses.push(handle.await?);
    }

    for response in responses {
        println!("{}", response.text().await?);
    }

    Ok(())
}

But it seems that the program is not making asynchronous requests as it does in Python.

I have not yet found a solution that doesn't produce errors. The configuration of Cargo.toml is as follows, and the Rust compiler version I am using is 1.72.1 (d5c2e9c34, 2023-09-13).

[package]
name = "reqwest_async_demo"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
reqwest = { version = "^0.11" }
tokio = { version = "^1", features = [ "full" ] }

Answer 1

You can use tokio::try_join! or futures::future::join_all to concurrently run the futures. These methods return a future that completes when all the futures have completed, and produces a collection of all their results. This is similar to the behavior of asyncio.gather in Python.

First, add the futures crate to your Cargo.toml:

[dependencies]
futures = "0.3"

main function:

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let urls = vec![
        "http://localhost:5000/hello",
        "http://localhost:5000/world",
    ];

    let client = reqwest::Client::new();
    let mut handles = Vec::new();

    for url in urls {
        let client = client.clone();
        let handle = tokio::spawn(async move { async_req(url, client).await });
        handles.push(handle);
    }

    let responses = futures::future::join_all(handles).await;

    for response in responses {
        match response {
            Ok(Ok(res)) => println!("{}", res.text().await?),
            Ok(Err(e)) => eprintln!("Error: {}", e),
            Err(e) => eprintln!("Error: {}", e),
        }
    }

    Ok(())
}

tokio::spawn is used to start each future concurrently in a new task, and then futures::future::join_all is used to await all tasks. The join_all function returns a future that resolves to a Vec of the results of the futures passed to it.

async fn async_req(
    url: &str,
    client: reqwest::Client,
) -> Result<reqwest::Response, reqwest::Error> {
    let response = client
        .get(url)
        .timeout(std::time::Duration::from_secs(180))
        .send()
        .await?;

    Ok(response)
}