Using aiohttp with requests_aws4auth

Question

I've asked a previous version of this, but I've progressed much since then.

I've been told that I need to get the authorization headers from AWS4auth and add them as headers into the aiohttp session calls. I'm not sure how to do that.

Currently I have:

    self.aws4auth = AWS4Auth(access_key, secret_key, 'us-east-1', 's3')
    with open(file_path, 'rb') as file:
        content = file.read()
  
    url = f'{self.host}/{bucket}/{object_name}'

    async with aiohttp.ClientSession() as session:
        for k,w in self.aws4auth.signing_key.__dict__.items():
            session.headers[k] = str(w)

        # pprint(list(session.headers.items()))
        async with session.put(url,  data=content) as resp:
            print(resp)
            pass

It gives me a 400 error

I've also tried putting the headers into the session.put with the same result.

When I use the AWS4auth directly with a syncronous request, it works fine, so the data itself isn't the problem.

ETA:

I've grabbed the headers out of a successful get call from request, and it gives me a 403, presumably because I've already used some part of the authentication...

current code:

    url = f'{self.host}/{bucket}/{object_name}'

    listurl = f'{self.host}/{bucket}'
    listresp = requests.request("GET", listurl, auth=self.aws4auth)

    headers= listresp.request.headers

    async with aiohttp.ClientSession(headers=headers) as session:
        async with session.put(url,  data=content) as resp:
            print(resp.text)
            pass

the listrequest gives a 200 response.

I should add that the entire purpose of this is to not have to re-implement: https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html if I don't have to.

ETA: I don't believe that this is currently possible with these tools, at least not without re-implementing the entire signing process above.... I'm looking at other tools.

Answer 1

The answer appears to be "Why are you trying to use aiohttp in the first place?"

If I use my previously working synchronous upload method:

def upload_file(self, file_path, object_name, bucket=''):
    with open(file_path, 'rb') as file:
        content = file.read()
    if not bucket:
        bucket=self.bucket_name

    url = f'{self.host}/{bucket}/{object_name}'
    response = requests.put(
        url=url,
        auth=self.aws4auth,
        data=content)

    return response.status_code, response.text

with this asyncio method:

async def upload_files(self, file_dict, bucket=''):
    coroutines = []
    for on, fn in file_dict.items():
        coroutines.append( asyncio.to_thread(self.upload_file, fn, on, bucket))
    await  asyncio.gather(*coroutines)

called with:

asyncio.run(objstore.upload_files(files))

it appears to work fine, and not need anything from the aiohttp library.