How to list all the contents in a given key on Amazon S3 using Apache Libcloud?

Question

The code to list contents in S3 using boto3 is known:

self.s3_client = boto3.client(
            u's3', 
            aws_access_key_id=config.AWS_ACCESS_KEY_ID, 
            aws_secret_access_key=config.AWS_SECRET_ACCESS_KEY, 
            region_name=config.region_name, 
            config=Config(signature_version='s3v4')
            )
        versions = self.s3_client.list_objects(Bucket=self.bucket_name, Prefix=self.package_s3_version_key)

However, I need to list contents on S3 using libcloud. I could not find it in the documentation.

Answer 1

If you are just looking for all the contents for a specific bucket:

from libcloud.storage.types import Provider
from libcloud.storage.providers import get_driver

client = driver(StoreProvider.S3)
s3 = client(aws_id, aws_secret)

container = s3.get_container(container_name='name')
objects = s3.list_container_objects(container)

s3.download_object(objects[0], '/path/to/download')

The resulting objects will contain a list of all the keys in that bucket with filename, byte size, and metadata. To download call the download_object method on s3 with the full libcloud Object and your file path.

If you'd rather get all objects of all buckets, change get_container to list_containers with no parameters.

Information for all driver methods: https://libcloud.readthedocs.io/en/latest/storage/api.html
Short examples specific to s3: https://libcloud.readthedocs.io/en/latest/storage/drivers/s3.html