I am programming a project in python 3.7. I use Great Expectations to verify some expectation and validation on my datasets. I am running the project on a containerized environment where I don't have access to file system. Moreover, I have access to S3 as permanent memory. I need to store my validation artifacts, what is the best way here? I have seen that great expectations has a CloudDataContext but I have not found a good document or example. Does anyone here know how I can use it? I have (an endpoint_url for my s3 too)
I found this code on the internet but it does not work too
context_config = {
"store_backend": {
"class_name": "TupleS3StoreBackend",
"bucket": "mybucket",
"prefix": "myprefix",
"protocol": "s3",
},
"store_backend_options": {
"aws_access_key_id": "YOUR_ACCESS_KEY_ID",
"aws_secret_access_key": "YOUR_SECRET_ACCESS_KEY",
"endpoint_url": "YOUR_S3_ENDPOINT_URL",
}
}
ured storage backend
data_context = DataContext(context_config)