is there any google/aws services to move data from google store to s3

270 Views Asked by At

In my usecase all google related app and ads data generation is going to store in google store.but my processing engine runs on Spark on AWS cloud. can some one please help how i can move this GS data S3 to process.

Thank You in advance

1

There are 1 best solutions below

0
On

If you have the google storage lib on your spark classpath, your EMR code just uses gs:// references for remote access for gcs cloud data. With the right credentials it's accessible from anywhere, including EMR.

You will run up bills though, and have to wait for slower reads and writes.