In my usecase all google related app and ads data generation is going to store in google store.but my processing engine runs on Spark on AWS cloud. can some one please help how i can move this GS data S3 to process.
Thank You in advance
In my usecase all google related app and ads data generation is going to store in google store.but my processing engine runs on Spark on AWS cloud. can some one please help how i can move this GS data S3 to process.
Thank You in advance
Copyright © 2021 Jogjafile Inc.
If you have the google storage lib on your spark classpath, your EMR code just uses gs:// references for remote access for gcs cloud data. With the right credentials it's accessible from anywhere, including EMR.
You will run up bills though, and have to wait for slower reads and writes.