Copy large data from one cloud storage bucket to another with CSEK

Question

I want to copy data from one GCP cloud storage bucket to another GCP cloud storage bucket.

Source bucket has million of small files and are encrypted with AES256 keys(i.e. customer supplied encryption keys).

customerEncryption:
  encryptionAlgorithm: AES256
  keySha256: pLLJNJ+/XXXXXXXXXXXXXXXXXXXXX/ckKJNK=

GCP Data Transfer service doesn't cover data transfer for files encrypted using CSEK.

Currently I can think of only one way to solve this i.e. read the object programmatically and upload again with encryption key to destination bucket.

Is there an efficient way to copy large number of files encrypted using Customer Supplied Encryption Keys ?

Answer 1

We can copy files encrypted with CSEK in a GCP bucket to another GCP bucket with following commands

gsutil cp

gsutil -m -o "GSUtil:encryption_key=XXXXXXXXXXXXXXXX" cp gs://test-1/abc gs://test-3/abc

gsutil rsync

gsutil -m -o "GSUtil:encryption_key=XXXXXXXXXXXXXXXX" rsync gs://test-1/abc gs://test-3/abc

gcloud storage cp

GCP Introduced gcloud storage: up to 94% faster data transfers for Cloud Storage

gcloud storage cp -r gs://test-1/abc gs://test-3/abc --encryption-key=XXXXXXXXXXXXXXXX --manifest-path=test-manifest.csv --storage-class=REGIONAL