I'm using CloudBerry Explorer to copy files from one bucket to another bucket. both buckets are under the same project on Google Cloud Platform. I can easily copy small size files but when trying to copy 1GB+ files from the same path to the other, I get an error:
"The remote server returned an error: (500) Internal Server Error.
Permissions are correct since moving small size files is fine.
What could be the issue?
CloudBerry Log:
2020-12-09 14:23:52,298 [Google] [9] INFO - InternalGetObjectExists start, bucket: DEDUCTED-BUCKET, key DEDUCTED-FILENAME.mp4
2020-12-09 14:23:52,781 [Google] [9] INFO - InternalGetObjectExists ready, bucket: DEDUCTED-BUCKET, key DEDUCTED-FILENAME.mp4, status: 404
2020-12-09 14:23:52,782 [Google] [9] INFO - InternalGetObjectExists start, bucket: DEDUCTED-BUCKET, key DEDUCTED-PATH/DEDUCTED-FILENAME.mp4
2020-12-09 14:23:52,992 [Google] [9] INFO - InternalGetObjectExists ready, bucket: DEDUCTED-BUCKET, key DEDUCTED-PATH/DEDUCTED-FILENAME.mp4, status: 200
2020-12-09 14:23:52,992 [Google] [9] INFO - InternalCopyObjectNonChunked start, src bucket: DEDUCTED-BUCKET, src key: DEDUCTED-PATH/DEDUCTED-FILENAME.mp4, dst bucket: DEDUCTED-BUCKET, dst key: DEDUCTED-FILENAME.mp4
2020-12-09 14:25:17,510 [Google] [9] INFO - InternalCopyObjectNonChunked ready, src bucket: DEDUCTED-BUCKET, src key: DEDUCTED-PATH/DEDUCTED-FILENAME.mp4, dst bucket: DEDUCTED-BUCKET, dst key: DEDUCTED-FILENAME.mp4, status: 500
2020-12-09 14:25:17,510 [Google] [9] ERROR - Http response status: 500: Internal Server Error
2020-12-09 14:25:17,511 [Google] [9] ERROR - <?xml version='1.0' encoding='UTF-8'?><Error><Code>InternalError</Code><Message>We encountered an internal error. Please try again.</Message><Details>AFfi+BFP7TevpPSJUbrZ2g1VXu3cm/uZHLEPxDMKeQro1D9stEU/xcIHUePCm9meA3u2N+gGUX41/UHUd1WIDNA+tOjFrbNRZsC2T8twlFOBURzaHcO5GfU=</Details></Error>
2020-12-09 14:25:17,511 [Google] [9] ERROR - InternalCopyObjectNonChunked failed. src bucket: DEDUCTED-BUCKET, src key: DEDUCTED-PATH/DEDUCTED-FILENAME.mp4, dst bucket: DEDUCTED-BUCKET, dst key: DEDUCTED-FILENAME.mp4
CloudBerryLab.Base.Exceptions.Status500Exception
The remote server returned an error: (500) Internal Server Error.
System.Net.WebException
The remote server returned an error: (500) Internal Server Error.
at System.Net.HttpWebRequest.GetResponse()
at gZ.A(HB , Action1` , HttpWebRequest , HS )
2020-12-09 14:25:17,513 [CL] [9] ERROR - Command::Run failed:
Copy; Source:DEDUCTED-PATH/DEDUCTED-FILENAME.mp4; Destination:/DEDUCTED-BUCKET/
CloudBerryLab.Base.Exceptions.Status500Exception: The remote server returned an error: (500) Internal Server Error. ---> System.Net.WebException: The remote server returned an error: (500) Internal Server Error.
at System.Net.HttpWebRequest.GetResponse()
at gZ.A(HB , Action1` , HttpWebRequest , HS )
--- End of inner exception stack trace ---
at Tm.a(String , String , String , String , String , MetadataDirective , Gw , Jt )
at Tm.A(String , String , String , String , String , String , Gw )
at sU.A(Tm , sU , Gw , lD )
at sU.BV(lL , String , lD )
at lF.RunInternal()
at SE.RunInternal()
at lE.fa()
I encountered the same issue recently. Initially, the issue was reported with a Talend job using the tGCSCopy component failing with a 500 error. It was also experienced with Cloudberry with manual testing. Works fine with small files, but larger ones are where the issue happens. Turns out the issue is related to bucket to bucket copies where the target bucket is using a customer managed encryption key for encryption. Standard Google managed encryption does not have the same issue and using Google provided utils like gsutil do not exhibit the same behavior.
I also tested this with a source object encrypted with the same key and didn't make a difference. Talend uses jets3t, an s3 compliant library to interact with Cloud Storage, and I'm assuming Cloudberry is doing something similar that doesn't play nice with this type of scenario.
No resolution as of now, Google is likely going to say don't use 3rd party products.