Allowing client to uploading large number of files to cloud storage bucket

Question

I have a React web application in which I allow users to upload DICOM files to Google Healthcare API. The current implementation is that the files first gets uploaded to my back-end server which uploads them to Healthcare API. I am allowing users to upload a full DICOM study (100MB - 2+GB) which could have anywhere from 1-500+ DICOM files (each usually 50KB-50MB). Our current approach as worked thus far but as we are expanding, it seems insufficient use of my server.

My goal is to allow user to directly upload to Google Cloud Storage bucket from the React app. I want to perform some validation logic before I export it to Google Healthcare API. I have looked into signed urls but since the files being uploaded are medical images I wasn't sure if they would be secure enough. The users don't necessarily have a google account.

What is the best way I can allow user to directly upload a directory to GCS bucket without going through my server? Are there dangers involved with this approach if the user uploaded a virus? Also signed urls are valid for a set amount of time, can I deactivate a signed url as soon the uploads are complete?

Answer 1

I have used Transloadit for a similar use cases in the past and I had a good experience. For your particular case, use: https://transloadit.com/demos/file-exporting/export-to-google-store/.

This would scale on-demand for you and it also allows you to build a pipeline to perform any pre-processing you'd like, such as virus scanning as you were concerned.

Their full list of pre-defined processing services: https://transloadit.com/services/

They have a pretty nice free tier, but based on your concerns, you'd likely need a premium tier.

Answer 2

A signed URL is valid for a time window you specify but can be used multiple times in that time window. For uploads you could use bucket lock to prevent overwrites. You can't deactivate a signed URL early, but you can remove IAM access of the signing account from the resource to deny access. As you'd use a fixed account for signing it's probably not a viable solution, but it's possible.

For a validation flow check the Google Cloud Blog Uploading images directly to Cloud Storage using Signed URL for considerations and a potential solution.