I am currently exploring Azure Synapse Analytics, focusing on near-real-time (NRT). I am planning to utilize Delta tables and we have a multi-tenant setup for our product where each tenant's data should be isolated and processed independently.
Here are the specific challenges I'm facing with spark structured streaming in synapse:
I need to run stream processing jobs separately for each tenant. The goal is to ensure that each tenant's data is processed independently to maintain data isolation. (data at rest, in transit and during processing should be isolated)
Data Storage in Tenant-Specific Containers: For data storage, each tenant should have their data stored in a dedicated container (one container per tenant).
How can I configure spark structured streaming using Synapse to run separate stream jobs for each tenant, ensuring that each job only processes data for its respective tenant? Do we have any framework/feature in azure synapse/structured streaming that can help us here?
