I plan to move data from a number of databases periodically using Azure Data Factory (ADF) and i want to move the data into Azure Parallel Data-Warehouse (APDW). However the 'destination' step in the ADF wizard offers me 2 functions; 1- in the case where data is retrieved from a view you are expected to map the columns to an existing table, and 2- when the data comes from a table you are expected to generate a table object in the APDW.
Realistically this is too expensive to maintain and it is possible to erroneously map source data to a landing zone.
What i would like to achieve is an algorithmic approach using variables to name schemas, customer codes and tables.
After the source data has landed i will be transforming it using our SSIS Integration Runtime. I am wondering also whether a SSIS package could request source data instead of an ADF pipeline.
- Are there any resources about connecting to on premises IRs through SSIS objects?
- Can the JSON of an ADF be modified to dynamically generate a schema for each data source?
For your question #2, Can the JSON of an ADF be modified to dynamically generate a schema for each data source:
You could put your generate table script in precopyscript.