Using autoloader to ingest files from adls gen2. However, I want to ingest only new files. Using the following config is still not preventing existing files from getting ingested. Is anyone else facing the same issue?
"cloudFiles.format": "json",
"multiline": True,
"pathGlobFilter": f"*.json",
"includeExistingFiles": "false",
"cloudFiles.schemaLocation": inferred_schema_folder,
"cloudFiles.schemaEvolutionMode": "addNewColumns",
"cloudFiles.inferColumnTypes": "true",
"cloudFiles.rescuedDataColumn": "unexpected_data",
"cloudFiles.maxFilesPerTrigger": 10000,
Tried switching to file notification mode instead of directory listing. But that too didnt work
"cloudFiles.useNotifications": "true",
"cloudFiles.subscriptionId": config.AZURE_SUB_ID,
"cloudFiles.tenantId": config.AZURE_TENANT_ID,
"cloudFiles.clientId": config.AZURE_CLIENT_ID,
"cloudFiles.clientSecret": config.AZURE_CLIENT_SECRET,
"cloudFiles.resourceGroup": "Development",