Databricks documentation says that autoloader can ingest millions of files per second from cloud storage using event notification mode. However, when I enable event notification mode, speed is at best reaching 300 files pers second. All files are quite small 10-50KB in size.
Have tried both directory listing and file notifications. Both show same speeds. I suspect file notification mode is not working properly. Even though I can see that it has setup the event grid and queue storage in azure automatically after I provided required access / permissions