We are seeing below error when we migrate to Apache Storm 2.2.0 from 1.2.3. We do not see any permission issues as well on the file which it is unable to find. Also, the file exists but just the /data part is being appended to the actual file name due to which it is unable to find it. We want to know if some config decides this value.
PS: We are using Plaintext auth and do see below logs. But assuming they are not cause of concern.
PlainSaslTransportPlugin [ERROR] SASL PLAIN client transport has been established. This is totally insecure. Please do not use this.
We think File not found is the issue. We want to know if any specific configuration is required.
nimbus.log-2022-11-01 15:18:38.502 o.a.s.c.StormClusterStateImpl [INFO] set-path: /blobstore/all_test-1-1667296118-stormjar.jar/10.32.17.238:6627-1
nimbus.log-2022-11-01 15:18:38.603 o.a.s.d.n.Nimbus [WARN] begin blob download exception.
nimbus.log:java.lang.RuntimeException: java.io.FileNotFoundException: /grid/1/storm/blobs/967/data_all_test-1-1667296118-stormjar.jar/data (No such file or directory)
nimbus.log- at org.apache.storm.blobstore.LocalFsBlobStore.getBlob(LocalFsBlobStore.java:396) ~[storm-server-2.2.0.jar:2.2.0]
nimbus.log- at org.apache.storm.daemon.nimbus.Nimbus.beginBlobDownload(Nimbus.java:3767) [storm-server-2.2.0.jar:2.2.0]
nimbus.log- at org.apache.storm.generated.Nimbus$Processor$beginBlobDownload.getResult(Nimbus.java:4013) [storm-client-2.2.0.jar:2.2.0]
nimbus.log- at org.apache.storm.generated.Nimbus$Processor$beginBlobDownload.getResult(Nimbus.java:3992) [storm-client-2.2.0.jar:2.2.0]
nimbus.log- at org.apache.storm.thrift.ProcessFunction.process(ProcessFunction.java:38) [storm-shaded-deps-2.2.0.jar:2.2.0]
nimbus.log- at org.apache.storm.thrift.TBaseProcessor.process(TBaseProcessor.java:38) [storm-shaded-deps-2.2.0.jar:2.2.0]
nimbus.log- at org.apache.storm.security.auth.sasl.SaslTransportPlugin$TUGIWrapProcessor.process(SaslTransportPlugin.java:152) [storm-client-2.2.0.jar:2.2.0]
nimbus.log- at org.apache.storm.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:313) [storm-shaded-deps-2.2.0.jar:2.2.0]
nimbus.log- at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_05]
nimbus.log- at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_05]
nimbus.log- at java.lang.Thread.run(Thread.java:745) [?:1.8.0_05]
nimbus.log:Caused by: java.io.FileNotFoundException: /grid/1/storm/blobs/967/data_all_test-1-1667296118-stormjar.jar/data (No such file or directory)