I am trying to debug locally in intelliJ an application running in a remote Spark cluster. I think I have half of it done, my problem is that I am connecting to the edge node via VDD without administrator provileges and I don't have much info about the remote server.
I tried to remote debug an app through Mobaxterm port forwarding and intelliJ creating a Agent JDWP acting as a server and attaching a socket connection in a specified free port in my machine. I was trying to do this so I can debug the whole application without harcoding in my local machine via intelliJ debugger. However, I am failing to attach the debugger to the JDWP.
I am running a tunnel in mobaXterm to the server through SSH. I run the application with variable SPARK_SUBMIT_OPTS= "-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=myPORT" where myPORT is a port I checked is free in my local.
The app runs and shows: spark-submit [...] --conf [..] Listening for transport dt_socket at address: myPort
I try to attach the remote JVM debugger from intelliJ with this port and it shows: "java.net.Connect.Exception "Connection refused: connect"
Any guidance on what to check or what I could be missing to work on? Thank you.