On our current deployment strategy, a Spring .jar file is built from the source code by the pipeline which is then pushed to a central artifactory. When a deployment is triggered manually, the .jar is pulled from the artifactory, the file is placed on the corresponding deployment environment server and properties values are replaced based on the deployment environment (done by deployment tool automatically) and the file is run. To be precise, we use Bamboo for build and Octopus Deploy for deployment. The deployment servers are Windows Server 2012 with JRE/JDK 17.
This was working fine for us for years and is still working fine for many projects but suddenly one of our .jar files stopped working as expected. This doesn't impact the whole application but a certain part of the application. There are no issues on developer's machines, it just occurs when deployed. The main issue in brief is, Spring boot server starts and serves the application (no issues anywhere), when the application is run, it picks a queue process from ActiveMQ, does the work and sends the response, which when the jar is not working can't formulate a response and sends an empty response which otherwise should have sent some response body back to ActiveMQ (Not exactly sure what does the app do and what does it send). The response can be seen on the ActiveMQ portal. Me not being a developer, this may not be the best description, but this what I've understood so far.
When the jar doesn't work, it throws a warning. Below are last few lines of the app log. The "WARN" line below shows the warning.
2024-02-21-11:31:10 INFO [main] com.comp.claims.Application Started Application in 5.381 seconds (process running for 6.203)
2024-02-21-11:31:21 INFO [http-nio-9900-exec-1] o.a.c.c.C.[.[localhost].[/msdrg] Initializing Spring DispatcherServlet 'dispatcherServlet'
2024-02-21-11:31:21 INFO [http-nio-9900-exec-1] o.s.web.servlet.DispatcherServlet Initializing Servlet 'dispatcherServlet'
2024-02-21-11:31:21 INFO [http-nio-9900-exec-1] o.s.web.servlet.DispatcherServlet Completed initialization in 0 ms
2024-02-21-11:32:46 INFO [Camel (camel-1) thread #1 - JmsConsumer[jms/MI/MSDRG/In/Queue]] o.a.c.converter.jaxp.StaxConverter Created XMLInputFactory: com.sun.xml.internal.stream.XMLInputFactoryImpl@3c600ee8. DOMSource/DOMResult may have issues with com.sun.xml.internal.stream.XMLInputFactoryImpl@3c600ee8. We suggest using Woodstox.
2024-02-21-11:32:46 WARN [Camel (camel-1) thread #1 - JmsConsumer[jms/MI/MSDRG/In/Queue]] o.a.camel.component.jms.JmsBinding Cannot determine specific JmsMessage type to use from body class. Will use generic JmsMessage. Body class: com.vh.arms.dto.ArmsServiceResponseTO. If you want to send a POJO then your class might need to implement java.io.Serializable, or you can force a specific type by setting the jmsMessageType option on the JMS endpoint.
As steps of troubleshooting, we tried different permutations and combinations to find out what caused the issue, but we couldn't reach a conclusion. We've focused our troubleshooting to platform rather than code because the exact same code is working on some places.
We tried to pull the jar from artifactory, replaced the properties values using 7Zip and also jar xf, jar cf and manually ran on our local machines (Win 10/11), it worked.
We tried to change our deployment target from Windows Server 2012 to a Linux server, moved the final replaced jar to our target Win server and it worked.
We tried to pull the jar from artifactory on our deployment target i.e. Win Server 2012 manually, edited the jar manually and it didn't work.
We tried to pull the jar from artifactory on a Win Server 2016 manually, edited the jar manually and it worked.
We tried to pull the jar from artifactory on a different Win Server 2012 manually, edited the jar manually and it worked.
Based on these findings, it seems like the issue exists only for that specific server. We tried matching encodings, line break types, editor versions etc. but nothing seemed to have a direct impact.
And now we have no idea what the root cause is. What might be the possible issues?