HTTP CONNECT to an Azure DPS Endpoint through proxy failing

53 Views Asked by At

In one of our customer location, we are trying to enroll a IoT Device to our Azure IoT Platform using Azure DPS.
Our Azure Cloud infrastructure has a proxy server in front of Azure DPS. When the IoT device is trying to connect to DPS, the request is failing with the below exception

ProvisioningTransportException: MQTT transport exception at Microsoft.Azure.Devices.Provisioning.Client.Transport.ProvisioningTransportHandlerMqtt.

<RegisterAsync>d__13.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at AzureDeviceModule.ProvisioningManager.<ProvisionWithDpsCertificate>d__18.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at AzureDeviceModule.ProvisioningManager.<ProvisionWithDpsCertificateAsync>d__12.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at AzureDeviceModule.AzureClient.<ProvisionWithDpsCertificateAsync>d__21.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at ICE.AgentCore.RMSConnectorAgent.AzureRegistrationMonitor.HandleRegistrationRequest(RMSConnectorAgent agent, ICloudClient client, Registration reg)
INNER Exception: WebSocketException: Unable to connect to the remote server
   at System.Net.WebSockets.ClientWebSocket.<ConnectAsyncCore>d__21.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Devices.Provisioning.Client.Transport.ProvisioningTransportHandlerMqtt.<ProvisionOverWssCommonAsync>d__21.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Devices.Provisioning.Client.Transport.ProvisioningTransportHandlerMqtt.<RegisterAsync>d__13.MoveNext()
INNER Exception: WebException: The underlying connection was closed: An unexpected error occurred on a receive.
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Net.WebSockets.ClientWebSocket.<ConnectAsyncCore>d__21.MoveNext()
INNER Exception: IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
   at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
   at System.Net.PooledStream.EndRead(IAsyncResult asyncResult)
   at System.Net.Connection.ReadCallback(IAsyncResult asyncResult)
INNER Exception: SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
   at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)
   at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)

Since the exception did not provide more details about the root cause, I collected network traces to analyze further using wireshark and noticed that the HTTP CONNECT request from my iot device to dps endpoint through my proxy server did not succeed. TCP connection to proxy server is successful, but the HTTP CONNECT to DPS endpoint fails as seen below in the wireshark image.

enter image description here

This issue is happening very specific in one customer location and not facing this issue at other places. Hence we rule out issue on the proxy server end and something is going wrong on the customer network side. Any thoughts on what could cause the HTTP CONNECT to fail would be very helpful.

0

There are 0 best solutions below