I wrote an assembly to receive messages from an Azure storage bus queue. This assembly is written as a regfree COM control. It is used in a Windows service written in language called Clarion for Windows. Windows Server is 2016. Microsoft.Azure.ServiceBus ver: 4.1.4.0 .Net version 4.7.2
The service receives messages successfully for a period of time and then stop receiving new messages. This service is also a webserver and I can browse to the server successfully when it is not receiving storage bus messages, so I know the service is still running. The thing that is really odd, is if I remote desktop into the Windows server the message service suddenly starts reading messages again. I don't do anything with the service. Just my logging into the server causes the messages to be received. The service is running under a domain account that is not the same account I use to remote into the server. I've also had the service running under Local System.
The period of time varies greatly. Recently it ran for 3 weeks without issue and then failed twice in 24 hours. Message volume is low, perhaps a dozen a day at this point. There are days without any messages but this is not necessarily when the failure occurs.
The service code is pretty simple:
public void Initialize(string pServiceBusConnection, string pQueueName)
{
try
{
WriteLog($"Initialize: " + pQueueName);
ServiceBusConnectionString = pServiceBusConnection;
QueueName = pQueueName;
queueClient = new QueueClient(ServiceBusConnectionString, QueueName);
RegisterOnMessageHandlerAndReceiveMessages();
}
catch (Exception e)
{
WriteLog(e.GetType().Name + " : " + e.Message);
}
}
private void RegisterOnMessageHandlerAndReceiveMessages()
{
var messageHandlerOptions = new MessageHandlerOptions(ExceptionReceivedHandler)
{
MaxConcurrentCalls = 1,
AutoComplete = false
};
WriteLog($"Register the function that will process messages");
queueClient.RegisterMessageHandler(ProcessMessagesAsync, messageHandlerOptions);
}
private async Task ProcessMessagesAsync(Microsoft.Azure.ServiceBus.Message pMessage, CancellationToken pToken)
{
WriteLog($"ProcessMessagesAsync");
//Save off message details and send event back to window saying new data is available.
MsgReceived = JsonConvert.DeserializeObject<ReceiveStatusUpdate>(Encoding.UTF8.GetString(pMessage.Body));
WriteLog($"Received message: SequenceNumber:{pMessage.SystemProperties.SequenceNumber} Body:{MsgReceived.LoanNumber}");
LoanNumber.Text = MsgReceived.LoanNumber;
SigningStatus.Text = MsgReceived.SigningStatus;
LocationId.Text = MsgReceived.LocationId.ToString();
try
{
Invoke((Action)(() => MessageReceived(Encoding.UTF8.GetString(pMessage.Body))));
}
catch (Exception e)
{
WriteLog(e.GetType().Name + " : " + e.Message);
}
await queueClient.CompleteAsync(pMessage.SystemProperties.LockToken);
}
private Task ExceptionReceivedHandler(ExceptionReceivedEventArgs arg)
{
WriteLog(arg.Exception.GetType() + ":" + arg.Exception.Message);
throw arg.Exception;
}
No error messages or exceptions are logged. Does anyone know what might be happening to cause the messages to stop being received? I am contemplating stopping the listen thread periodically and restarting it. The fact that remoting into server impact the listening service is very confusing.
I have additional information to add to this problem.
First, I updated the Azure Service bus assemblies to the latest 5.0.
Second, I re-wrote the service entirely in C# .Net eliminating the Clarion from the equation.
The errors are still occurring with the new service and still resolves itself when remoting into the server and logging in.
This is the exception being thrown.
Microsoft.Azure.ServiceBus.ServiceBusCommunicationException : An existing connection was forcibly closed by the remote host ErrorCode: ConnectionReset
In my code now, when the exception occurs I am closing my listener and creating a new QueueClient. However, once the service is in this "error" state the new QueueClient fails with the same error. Not until I log into the server does the "error" state go away and the service starts reading the queue messages again.
Also, I should mention my number of messages is very low at this point. Probably a max of 20 a day. We are not scaling this up until we can resolve this problem.