ServiceBusSender stops working after a day

Question

I'm struggeling with a Azure ServiceBusSender which is executed in a Azure Function Http endpoint. It works fine for about a day. Then suddenly it will throw the following error and I will have to restart the azure function:

System.Exception: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. ErrorCode: TimedOut (ServiceCommunicationProblem). For troubleshooting information, see https://aka.ms/azsdk/net/servicebus/exceptions/troubleshoot.

If I restart the azure functions, it will continue to work for about a day.

Utilizing the Azure.Messaging.ServiceBus 7.16.1

Depency injection:

builder.Services.AddAzureClients(clientsBuilder =>
{
    clientsBuilder.AddServiceBusClient(appConfig.GetValue<string>("CRMSBActivityConnectionString"))
    .ConfigureOptions(options =>
      {
          options.RetryOptions.Delay = TimeSpan.FromMilliseconds(50);
          options.RetryOptions.MaxDelay = TimeSpan.FromSeconds(5);
          options.RetryOptions.MaxRetries = 3;
      });
    clientsBuilder.AddClient<ServiceBusSender, ServiceBusClientOptions>((_, _, provider) =>
        provider.GetService<ServiceBusClient>().CreateSender(appConfig.GetValue<string>("CRMSBActivityTopicName")
        
    )).WithName("SBSender");
});

Http endpoint:

private IServiceLogger _logger;
private const string _commonMessage = "SMSActivityStopGetHttpListenerV1:";
private readonly ServiceBusSender _sender;

public SMSActivityStopGetHttpListener(IServiceLogger logger, IAzureClientFactory<ServiceBusSender> serviceBusSenderFactory)
{
    _logger = logger;
    _sender = serviceBusSenderFactory.CreateClient("SBSender");
}

[ProducesResponseType((int)HttpStatusCode.Accepted)]
[ProducesResponseType((int)HttpStatusCode.InternalServerError)]
[ProducesResponseType((int)HttpStatusCode.BadRequest)]
[FunctionName("SMSActivityStopGetHttpListenerV1")]
public async Task<HttpResponseMessage> Run(
    [HttpTrigger(AuthorizationLevel.Function, "get", Route = "v1/smsactivity/stop")] HttpRequest req,
    ILogger log)
{
//Simple logic

try
{
    await _sender.SendMessageAsync(servicebusMessage);
}
catch (Exception ex)
{
    response = new HttpResponseMessage(HttpStatusCode.InternalServerError) { Content = new StringContent(errorMessage, Encoding.UTF8, "application/json") };
    SplunkLogger.LogExceptionToSplunk(_logger, _commonMessage, null, correlationId, "SMSActivityStop API failed", ex.Message);
}
return response;

}

Tried to update the packages. removed all sorts of extra details from the functions. But the problem persists. Stops working after one day even with very few requests. less than twenty.

Answer 1

I tried your code in my environment and got the same error below.

I made some changes to your code and received the message sent to the Service Bus Queue in Azure Portal.

Code :

using System;
using System.Net;
using System.Net.Http;
using System.Text;
using Microsoft.Azure.Functions.Extensions.DependencyInjection;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using Azure.Messaging.ServiceBus;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.Http;
using Microsoft.Azure.WebJobs.ServiceBus; 
using System.Threading.Tasks;

[assembly: FunctionsStartup(typeof(ServiceBusFunc.Startup))]

namespace ServiceBusFunc
{
    public class Startup : FunctionsStartup
    {
        public override void ConfigureAppConfiguration(IFunctionsConfigurationBuilder builder)
        {
            builder.ConfigurationBuilder
                .SetBasePath(Environment.CurrentDirectory)
                .AddJsonFile("local.settings.json", optional: true, reloadOnChange: true)
                .AddEnvironmentVariables()
                .Build();
        }

        public override void Configure(IFunctionsHostBuilder builder)
        {
            IConfiguration config = builder.GetContext().Configuration;
            builder.Services.AddSingleton(config);

            string serviceBusConnectionString = config.GetValue<string>("ServiceBusConnectionString");
            builder.Services.AddSingleton(x => new ServiceBusClient(serviceBusConnectionString));

            builder.Services.AddSingleton<ServiceBusSender>(x =>
            {
                var client = x.GetRequiredService<ServiceBusClient>();
                string queueName = config.GetValue<string>("QueueName");
                return client.CreateSender(queueName);
            });
        }
    }

    public static class SMSActivityStopGetHttpListener
    {
        [FunctionName("SMSActivityStopGetHttpListenerV1")]
        public static async Task<HttpResponseMessage> Run(
            [HttpTrigger(AuthorizationLevel.Function, "get", Route = "v1/smsactivity/stop")] HttpRequestMessage req,
            [ServiceBus("<your_queue_name>", Connection = "ServiceBusConnectionString")] IAsyncCollector<ServiceBusMessage> outputMessages,
            ILogger log)
        {
            string messageContent = "hello kamali, how are you?"; // Replace with your message content

            try
            {
                var message = new ServiceBusMessage(messageContent);
                var sender = outputMessages as IAsyncCollector<ServiceBusMessage>; 

                int maxRetries = 3;
                TimeSpan delayBetweenRetries = TimeSpan.FromSeconds(5);

                for (int retryCount = 0; retryCount < maxRetries; retryCount++)
                {
                    try
                    {
                        await sender.AddAsync(message);
                        return new HttpResponseMessage(HttpStatusCode.Accepted);
                    }
                    catch (ServiceBusException ex) when (ex.Reason == ServiceBusFailureReason.ServiceCommunicationProblem)
                    {
                        log.LogWarning($"Service Bus communication problem. Retrying in {delayBetweenRetries.TotalSeconds} seconds.");
                        await Task.Delay(delayBetweenRetries);
                    }
                    catch (Exception ex)
                    {
                        log.LogError(ex, "Error sending message to Service Bus");
                        return new HttpResponseMessage(HttpStatusCode.InternalServerError)
                        {
                            Content = new StringContent("Internal Server Error", Encoding.UTF8, "application/json")
                        };
                    }
                }
                log.LogError("Failed to send message after multiple retries.");
                return new HttpResponseMessage(HttpStatusCode.InternalServerError)
                {
                    Content = new StringContent("Internal Server Error - Message delivery failed", Encoding.UTF8, "application/json")
                };
            }
            catch (Exception ex)
            {
                log.LogError(ex, "Error processing message");
                return new HttpResponseMessage(HttpStatusCode.InternalServerError)
                {
                    Content = new StringContent("Internal Server Error", Encoding.UTF8, "application/json")
                };
            }
        }
    }
}

local.settings.json :

{
  "IsEncrypted": false,
  "Values": {
    "AzureWebJobsStorage": "UseDevelopmentStorage=true",
    "FUNCTIONS_WORKER_RUNTIME": "dotnet",
    "ServiceBusConnectionString": "<ServiceBus_Queue_Connection_String>",
    "QueueName": "<queue_name>"
  }
}

Note : Make sure that you have a stable Internet connection before going to run the above code.

Output :

It runs successfully and sends a message to the service bus queue.

Below is the output from the browser:

Azure Portal :

I got the message in my Service Bus Queue in Azure Portal as below:

Answer 2

All senders and receivers can be created using the injected into function IAzureClientFactory<ServicBusCient> factory object. This object acts as a singleton to avoid re-establishing the connection to the namespace as it's an expensive operation. Senders and receivers use the connection maintained by the ServiceBusClient and are cheap to create and dispose on demand.