PubNub service stops listening after a few days

Question

I have an ASP.NET Core 6 application that's hosted in IIS. In the application I use PubNub 6.13.0 service to subscribe to a channel and listen/respond to incoming requests. The problem is the service stops listening after a while (usually a couple of days). The application works otherwise (handling REST and SignalR requests just fine), but the only way I can get PubNub to work is by Recycling the app pool.

I have the app deployed to about 20 servers at this point (both in AWS and on-prem), and this happens on every single instance. Any idea of what could potentially cause it, or how I could troubleshoot what's causing it, or at the very least hack some workaround? Thanks.

I initialize the service as singleton:

services.AddSingleton<IPubNubService, PubNubService>();

I 'start up' the instance on startup like this:

public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
   //...
   app.ApplicationServices.GetService<IPubNubService>(); 
}

Here's a simplified view of the PubNubService itself:

public class PubNubService : IPubNubService
{
    public PubNubService()
    {
        _pubNub = new Pubnub(new PNConfiguration(new UserId($"PubNub:UserId"))
        {
            SubscribeKey = config.GetValue<string>("PubNub:SubscribeKey"), 
            PublishKey = config.GetValue<string>("PubNub:PublishKey"), 
        });
        
        _pubNub.Subscribe<string>()
            .Channels(new[] { "PubNub:BroadcastChannel" })
            .Execute();

        _pubNub.AddListener(CreateListener());
    }

    // https://github.com/pubnub/c-sharp/
    // https://www.pubnub.com/docs/sdks/c-sharp/api-reference/configuration
    private SubscribeCallbackExt CreateListener()
    {
        return new SubscribeCallbackExt
        (
            messageCallback: async delegate(Pubnub pnObj, PNMessageResult<object> pubMsg)
            {
                await HandleReceivedMessage(pubMsg);
            },
        );
    }

    rivate async Task HandleReceivedMessage(PNMessageResult<object> pubNubMessage)
    {
        // ...
    }
}

EDIT: I obtained these logs after the application failed after a few days of use

DateTime 05/30/2023 15:50:55 CheckSocketConnect Entered 
DateTime 05/30/2023 15:50:55, SubscribeManager - ok. expected subscribe within threshold limit of SubscribeTimeout. No action needed 
DateTime 05/30/2023 15:51:05 CheckSocketConnect (HttpClient Or Task.Factory) Failed The request was canceled due to the configured HttpClient.Timeout of 10 seconds elapsing. 
DateTime 05/30/2023 15:51:05 ParseCheckSocketConnectException Error. The request was canceled due to the configured HttpClient.Timeout of 10 seconds elapsing. 
DateTime 05/30/2023 15:53:30 CheckSocketConnect Entered 
DateTime 05/30/2023 15:53:30, SubscribeManager - StartSubscribeHeartbeatCheckCallback - No network or no pubnub instance mapping 
DateTime: 05/30/2023 15:53:30, SendRequestAndGetJsonResponseHttpClient Exception The operation was canceled. 
DateTime 05/30/2023 15:53:30 GetTimeWithHttpClient Resp OK 
DateTime 05/30/2023 15:53:30, reconnection policy is DISABLED, please handle reconnection manually. 
DateTime 05/30/2023 15:56:05 CheckSocketConnect Entered 
DateTime 05/30/2023 15:56:05, SubscribeManager - **No auto subscribe within threshold limit of SubscribeTimeout**. Calling MultiChannelSubscribeRequest 
DateTime 05/30/2023 15:56:05, reconnection policy is DISABLED, please handle reconnection manually.  

So the problem comes down to basically the following error: reconnection policy is DISABLED, please handle reconnection manually

Answer 1

Solved the problem by adding PubNub logging, and observing the following message when connection stopped:

reconnection policy is DISABLED, please handle reconnection manually

The solution was changing PNReconnectionPolicy from default NONE to LINEAR or EXPONENTIAL.

The SDK has three types of Reconnection Policies when there is a network or internet issue. (default) PNReconnectionPolicy.NONE - indicates that NO action will taken when there is a network or internet issue. PNReconnectionPolicy.LINEAR - SDK will try to reconnect each 3 seconds. PNReconnectionPolicy.EXPONENTIAL - SDK uses the Exponential Backoff algorithm to reconnect when there is a network or internet issue. SDK uses MINEXPONENTIALBACKOFF = 1 second and MAXEXPONENTIALBACKOFF = 32 seconds. See: https://en.wikipedia.org/wiki/Exponential_backoff for more details.