SignalR client never connects in CI environment. Goes from connecting to disconnected and throws a 500 error

Question

I'm troubleshooting some integration tests that use Asp.Net (framework) SignalR in a scenario where we have multiple clients that connect to a server.

The client and server are actually on the same process (MSTest). It was designed this way so we can use unity injection to inject some fake services into the clients. When I troubleshoot locally, everything works and my tests always pass.

When I try to run the tests in our CI environment (Jenkins), the clients attempt to connect and then immediately disconnect and return a 500 error. The tests fail in Jenkins about 90% of the time but sometimes the clients connect and my tests pass.

The exception I get looks like this:

"Exception":"System.Exception: 
    Microsoft.AspNet.SignalR.Client.HttpClientException: 
        StatusCode: 500, 
        ReasonPhrase: 'Internal Server Error', 
        Version: 1.1, 
        Content: System.Net.Http.StreamContent, 
        Headers:\r\n{\r\n  
        Date: Fri, 09 Feb 2024 22:18:39 GMT\r\n  
        Server: Microsoft-HTTPAPI/2.0\r\n  
        Content-Length: 0\r\n}\r\n   
        at Microsoft.AspNet.SignalR.Client.Http.DefaultHttpClient.<>c__DisplayClass5_0.<Get>b__1(HttpResponseMessage responseMessage) 
        in /_/src/Microsoft.AspNet.SignalR.Client/Http/DefaultHttpClient.cs:line 95\r\n   
        at Microsoft.AspNet.SignalR.TaskAsyncHelper.TaskRunners`2.<>c__DisplayClass3_0.<RunTask>b__0(Task`1 t) 
        in /_/src/Microsoft.AspNet.SignalR.Core/TaskAsyncHelper.cs:line 1280

I thought it was a bit strange that the headers are empty

Here's the server startup code:

private void StartHub(string hubUrl)
        {
            m_webThread = new Thread(() =>
           {
               m_webApp = WebApp.Start<HubStartup>(hubUrl.Trim());
           });

            m_webThread.Start();
        }

HubStartup will then load the AgentHub assembly onto the current AppDomain and configure signalR like this:

public void Configuration(IAppBuilder app)
{

            HubConfiguration hubConfiguration = new HubConfiguration();
            hubConfiguration.EnableDetailedErrors = true;
            app.MapSignalR(hubConfiguration);
}

When its time to start the clients, they initialize a HubConnection like this:

// Ensure that connections to the hub are not constrained by the ServicePointManager.
            ServicePoint hubServicePoint = ServicePointManager.FindServicePoint(new Uri(HubUrl));
            hubServicePoint.ConnectionLimit = int.MaxValue;

            
            bool debugEnv = false;

            LogDebug($"Creating connection to SignalR Hub {HubName} at {HubUrl}.");
            m_hubConnection = new HubConnection(HubUrl);
            m_hubConnection.StateChanged += HubConnection_StateChanged;

#if DEBUG
            debugEnv = true;
#endif

            string jenkinsEnv = Environment.GetEnvironmentVariable("JENKINS", EnvironmentVariableTarget.Machine);

            // Ignore invalid certificates in DEBUG and Jenkins builds
            if (!string.IsNullOrWhiteSpace(jenkinsEnv) || debugEnv)
            {
                // Enable detailed SignalR tracing in debug and Jenkins builds.
                m_hubConnection.TraceLevel = TraceLevels.All;
                m_hubConnection.TraceWriter = Console.Out;
                IgnoreBadCertificates();
                m_hubConnection.Error += ex => LogError($"SignalR Error {ex.Message}", ex);
            }
                

            LogDebug($"Configuring shared key for SignalR Hub {HubName} at {HubUrl}.");
            m_hubConnection.Headers.Add(Constants.AgentAuthorizationKey, SharedKey);

            LogDebug($"Creating proxy to SignalR Hub {HubName} at {HubUrl}.");
            m_hubProxy = m_hubConnection.CreateHubProxy(HubName);

            // Allow the consumer to initialize the hub proxy before starting the hub connection.
            HubProxyInitializer?.Invoke(HubProxy);
            
            
            LogInfo($"Starting connection to SignalR Hub {HubName} at {HubUrl}.");
            Stopwatch sw = Stopwatch.StartNew();
            await m_hubConnection.Start().ContinueWith(task =>
            {
                sw.Stop();
                
                if (task.IsFaulted) {
                    LogError($" There was an error opening the connection. (took {sw.ElapsedMilliseconds}ms)", task.Exception.GetBaseException());
                } else {
                    LogDebug($"StartHubConnectionAsync: Connection start complete (took {sw.ElapsedMilliseconds}ms)");
                }
            });

A lot of the code I've added just for troubleshooting purposes include the detailed tracing and error event handler. IgnoreBadCertificates() sets the ServicePointManager.ServerCertificateValidationCallback to always return true.

I should also note we are using plain http not https here so I don't think the certificate will even come into play. I've looked a bit into the transport methods and the jenkins drones are not setup to use websockets. However, neither is my local machine and the connection works fine. I've also tried manually setting a long polling transport but it didn't make a difference.

If anyone has any further troubleshooting suggestions, I would greatly appreciate it!