C# HttpClient Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host

Question

I have a .NET 6 console app that goes through a list of websites and checks for any issues with SSL certificates or upcoming expiring certs. It works for all of the websites except a couple that are hosted on Azure. For these websites everything works as expected when debugging locally. However when the console app is run on a Windows 2012 server the http get request fails 9/10 times with this exception:

System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.. ---> System.Net.Sockets.SocketException (10054): 

These are the two main methods in my code:

        async Task<CertStatus> SendRequest(string url)
        {
            _certStatus = new CertStatus
            {
                URL = url,
                Success = false
            };

            var handler = new HttpClientHandler();

            handler.ServerCertificateCustomValidationCallback = ServerCertificateCustomValidation;


            var client = new HttpClient(handler);

            try
            {
                var response = await client.GetAsync(url);

                response.EnsureSuccessStatusCode();

                _certStatus.Success = true;
            }
            catch (HttpRequestException ex)
            {
                _certStatus.ErrorMessage = ex.Message;
            }

            handler.Dispose();
            client.Dispose();

            var retval = _certStatus;

            return retval;
        }

        static bool ServerCertificateCustomValidation(HttpRequestMessage requestMessage, X509Certificate2? certificate, X509Chain? chain, SslPolicyErrors sslErrors)
        {
            if(_certStatus != null && certificate != null)
            {
                _certStatus.ExpireDate = DateTime.Parse(certificate.GetExpirationDateString());
            }


            Console.WriteLine($"Requested URI: {requestMessage.RequestUri}");
            Console.WriteLine($"Effective date: {certificate?.GetEffectiveDateString()}");
            Console.WriteLine($"Exp date: {certificate?.GetExpirationDateString()}");
            Console.WriteLine($"Issuer: {certificate?.Issuer}");
            Console.WriteLine($"Subject: {certificate?.Subject}");

            Console.WriteLine($"Errors: {sslErrors}");
            return sslErrors == SslPolicyErrors.None;
        }

I've tried setting ConnectionClose to true

client.DefaultRequestHeaders.ConnectionClose = true;

Increasing the the timeout

client.Timeout = TimeSpan.FromSeconds(120);

And setting the SSL protocol to tls 1.2 even though this is .Net 6.0 and I don't think that should be necessary

System.Net.ServicePointManager.SecurityProtocol |= System.Net.SecurityProtocolType.Tls12;

Any other ideas on what I could try? It's strange to me that it is sometimes successful and fails other times. This console app is ran daily by windows task scheduler and it has always failed during those runs. However, when the exe is ran manually it will sometimes succeed