RabbitMQ "A call to SSPI failed" - "The message was unexpected or badly formatted"

Question

So, we were trying to use SignalR with ASP.NET, but because ASP.NET is no longer supported, and we have not moved to CORE yet, we needed a way for the servers to communicate. This brought us to RabbitMQ to use as a message broker service to communicate between the servers, and then SignalR will handle the client side communication to get messages out.

This implementation was working fine on one of our dev servers which is running on Windows 2019, however it doesnt work on our Windows 2012 r2 servers (all prod servers are Windows 2012 r2).

The error message says

A call to SSPI failed, see inner exception.
   at System.Net.Security.SslState.InternalEndProcessAuthentication(LazyAsyncResult lazyResult)
   at System.Net.Security.SslState.EndProcessAuthentication(IAsyncResult result)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at RabbitMQ.Client.Impl.SslHelper.<>c__DisplayClass2_0.<TcpUpgrade>b__0(SslOption opts)
   at RabbitMQ.Client.Impl.SslHelper.TcpUpgrade(Stream tcpStream, SslOption options)
   at RabbitMQ.Client.Impl.SocketFrameHandler..ctor(AmqpTcpEndpoint endpoint, Func`2 socketFactory, TimeSpan connectionTimeout, TimeSpan readTimeout, TimeSpan writeTimeout)
   at RabbitMQ.Client.ConnectionFactory.CreateFrameHandler(AmqpTcpEndpoint endpoint)
   at RabbitMQ.Client.EndpointResolverExtensions.SelectOne[T](IEndpointResolver resolver, Func`2 selector)
The message received was unexpected or badly formatted

I have looked all over google and it says to use TLS 1.2 which we are doing. Does anyone have any ideas on what could be causing the issues, and how we can fix it?

Here is the C# code

factory = new ConnectionFactory()
{
    HostName = url,
    Port = int.Parse(port),
    UserName = username,
    Password = password
};

Uri uri = new Uri(url + ":" + port);
factory.Ssl.Enabled = true;
factory.Uri = uri;
factory.VirtualHost = "/";

connection = factory.CreateConnection();
sendChannel = connection.CreateModel();
receiveChannel = connection.CreateModel();