SSL Certificate not trusted on initial connection

Question

I have a really frustrating SSL bug where, depending on connection order, a valid certificate is not being trusted.

So if on startup, I connect directly to server A. the trust manager will fail to verify the chain with the error: java.security.cert.CertPathValidatorException: Trust anchor for certification path not found.

But if instead I connect to server B, and then subsequently to server A it will trust the same certificate it previously claimed was invalid.

Does anyone have any ideas on what might be happening to cause this flip flop in trust?

This is how I'm setting up the trust manager:

    try {

      TrustManagerFactory tmf = TrustManagerFactory.getInstance("X509");
      tmf.init((KeyStore) null);
      X509TrustManager manager = (X509TrustManager)tmf.getTrustManagers()[0];
      manager.checkServerTrusted(chain, authType);
      Log.d("Auth","Did trust it");
    } catch (Exception e) {
      Log.e("Auth","Didn't trust it", e);
    }

As far as I can tell everything that happens up to the handshake is identical between the different connections. The only difference is the address. This is happening while running on android 10 on a pixel 3.