I am adding the certificates to the cluster via a bootstrap using keytool
keytool -import -file impl-stable.crt -alias impl-stable.crt -keystore /usr/lib/jvm/java/jre/lib/security/cacerts -storepass changeit -noprompt
I have dumped the same cert from the key store and verified that it contains the SAN name that Spark is complaining about. Please note it does have the appropriate SAN names.
keytool -list -rfc -keystore /usr/lib/jvm/java/jre/lib/security/cacerts -alias certs/impl-stable.crt
cut the PEM formatted cert out of the output from a.
openssl x509 -in cert.pem -noout -text
X509v3 Subject Alternative Name: critical
DNS:mongodb-0, DNS:mongodb-0.mongodb-svc.core.svc.cluster.local, DNS:mongodb-1, DNS:mongodb-1.mongodb-svc.core.svc.cluster.local, DNS:mongodb-2, DNS:mongodb-2.mongodb-svc.core.svc.cluster.local, DNS:DNS:dev-canary-v4-mongodb0.ams-np-v4.internal.stuff.org, DNS:DNS:test-canary-v4-mongodb0.ams-np-v4.internal.stuff.org, DNS:DNS:impl-stable-v4-mongodb0.ams-np-v4.internal.stuff.org, DNS:DNS:impl-stable-v4-mongodb1.ams-np-v4.internal.stuff.org, DNS:DNS:impl-stable-v4-mongodb2.ams-np-v4.internal.stuff.org
Signature Algorithm: sha256WithRSAEncryption
The error:
"errorMessage": "com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting for a server that matches com.mongodb.client.internal.MongoClientDelegate$1@508e2f4c. Client view of cluster state is {type=REPLICA_SET, servers=[{address=impl-stable-v4-mongodb2.ams-np-v4.internal.stuff.org:30019, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketWriteException: Exception sending message}, caused by {javax.net.ssl.SSLHandshakeException: No subject alternative DNS name matching impl-stable-v4-mongodb2.ams-np-v4.internal.stuff.org found.}, caused by {java.security.cert.CertificateException: No subject alternative DNS name matching impl-stable-v4-mongodb2.ams-np-v4.internal.stuff.org found.}}, {address=impl-stable-v4-mongodb0.ams-np-v4.internal.stuff.org:30017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketWriteException: Exception sending message}, caused by {javax.net.ssl.SSLHandshakeException: No subject alternative DNS name matching impl-stable-v4-mongodb0.ams-np-v4.internal.stuff.org found.}, caused by {java.security.cert.CertificateException: No subject alternative DNS name matching impl-stable-v4-mongodb0.ams-np-v4.internal.stuff.org found.}}, {address=impl-stable-v4-mongodb1.ams-np-v4.internal.stuff.org:30018, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketWriteException: Exception sending message}, caused by {javax.net.ssl.SSLHandshakeException: No subject alternative DNS name matching impl-stable-v4-mongodb1.ams-np-v4.internal.stuff.org found.}, caused by {java.security.cert.CertificateException: No subject alternative DNS name matching impl-stable-v4-mongodb1.ams-np-v4.internal.stuff.org found.}}]\n\tat com.mongodb.internal.connection.BaseCluster.createTimeoutException(BaseCluster.java:403)
We created the SAN with "DNS:" as part of the name...that explains things. Removing it and only using the FQDN seems to have resolved the issue.