I have the following nginx config:
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
log_format custom '
$remote_addr - $remote_user [$time_local]
"$request" $status $body_bytes_sent
"$http_referer" "$http_user_agent"
Certificate: "$ssl_client_cert"
Client Key: "$ssl_client_raw_cert"
';
server {
listen 80;
listen [::]:80;
server_name patches.lan 172.16.192.128 ;
return 301 https://$host$request_uri;
}
server {
listen 443 ssl;
listen [::]:443 ssl;
server_name patches.lan 172.16.192.128 ;
access_log /var/log/nginx/access.log custom;
# Client-facing certificate and key
ssl_certificate "/patches/server_certs/patches.lan.crt";
ssl_certificate_key "/patches/server_certs/patches.lan.key";
# Request client certificates but skip verification
ssl_verify_client optional_no_ca;
ssl_trusted_certificate "/patches/server_certs/root_certs/rootCA.lan.crt";
location /api {
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header Host $http_host;
# Set X-SSL-CERT header with client certificate
proxy_set_header X-SSL-CERT $ssl_client_escaped_cert;
proxy_pass "https://patches-backend:9000";
}
location / {
proxy_set_header Host $http_host;
# Set X-SSL-CERT header with client certificate
proxy_set_header X-SSL-CERT $ssl_client_escaped_cert;
proxy_pass http://patches-frontend:3000;
}
}
}
Sometimes when I send a certificate that is not signed by nginx's root CA, it presents the below error:
If I use a certificate generated in exactly the same way except it is signed by the root CA, nginx works fine. The certificates are in .p12 format. This is one of the test certs I am using - openssl has no trouble with it and like I said, as long as the root CA has signed off on it, it work.
grant@test test]$ openssl pkcs12 -info -in server_certs/grant.p12
Enter Import Password:
MAC: sha256, Iteration 1
MAC length: 32, salt length: 8
PKCS7 Encrypted data: PBES2, PBKDF2, AES-256-CBC, Iteration 20000, PRF hmacWithSHA256
Certificate bag
Bag Attributes
friendlyName: grant
localKeyID: D0 8E 7D 53 4C FD 79 D9 77 7A 4B 48 3E 64 A9 A3 17 CB 52 E2
subject=C = US, ST = Ohio, L = Dayton, O = Grant, OU = The Grant Unit, CN = grant
issuer=C = US, ST = Ohio, L = Dayton, O = The CA Unit, CN = rootCA.lan
-----BEGIN CERTIFICATE-----
MIIESjCCAjKgAwIBAgIUK7H86F45GLtsMj8BSDqG8BRTt8UwDQYJKoZIhvcNAQEL
BQAwWDELMAkGA1UEBhMCVVMxDTALBgNVBAgMBE9oaW8xDzANBgNVBAcMBkRheXRv
bjEUMBIGA1UECgwLVGhlIENBIFVuaXQxEzARBgNVBAMMCnJvb3RDQS5sYW4wHhcN
MjMxMDEzMTgwMDA2WhcNMjYxMDEyMTgwMDA2WjBmMQswCQYDVQQGEwJVUzENMAsG
A1UECAwET2hpbzEPMA0GA1UEBwwGRGF5dG9uMQ4wDAYDVQQKDAVHcmFudDEXMBUG
A1UECwwOVGhlIEdyYW50IFVuaXQxDjAMBgNVBAMMBWdyYW50MIIBIjANBgkqhkiG
9w0BAQEFAAOCAQ8AMIIBCgKCAQEAvn9IsfL4F8mPIvnK1Xow254tqLCE09WPVNqV
91q2bj5a1BI+H1+Wx/+dS14fWNh76MRZWCtaQBsX4DgFn2+2glH24blfG8/boPof
SSDo6F0YGBJ0VlH7VInxEMC8vx8qNPxd9//Dw1cgk1z+sgT17j8SuspU+aTdV4DY
vGYwc1VoEATUKWx4aqBHXX3Nbo6Tj/1Jga2dktiIIMrknfcQaydkrzSewo4k42e1
95ZpasBNAa8D56NTJNC3o2WnEEiNQBoowULcI0Os5iPTl4rAtULHKvSTLiON1e9+
tazIzD9hQBc5YoyVvTLJD+H2aLP14INOA7wP84CU02u+p/6a8QIDAQABMA0GCSqG
SIb3DQEBCwUAA4ICAQBiWZuna6tyScqcXgsQwASpshx4o+nw0qvFlXpQM7opGZaj
/RLKvEIHAKK2YwrpqhwrWkAML3Qs03tBAEUYPMFy90nY1dkejvNa7PyxFd11L8ew
VDT7AXRR8sv/Y0MWpVK3PcsCqD8NSz3iMfhv6oS2ordTFelQn4RK26s9Xu3l5yg0
IbJ8d06aeR4kpMJsW/cT/6jwDTrZT030FnD3sDl05fKdACyz6XOmxCfNTB9RdeD+
TXxyv7JdNnJaY8eNzMaOMx7XUe5mOnPoEne9DlrNDjpQJ7QNfLNqgaIc2B5P2OFO
vwVJWGxYRy183xeTFfaE/0c95IoaySnuDG7kjEvkpOGH0gIFbcVEDJiY2vO7OrwR
q2+idfMq1/pskdQzY4yZvOpQNaHzfdYilEXZgBa8Rj20hrMpgX0jCRBSZ5PU+Qxx
FR1RpFEV7kF97NSYm1MCDLRXuEF6VekdbzidEd5nlMH+Y36OeXUAKXiFOS5A0QJd
M0gnfvUbLapu15dk8lzBqh/ZK9pwI8+RH+bjrzzkyixB9KYqTggfmwcdIobOUnWH
2X7xb1EbHLw3hBRfWoVbtrVlDtQS0tqViY6Pna0w7OtN6FZ16Wtwc7wbygMWUsaM
DwFJpnwk6EXZAmgwCrMuYGoYeur2wZp2BFNqG3jEPV3VXmHu/AHTlwlYdYh9Pg==
-----END CERTIFICATE-----
Certificate bag
Bag Attributes: <No Attributes>
subject=C = US, ST = Ohio, L = Dayton, O = The CA Unit, CN = rootCA.lan
issuer=C = US, ST = Ohio, L = Dayton, O = The CA Unit, CN = rootCA.lan
-----BEGIN CERTIFICATE-----
MIIFUTCCAzmgAwIBAgIULoPptx4mHKXsPV3irTJda0yEr3swDQYJKoZIhvcNAQEL
BQAwWDELMAkGA1UEBhMCVVMxDTALBgNVBAgMBE9oaW8xDzANBgNVBAcMBkRheXRv
bjEUMBIGA1UECgwLVGhlIENBIFVuaXQxEzARBgNVBAMMCnJvb3RDQS5sYW4wHhcN
MjMxMDEzMTgwMDAyWhcNMzMxMDEwMTgwMDAyWjBYMQswCQYDVQQGEwJVUzENMAsG
A1UECAwET2hpbzEPMA0GA1UEBwwGRGF5dG9uMRQwEgYDVQQKDAtUaGUgQ0EgVW5p
dDETMBEGA1UEAwwKcm9vdENBLmxhbjCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCC
AgoCggIBAL9XiVd+e6DFKUpd9mJuZ+y4yH9GxZBSHF/7IQ24KDN2ptBX4N2cDUdK
aaZzTa7bE4gybjZStWX432YEVwuVIplV4LNhc6/5hm81lBJM9zgD6+9wHvpsDjfj
/WGCIm84/odVHBDlR2XbWUDFi75TsUhoGquH93maBT+NMmBJn+vYJDyTHLAt+fyy
rRrfB8xp0MYuM2SF8uGmA1Bwkm5/hfX2f3cpU4r9lsKM0NXsX7jx3MKe7zO0eqSn
9ThVbPNafOyPxo92lOZ+vhSmNCvJKSYQUvtZnj2AM8HPZ6K+CeXNXwzJyARBgfUr
HxsK+T/5RbPffGwwyDKuBYYOQgGGk/oI7ATsWRo1SH7zRNzffvXk1YNZE9QMMC4z
1IDuNpmAn0Zgb8eXof/R5dpcOrIzHyYghNNfx5MpL+QSplLxWDZIuyoHiLeQVRGX
A9ynPopNyINgA8dV5iz37+Avx6iM2Hejmnwj0r5J8dHcGRe119NfYOvkw7JB5xhi
QFW/D/nnr2HGfJLP3ZmUtmWWJvPiLrEe9BfUukONfek00KHqQ219bww2iqln7L8u
jycOm3zyzuXnkq59mjUOVn546O+q/ADTrzKfstl5pJWfRrz5xgigVWSK9drXBmKZ
MrJBjlyXXU8fPQFDMHm/NxVR6toAq51UhaSYwg7e4tvQsPyK/wXlAgMBAAGjEzAR
MA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggIBACyvwoCHoVggCOO8
swYSRtHfU12nQ0+DWG315h2xDjMnFWPLntBkhli999K+Tp+QpSlqSrVKhEk0wA0f
eFAMLdlO3jqbtykI8EUlN1kxRT8L3BJvnvg0PEx9TvrX7PrkAIJhsvDo4scO1kfO
7gj8m6ZDVIGLeSksDn7x3Xx13OOd4xo6pC6z0GSJqPJ37cqOMPo2q3+OPvcbRT2t
5mPJYqKkmpDwfb8eq9YemGVysiDebE0AYdCNWYunu0Wty/MyBnGCAKr9uuGS95lk
y6KDHPcpH24Icvcbels2QtsoJbGjV4VOptQ84b4ZrSUMXwAXW91MKNaUC9BuzxJK
vKK7UlhAZqUAVG4GS0AQQVR+E6iO4Of9tF5HSKkpIwdEG2yFKDE4FnX4ny4FaQHR
dLmGrx3Qv/4B4p/RlRM5sSgIL1P4+XsTxB3xd5yn6PwXPiBKefzXKzR8ywzePUtA
Je8yITHZy0obmL43oUY4qLH/3cAoDG2pCYtGzCJgv2mB8/51yry1qG1GW4MjXcBl
m2nP8qDu++aLJdCLn+aB7FTBye23qvx419hOFWCYjZeh+89OyqiK+L7vGL9soLEV
aqgjB70AwIyU2tDgOHk+NK675GoBpoTLKm25E1XDcwGICi4xVVAvEstMMJ/wDkZC
yYWw6NY9TV4zOOPcKy1BXzfJHmmH
-----END CERTIFICATE-----
PKCS7 Data
Shrouded Keybag: PBES2, PBKDF2, AES-256-CBC, Iteration 20000, PRF hmacWithSHA256
Bag Attributes
friendlyName: grant
localKeyID: D0 8E 7D 53 4C FD 79 D9 77 7A 4B 48 3E 64 A9 A3 17 CB 52 E2
Key Attributes: <No Attributes>
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:
-----BEGIN ENCRYPTED PRIVATE KEY-----
MIIFLTBXBgkqhkiG9w0BBQ0wSjApBgkqhkiG9w0BBQwwHAQIzLY2P41vu+oCAggA
MAwGCCqGSIb3DQIJBQAwHQYJYIZIAWUDBAEqBBCzj0ilNyorH++6aDP7sVQhBIIE
0IMBc1w0ZmSIML7t+7nNIOJd60t18tuzzsvx/lC1a5bHC0AO4BIJtJhJY+ueTBZH
cUZqI7iAqEJMBizmS4NsVG23e5xkeI2XJRahQnyKn+m+XHOfcQbAyDW0C/1uapMo
SCBrYgkrrsfGbJF7sablGEqfdSe0LlC+6d+hTeERqYaVJdkhAQDuKhfPWiDMvmp7
qpD8/7GSNW8RsMhl27+F/8hwEW4IhhUbSaHwPNBFj4LLbTNllp8o9ajIIQytGXHe
9zrvquaPcYcttSvsPesqoTVzDpRJ7ZNbHkOo+doh/sqfXqynuL4IOUMuPsD33VKP
iKswVlO+l8l/tgEWpVcB2RnjOlPAsqen2rS3wlMP04esdBppqiYULQJ5C+SCOtgE
Hzzp952L2i7rpEKPrRoFMMTzoZTimRJkIa94d8jjqZKeYH8Pw36wOyW+4DviYf50
jeru5R89r8y5+Kh4tZr1FEsL7bH1i6k9bKd1U4OAQ5VODUcmKy2ZHpuL3UbgoYtn
eAATp7/HsD7XWC4Txm/XdFvGVZCp/Z0m0sP8GqhAos+uNBo0YvySofiR8FBSFYJd
3qHZ4xImUMoDy56jvT+yNqMF1VGnQ6ognVoPv2CxAcVSqRUZZGl36Gh8R/06nUci
QDHth4bA2S7Qkg+bzsxzmK6XfiQZ24qvDwVzgS9JvAMZc/reX4yHHIvOUT1P0WFQ
shm3DrjIiCNnZ0Hl3j0ekNX0BAGsBeMYfaSJ1y/kSdNR9UZzJW4bOjgnsmyyVYEl
s0TaR4vVqEgJqSR3T7WMwyXmuh75W/kxuk2u1FxfAk3EKI+lXR/YlrUzrR/WlOC7
/xhcI10dRC7GNGZXSQpsQViZBfazfNxwAC3g8PKlfqQ9l8Mwd/0tA1lISiSduutI
A+VfaXA1uJ8DXrL2p04KR1F9oxcJCusU4MLZZKHh0WDzzQt4k+nC3uNF1hUbN357
qVPqwY9HAzRRBoXcP+gs93Ddya7oXYTfSt1W0BRy6maLI62mB48kbUilqqPWzk8y
9lOfw3AtpMbktfHX0fQ6A+SUhGGHLTzqNJGsqqC7nopF4dSv1/O2FgQTO5gp6qZI
74aCp0/IhPUapdbByz+ZwuZnKLOm+mVvrWjudDgVsDGU6650yI4AoF03EW4fNpsW
nWgYiyW+iXIWwadZSQgoIlyo8kgdFpnYjRpkPD5u0CyqNIL2rGtHncWs5DNb8U8c
J3KBQpomHOkEOkbYndg0/wFkOgcOxkhj56RN7bC/bfcSYbc3pylSkDAd3cuvMpaA
P8ikjkmcxk/LDhcTs7h9xdKEg6Lkd3Udz72O/anKUJEbblgwVotK0WlO8d6NFrZM
psy6RRnUclIE8aLoSvUPThIH+oaSKwl072OcIDrQDaBrxfMzI2U9wOzjU28fpQWA
XhQ/qunZYO1da/pOSYpGI7gx0E++KrGnG5isJnlkIKmWYG4OmEEaXvUyPJ5kh9EB
IO/sXtgPMMdNqPJYObrO4UZXE59R0NftxhgEoXoBwVyDozzXrlprgwJr9ZcNVJjN
4xeSEqmUEaQk8pwW/Nlr2cTyBjxQNlgwoE1oY+uOfFmEUsy/yFGN8RYUBBMDtLhB
DP22lDicWQ157xJGos3f+bPkLVl3m+EZxRRgFr0ew+Av
-----END ENCRYPTED PRIVATE KEY-----
Sometimes, without any pattern I have been able to discern, I will get the below on certs that are not signed by the CA. But not always and I have no idea why.
2023/10/13 17:53:56 [alert] 24#24: *15 ignoring stale global SSL error (SSL: error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed error:0D0C5006:asn1 encoding routines:ASN1_item_verify:EVP lib) while waiting for request, client: 10.89.0.21, server: 0.0.0.0:443
Where I'm lost is that the cert, as far as I can tell, is completely valid. It imports into the browser, openssl can use it, and all of these certs are generated identically with the only difference being who the root CA is and the common name. I'm not sure if there's a bug somewhere else in the works and the certs are actually invalid but it's not clear to me what I need to change in the cert to make nginx happy.
This is running on a VM in VMWare Workstation local to me so there's no wonky networking going on here. Just direct from host, through internal network, straight to the VM. The problem also presents on a test production server across a real network so I am pretty convinced at this point that it is something to do with either the cert or nginx.