Extract public key from PKCS11 keystore to do signature verification

Question

I'm generating a keypair in a softHSM keystore, and uses it to sign a software.

Now I need to extract the public key and move it to my target.
I can sign and verify with pkcs11-tool, but I have not succeeded with using openssl verifying.

All I get is

80DB511C4A7F0000:error:02000068:rsa routines:ossl_rsa_verify:bad signature:../crypto/rsa/rsa_sign.c:430:

This is the series of commands I use to generate the keys.

softhsm2-util --init-token --free --label "token-label" --so-pin mysecret1 --pin mysecret1

pkcs11-tool --module /usr/lib/softhsm/libsofthsm2.so --login --login-type so --so-pin mysecret1 --init-pin --new-pin mysecret1

# create a public-private key pair. 
pkcs11-tool --module /usr/lib/softhsm/libsofthsm2.so -l --token-label token-label -k --key-type rsa:2048      --usage-sign --id 1002 --label rsatest     --pin mysecret1

pkcs11-tool --modul /usr/lib/softhsm/libsofthsm2.so --id 1002 --read-object --type pubkey -o rsa.der
openssl ec -pubin -inform DER -in rsa.der -outform PEM -out rsa.pem

pkcs11-tool --module /usr/lib/softhsm/libsofthsm2.so --login --pin mysecret1 --sign --id 1002 -m RSA-PKCS   --input text.txt --output /tmp/rsa.signature

pkcs11-tool --module /usr/lib/softhsm/libsofthsm2.so --id 1002  --verify -m RSA-PKCS   --input-file text.txt --signature-file  /tmp/rsa.signature

openssl dgst -sha512 -keyform PEM -verify rsa.pem -signature /tmp/rsa.signature text.txt

I have tried several combinations of PEM and DER files. Also to change the -sha512 parameter.

A demo project with Dockerfile can be found here: https://gitlab.com/kjeld.flarup/consoletest

Answer 1

A little inspired by https://stackoverflow.com/users/589259/maarten-bodewes I searched for RSA-PSS and found that I could specify SHA512 in the mechanism. Then it worked.

pkcs11-tool --module /usr/lib/softhsm/libsofthsm2.so --login --pin mysecret1 --sign --id 1002 -m SHA512-RSA-PKCS --input text.txt --output /tmp/rsa.signature