Java verifiying the response signature with a EC public key and detached payload

Question

I'm trying to verify a reponse signature a pem certificate(X50) and a EC Public key. The payload is in XML and i'm using a JWSVerifier

I've tried the following

        String reconstructed = header + Base64.getEncoder().encodeToString(rawPayloadString.getBytes()) + signature;

            CertificateFactory fact = CertificateFactory.getInstance("X.509");
            X509Certificate cer = (X509Certificate) fact.generateCertificate(new ByteArrayInputStream(Base64.getDecoder().decode(base64Cert)));
            ECPublicKey publicKey = (ECPublicKey) cer.getPublicKey();
            JWSObject jwsObject = JWSObject.parse(reconstructed);
            JWSVerifier verifier = new ECDSAVerifier(publicKey);
            boolean isVerified = jwsObject.verify(verifier);

            if (isVerified) {
                System.out.println("Signature is valid.");
            } else {
                System.out.println("Signature is invalid.");
            }
            assertTrue(isVerified);

but it always fails. the header and signature are json and are already base64 encode. the header has a . at the end and the signature starts with a . so the full reconstructed string is JSON_HEADER64.XML_PAYLOAD64.SIGNATURE64

I then use the public key from the generator of the response signature to verify it. but it always fails. I don't know what is wrong. To me this looks correct.