Python's email package from the standard library inserts spurious linebreaks after S/MIME boundaries which seems to break S/MIME signing

Question

I'm trying to use the cryptography library to sign a document.

The following script (working.py) works when it is executed from a directory that contains a private key and a certificate:

#!/usr/bin/python

import sys

from cryptography.x509 import load_pem_x509_certificate
from cryptography.hazmat.primitives.serialization import load_pem_private_key, pkcs7, Encoding
from cryptography.hazmat.primitives import hashes

with open('private_key.pem', 'rb') as key_file:
    private_key_data = load_pem_private_key(key_file.read(),password=b'password')

with open('certificate.pem', 'rb') as cert_file:
    certificate_data = load_pem_x509_certificate(cert_file.read())

text_to_sign = ('Some text\n'
                'Hällöchen W€lt!\n'
                'Ändosa\n')

sys.stdout.buffer.write(
    pkcs7.PKCS7SignatureBuilder(
    ).set_data(
        bytes(text_to_sign, 'utf-8')
    ).add_signer(
        certificate_data, private_key_data, hashes.SHA256()
    ).sign(
        Encoding.SMIME, [pkcs7.PKCS7Options.DetachedSignature]
    )
)

I can execute this script as follows:

$ ./working.py > working.signed
$ openssl smime -verify -CAfile certificate.pem -in working.signed
Some text
Hällöchen W€lt!
Ändosa
Verification successful

I would like to add some extra metadata to the document. Since I'm using pkcs7 with S/MIME encoding, I reached for Python's email and mime handling package from the standard library to accomplish this. Note that I'm preparing a specially crafted document, and not really an email. My goal is to enable the receiver to verify its authenticity of the document's payload (which is "Some text\nHällöchen W€lt!\nÄndosa\n" in my example). The payload can contain line breaks and Unicode characters beyond ASCII. I enhanced the script from above as follows (nonworking.py):

#!/usr/bin/python

import sys
import email
import email.policy

from cryptography.x509 import load_pem_x509_certificate
from cryptography.hazmat.primitives.serialization import load_pem_private_key, pkcs7, Encoding
from cryptography.hazmat.primitives import hashes

with open('private_key.pem', 'rb') as key_file:
    private_key_data = load_pem_private_key(key_file.read(),password=b'password')

with open('certificate.pem', 'rb') as cert_file:
    certificate_data = load_pem_x509_certificate(cert_file.read())

text_to_sign = ('Some text\n'
                'Hällöchen W€lt!\n'
                'Ändosa\n')

signed_document = email.message_from_bytes(
    pkcs7.PKCS7SignatureBuilder(
    ).set_data(
        bytes(text_to_sign, 'utf-8')
    ).add_signer(
        certificate_data, private_key_data, hashes.SHA256()
    ).sign(
        Encoding.SMIME, [pkcs7.PKCS7Options.DetachedSignature]
    ),
    policy=email.policy.default
)

# signed_document.set_param('charset', 'utf-8')
# signed_document['Date'] = email.utils.formatdate()
# signed_document['From'] = '[email protected]'
# signed_document['To'] = '[email protected]'
# signed_document['Subject'] = 'What the document is about'

sys.stdout.buffer.write(signed_document.as_bytes())

I can execute this script as follows:

$ ./non_working.py > non_working.signed
$ openssl smime -verify -CAfile certificate.pem -in non_working.signed

Some text
Hällöchen W€lt!
Ändosa
Verification failure
005EBA74547F0000:error:10800065:PKCS7 routines:PKCS7_signatureVerify:digest failure:crypto/pkcs7/pk7_doit.c:1081:
005EBA74547F0000:error:10800069:PKCS7 routines:PKCS7_verify:signature failure:crypto/pkcs7/pk7_smime.c:361:

The verification fails. The problem seems to be that non_working.signed contains an extra spurious newline after the first boundary:

MIME-Version: 1.0
Content-Type: multipart/signed; protocol="application/x-pkcs7-signature";
 micalg="sha-256"; boundary="----2E2B6EF0B6E7CB564955721240B068C0"

This is an S/MIME signed message

------2E2B6EF0B6E7CB564955721240B068C0

Some text
Hällöchen W€lt!
Ändosa

------2E2B6EF0B6E7CB564955721240B068C0
Content-Type: application/x-pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"

MIIGvQYJKoZIhvcNAQcCoIIGrjCCBqoCAQExDzANBglghkgBZQMEAgEFADALBgkq
hkiG9w0BBwGgggPbMIID1zCCAr+gAwIBAgIUDO1y0dqi2G2ds/8qXampRXN3+a8w
DQYJKoZIhvcNAQELBQAwezELMAkGA1UEBhMCZGUxEjAQBgNVBAcMCU51cmVtYmVy
ZzEPMA0GA1UECgwGRXhhc29sMQwwCgYDVQQLDANDT1MxETAPBgNVBAMMCHBhdWwu
...

That is not the case for working.signed:

MIME-Version: 1.0
Content-Type: multipart/signed; protocol="application/x-pkcs7-signature"; micalg="sha-256"; boundary="----0D7398911FD63AD56872B79624A2C78F"

This is an S/MIME signed message

------0D7398911FD63AD56872B79624A2C78F
Some text^M
Hällöchen W€lt!^M
Ändosa^M

------0D7398911FD63AD56872B79624A2C78F
Content-Type: application/x-pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"

MIIGvQYJKoZIhvcNAQcCoIIGrjCCBqoCAQExDzANBglghkgBZQMEAgEFADALBgkq
hkiG9w0BBwGgggPbMIID1zCCAr+gAwIBAgIUDO1y0dqi2G2ds/8qXampRXN3+a8w
DQYJKoZIhvcNAQELBQAwezELMAkGA1UEBhMCZGUxEjAQBgNVBAcMCU51cmVtYmVy
ZzEPMA0GA1UECgwGRXhhc29sMQwwCgYDVQQLDANDT1MxETAPBgNVBAMMCHBhdWwu
Y29tMSYwJAYJKoZIhvcNAQkBFhdwYXVsLmthbGV0dGFAZXhhc29sLmNvbTAeFw0y
...

The document non_working.signed can be made to verify, if one manually removes the spurious newline (the one before "Some text").

As you might have noticed, working.signed uses \r\n line breaks for the payload, while non-working.signed uses \n. This is probably explained by the email.policy.default that I specified. Surprisingly, to me, this does not seem to affect the verification of the signature.

In the source code of non_working.py I commented out the extra metadata that I wanted to add to the document headers. When it is included non_working.signature can still be made to verify, if one only manually removes the newline after the first boundary.

The email package is configurable with countless knobs, so it is possible that I overlooked something. Is this a problem with the email package, or did I do something wrong? If this is a bug in the email package, can you think of a work around?

How can I modify non_working.py to generate a document that will verify?