Mbed TLS: in-place en-/decryption for OAEP doesn't seem to work

Question

RSA encryption/decryption works in Mbed TLS when using separate buffers where plainText, cipherText, and decryptedText (i.e. the content of plainText and decryptedText) are the same.

However, it does not when using just one buffer to perform in-place encryption/decryption. In that case I get gibberish/not correctly encrypted data. Is that just a general limitation or is my code wrong?

It says here that "In place cipher is allowed in Mbed TLS, unless specified otherwise." I'm not sure if they are talking about AES or RSA though. I didn't see any remark indicating "otherwise" for OAEP encryption/decryption so I presumed it should work.

Code:

size_t sizeDecrypted;
unsigned char plainText[15000] = "yxcvbnm";
unsigned char cipherText[15000];
unsigned char decryptedText[15000];

rtn = mbedtls_rsa_rsaes_oaep_encrypt(&rsa, mbedtls_ctr_drbg_random, &ctr_drbg, NULL, 0, sizeof("yxcvbnm"), &plainText,  &cipherText);
rtn = mbedtls_rsa_rsaes_oaep_decrypt(&rsa, mbedtls_ctr_drbg_random, &ctr_drbg, NULL, 0, &sizeDecrypted, &cipherText,  &decryptedText, 15000);
//decryptedText afterwards contains the correctly decrypted text just like plainText
//sizeDecrypted is 8 (because of the binary zero at the end of the string)

unsigned char text[15000] = "yxcvbnm";
rtn = mbedtls_rsa_rsaes_oaep_encrypt(&rsa, mbedtls_ctr_drbg_random, &ctr_drbg, NULL, 0, sizeof("yxcvbnm"), &text,  &text);
rtn = mbedtls_rsa_rsaes_oaep_decrypt(&rsa, mbedtls_ctr_drbg_random, &ctr_drbg, NULL, 0, &sizeDecrypted, &text,  &text, 15000);
//someText afterwards doesn't contain the correctly decrypted text/has a different content than plainText
//rtn is always 0, i.e. no error is returned
//sizeDecrypted is 8

Answer 1

No, this will not work.

From the repo https://github.com/Mbed-TLS/mbedtls we look at library/rsa.c to get the body of mbedtls_rsa_rsaes_oaep_encrypt:

int mbedtls_rsa_rsaes_oaep_encrypt(mbedtls_rsa_context *ctx,
                                   int (*f_rng)(void *, unsigned char *, size_t),
                                   void *p_rng,
                                   const unsigned char *label, size_t label_len,
                                   size_t ilen,
                                   const unsigned char *input,
                                   unsigned char *output)
{
    size_t olen;
    int ret = MBEDTLS_ERR_ERROR_CORRUPTION_DETECTED;
    unsigned char *p = output;
    unsigned int hlen;

    if (f_rng == NULL) {
        return MBEDTLS_ERR_RSA_BAD_INPUT_DATA;
    }

    hlen = mbedtls_md_get_size_from_type((mbedtls_md_type_t) ctx->hash_id);
    if (hlen == 0) {
        return MBEDTLS_ERR_RSA_BAD_INPUT_DATA;
    }

    olen = ctx->len;

    /* first comparison checks for overflow */
    if (ilen + 2 * hlen + 2 < ilen || olen < ilen + 2 * hlen + 2) {
        return MBEDTLS_ERR_RSA_BAD_INPUT_DATA;
    }

    memset(output, 0, olen);

    *p++ = 0;

    /* Generate a random octet string seed */
    if ((ret = f_rng(p_rng, p, hlen)) != 0) {
        return MBEDTLS_ERROR_ADD(MBEDTLS_ERR_RSA_RNG_FAILED, ret);
    }

    p += hlen;

    /* Construct DB */
    ret = compute_hash((mbedtls_md_type_t) ctx->hash_id, label, label_len, p);
    if (ret != 0) {
        return ret;
    }
    p += hlen;
    p += olen - 2 * hlen - 2 - ilen;
    *p++ = 1;
    if (ilen != 0) {
        memcpy(p, input, ilen);
    }

    /* maskedDB: Apply dbMask to DB */
    if ((ret = mgf_mask(output + hlen + 1, olen - hlen - 1, output + 1, hlen,
                        (mbedtls_md_type_t) ctx->hash_id)) != 0) {
        return ret;
    }

    /* maskedSeed: Apply seedMask to seed */
    if ((ret = mgf_mask(output + 1, hlen, output + hlen + 1, olen - hlen - 1,
                        (mbedtls_md_type_t) ctx->hash_id)) != 0) {
        return ret;
    }

    return mbedtls_rsa_public(ctx, output, output);
}

Note the following:

After some initial integrity checks, we have:

memset(output, 0, olen);

If output == input, this trashes the input buffer (or portion thereof) before the input buffer is examined.

So, the buffers must be distinct and non-overlapping.