Azure KeyVault Key Client returns JSONWebKey on getKey method. How to get private key as blob or buffer?

Question

I'm using the @azure/keyvault-keys package to get the pem private key I uploaded to the Azure KeyVault. Now I would like to retrieve that and connect to SFTP like this

return this.sftpClient.connect({
  host,
  port,
  username,
  privateKey: secret.key,
});

Where private key is defined as

    /** Buffer or string that contains a private key for either key-based or hostbased user authentication (OpenSSH format). */
privateKey?: Buffer | string | undefined;

I'm however unable to figure out a way to get the keyvault uploaded PEM file as I would locally using

privateKey: fs.readFileSync('/path/id_dsa_key.pem'),

How can I utilize Azure Key Vault to store my private key and then connect to the SFTP client with this key vault key?