How to throw while using a CryptoStream without causing a CryptoGraphicException

Question

I'm reading an encrypted file and want to throw a specific error whenever the password is incorrect. Since decrypting with an incorrect password would simply succeed but return gibberish data, I'd like to determine if the password is correct.

The way I do this is I write a "magic value" to the encrypted file; let's say I write the value MaGiC to the file. Whenever I'm decrypting the file I ensure the first 5 bytes contain MaGiC. If not, the password used to decrypt the file must be wrong and so I throw an InvalidPasswordException.

MAGICHEADER = new byte[] { 0x4D, 0x61, 0x47, 0x69, 0x43 };      // "MaGiC"

using (var fs = File.OpenRead(path))
using (var algo = ...)
using (var cs = new CryptoStream(fs, algo.CreateDecryptor(), CryptoStreamMode.Read))
{
    var hdr = new byte[MAGICHEADER.Length]; // Buffer
    cryptoStream.Read(hdr, 0, hdr.Length);  // Read magic header from encrypted stream

    // If the decrypted data doesn't equal the magic header this means the password is wrong
    if (!MAGICHEADER.SequenceEqual(hdr))
        throw new InvalidPasswordException();

    // ... read rest of file
}

For brevity I've left out a lot of stuff (which algorithm, key, IV, blocksize, padding, mode etc.). For this particular case, what may be important: I'm using PKCS7 for padding (but I'm quite sure all padding will cause the described issue).

Now, on the calling side I wrote code like:

try {
    var data = ReadFile("C:\foo\bar", "INCORRECTPASSWORD");
    // ... Do stuff with data
} catch (InvalidPasswordException) {
    // Oh noes! Wrong password! Ask to enter password again
}

However, to my surprise the InvalidPasswordException was not caught; I did get an exception but of type CryptographicException: Padding is invalid and cannot be removed..

When I debug the issue I see the InvalidPasswordException being thrown. I also think I know what's going on: the InvalidPasswordException is thrown, the using causes the CryptoStream to be disposed and the CryptoStream determines that since we're not at the end of the stream what is left must be "padding" which then causes an exception. Or at least something along those lines.

To 'force' MY exception getting thrown I could do something like:

if (!MAGICHEADER.SequenceEqual(hdr)) {
    try { cs.FlushFinalBlock(); } catch { }  // YUK!!
    throw new InvalidPasswordException();
}

Or:

if (!MAGICHEADER.SequenceEqual(hdr)) {
    try { cs.Dispose(); } catch { }  // YUK!!
    throw new InvalidPasswordException();
}

I was, however, hoping for a more elegant solution. Is there a way I can initialize the CryptoStream not to throw on disposal? Or a way to 'close' the CryptoStream cleanly without it throwing? I've tried Close(), Clear() etc. but all of these methods also cause the CryptoGraphicException forcing me to wrap them in a try { ... } catch { ... } just before throwing my own InvalidPasswordException.