Converting char from CP437 encoding to UTF-8 encoding always yields the same character code, thus not the same character

Question

Problem

I'm trying to convert a character and/or byte array from the CP437 encoding to UTF-8 (Encoding.UTF8). The problem is that no matter what I try the code always yields the same character code, but since the two encodings have a different set of characters mapped to the character codes the resulting char is not the same.

As an example I'm trying to convert the character with char code 3 from CP437 (a heart: ♥) to UTF-8, and I still want it to be the same character. However when converting to UTF-8 it still uses char code 3 which results in a control character called ETX (see UTF-8's codepage layout for a list of characters).

My attempts

Here are some of my attempts:

(General code)

Public Shared ReadOnly CP437 As Encoding = Encoding.GetEncoding("IBM437")
Public Shared ReadOnly BytesToConvert As Byte() = New Byte(3 - 1) {3, 4, 5} 'Characters: ♥, ♦, ♣.

Public Sub DebugEncodedArray(ByVal Bytes As Byte(), ByVal Encoding As Encoding)
    Dim ResultingString As String = Encoding.GetString(Bytes)
    MessageBox.Show( _
            String.Format("Encoding: {1}{0}" & _
                          "String: ""{2}""{0}" & _
                          "Bytes: {{{3}}}{0}", _
                          Environment.NewLine, _
                          Encoding.EncodingName, _
                          ResultingString, _
                          String.Join(", ", Bytes)), _
        "Debug", MessageBoxButtons.OK, MessageBoxIcon.Information _
    )
End Sub

Using Encoding.Convert():

Dim ConvertedBytes As Byte() = Encoding.Convert(CP437, Encoding.UTF8, BytesToConvert)
DebugEncodedArray(ConvertedBytes, Encoding.UTF8)

Using a StreamWriter, writing to a MemoryStream with a specific encoding:

Using MStream As New MemoryStream(16)
    Using Writer As New StreamWriter(MStream, CP437)
        Writer.Write(CP437.GetChars(BytesToConvert))
    End Using

    Dim UTF8Bytes As Byte() = Encoding.Convert(CP437, Encoding.UTF8, MStream.ToArray())
    DebugEncodedArray(UTF8Bytes, Encoding.UTF8)
End Using

Writing to a file, then reading it and convert the bytes (not optimal for what I need this code for):

File.WriteAllText("C:\Users\Vincent\Desktop\test.txt", CP437.GetString(BytesToConvert), CP437)

Dim FileBytes As Byte() = File.ReadAllBytes("C:\Users\Vincent\Desktop\test.txt")
Dim UTF8Bytes As Byte() = Encoding.Convert(CP437, Encoding.UTF8, FileBytes)

DebugEncodedArray(UTF8Bytes, Encoding.UTF8)

Results

All the above attempts give the same result:

and also if I pass CP437 to DebugEncodedArray() instead of Encoding.UTF8:

Expected result

The result I am expecting is:

Dim UTF8Bytes As Byte() = Encoding.UTF8.GetBytes("♥♦♣")
DebugEncodedArray(UTF8Bytes, Encoding.UTF8)

Any clues on what I'm doing wrong?

Answer 1

The low range of CP437 is contextual. I think you have proven that for 1-31 & 127 you are going to need a simple lookup as .Net is interpreting them in the control code context not in the graphical context - i.e. ◙ (0xA) is \n not the equivalent Unicode code point for that graphic.

Answer 2

(Reference for future readers) This is how I finally managed to solve my problem using Alex K.'s suggestion:

Dim Heart As Char = Convert.ToChar(CP437LookupTable(3)) 'Results in: ♥. YAY!

Lookup table:

'Lookup table for Codepage 437-to-Unicode character codes.
Private Shared ReadOnly CP437LookupTable As Integer() = _
    New Integer(256 - 1) { _
        0, 9786, 9787, 9829, 9830, 9827, 9824, _
        8226, 9688, 9675, 9689, 9794, 9792, 9834, 9835, _
        9788, 9658, 9668, 8597, 8252, 182, 167, 9644, _
        8616, 8593, 8595, 8594, 8592, 8735, 8596, 9650, _
        9660, 32, 33, 34, 35, 36, 37, 38, _
        39, 40, 41, 42, 43, 44, 45, 46, _
        47, 48, 49, 50, 51, 52, 53, 54, _
        55, 56, 57, 58, 59, 60, 61, 62, _
        63, 64, 65, 66, 67, 68, 69, 70, _
        71, 72, 73, 74, 75, 76, 77, 78, _
        79, 80, 81, 82, 83, 84, 85, 86, _
        87, 88, 89, 90, 91, 92, 93, 94, _
        95, 96, 97, 98, 99, 100, 101, 102, _
        103, 104, 105, 106, 107, 108, 109, 110, _
        111, 112, 113, 114, 115, 116, 117, 118, _
        119, 120, 121, 122, 123, 124, 125, 126, _
        8962, 199, 252, 233, 226, 228, 224, 229, _
        231, 234, 235, 232, 239, 238, 236, 196, _
        197, 201, 230, 198, 244, 246, 242, 251, _
        249, 255, 214, 220, 162, 163, 165, 8359, _
        402, 225, 237, 243, 250, 241, 209, 170, _
        186, 191, 8976, 172, 189, 188, 161, 171, _
        187, 9617, 9618, 9619, 9474, 9508, 9569, 9570, _
        9558, 9557, 9571, 9553, 9559, 9565, 9564, 9563, _
        9488, 9492, 9524, 9516, 9500, 9472, 9532, 9566, _
        9567, 9562, 9556, 9577, 9574, 9568, 9552, 9580, _
        9575, 9576, 9572, 9573, 9561, 9560, 9554, 9555, _
        9579, 9578, 9496, 9484, 9608, 9604, 9612, 9616, _
        9600, 945, 223, 915, 960, 931, 963, 181, _
        964, 934, 920, 937, 948, 8734, 966, 949, _
        8745, 8801, 177, 8805, 8804, 8992, 8993, 247, _
        8776, 176, 8729, 183, 8730, 8319, 178, 9632, _
        160 _
    }