Reading data from a file that contains special characters using `io:fileReadString`

Question

I'm reading a file with the extension .p12, which contains special characters. I want to verify that io:fileReadString accurately returns the data from the file. That is,

string inputData = check io:fileReadString(inputFile);
check io:fileWriteString(outputFile, inputData);
string outputData = check io:fileReadString(outputFile);
test:assertEquals(inputData, outputData);

Answer 1

io:fileReadString assumes you are reading a text file encoded with UTF-8 encoding. Furthermore, it strips away the carriage return (\n or \r) from the end of the file, which means if you read a file using io:fileReadString and write it back even if it is valid UTF-8 it could still be different since it stripped away the carriage return at the end of the file. Also if the file in question has byte sequences that are not valid UTF-8 byte sequences it may write completely different values to the file. To better illustrate the point consider the example,

import ballerina/io;

public function main() returns error? {
    byte[] seq = [97, 161, 10, 98, 10]; // <invalid>, \n, b, \n
    string fileName = "test.b";
    check io:fileWriteBytes(fileName, seq);
    byte[] data = check io:fileReadBytes(fileName);
    io:println(data); // [97, 161, 10, 98, 10]
    string str = check io:fileReadString(fileName);
    io:println(str); // a<invalidchar>\nb (note no \n at the end)
    check io:fileWriteString(fileName, str);
    data = check io:fileReadBytes(fileName); // [97, 239, 191, 189, 10, 98]
    io:println(data); // [97, 10, 98]
}

Therefore if you are reading a file that is not valid UTF-8 or need to rewrite the exact content back you must use the io:fileReadBytes read the file and write it back using io:fileWriteBytes