Best way to read a short array from disk in C#?

Question

I have to write 4GB short[] arrays to and from disk, so I have found a function to write the arrays, and I am struggling to write the code to read the array from the disk. I normally code in other languages so please forgive me if my attempt is a bit pathetic so far:

using UnityEngine;
using System.Collections;
using System.IO;

public class RWShort : MonoBehaviour {

    public static void WriteShortArray(short[] values, string path)
    {
        using (FileStream fs = new FileStream(path, FileMode.OpenOrCreate, FileAccess.Write))
        {
            using (BinaryWriter bw = new BinaryWriter(fs))
            {
                foreach (short value in values)
                {
                    bw.Write(value);
                }
            }
        }
    } //Above is fine, here is where I am confused: 


    public static short[] ReadShortArray(string path) 
    {
        byte[]  thisByteArray= File.ReadAllBytes(fileName);
        short[] thisShortArray= new short[thisByteArray.length/2];      
                for (int i = 0; i < 10; i+=2)
                {
                    thisShortArray[i]= ? convert from byte array;
                }


        return thisShortArray;
    }   
}

Answer 1

Shorts are two bytes, so you have to read in two bytes each time. I'd also recommend using a yield return like this so that you aren't trying to pull everything into memory in one go. Though if you need all of the shorts together that won't help you.. depends on what you're doing with it I guess.

void Main()
{
    short[] values = new short[] {
        1, 999, 200, short.MinValue, short.MaxValue
    };

    WriteShortArray(values, @"C:\temp\shorts.txt");

    foreach (var shortInfile in ReadShortArray(@"C:\temp\shorts.txt"))
    {
        Console.WriteLine(shortInfile);
    }
}

public static void WriteShortArray(short[] values, string path)
{
    using (FileStream fs = new FileStream(path, FileMode.OpenOrCreate, FileAccess.Write))
    {
        using (BinaryWriter bw = new BinaryWriter(fs))
        {
            foreach (short value in values)
            {
                bw.Write(value);
            }
        }
    }
}

public static IEnumerable<short> ReadShortArray(string path)
{
    using (FileStream fs = new FileStream(path, FileMode.Open, FileAccess.Read))
    using (BinaryReader br = new BinaryReader(fs))
    {
        byte[] buffer = new byte[2];
        while (br.Read(buffer, 0, 2) > 0)
            yield return (short)(buffer[0]|(buffer[1]<<8)); 
    }
}

You could also define it this way, taking advantage of the BinaryReader:

public static IEnumerable<short> ReadShortArray(string path)
{
    using (FileStream fs = new FileStream(path, FileMode.Open, FileAccess.Read))
    using (BinaryReader br = new BinaryReader(fs))
    {
        while (br.BaseStream.Position < br.BaseStream.Length)
            yield return br.ReadInt16();
    }
}

Answer 2

Memory-mapping the file is your friend, there's a MemoryMappedViewAccessor.ReadInt16 function that will allow you to directly read the data, with type short, out of the OS disk cache. Also a Write() overload that accepts an Int16. Also ReadArray and WriteArray functions if you are calling functions that need a traditional .NET array.

Overview of using Memory-mapped files in .NET on MSDN

If you want to do it with ordinary file I/O, use a block size of 1 or 2 megabytes and the Buffer.BlockCopy function to move data en masse between byte[] and short[], and use the FileStream functions that accept a byte[]. Forget about BinaryWriter or BinaryReader, forget about doing 2 bytes at a time.

It's also possible to do the I/O directly into a .NET array with the help of p/invoke, see my answer using ReadFile and passing the FileStream object's SafeFileHandle property here But even though this has no extra copies, it still shouldn't keep up with the memory-mapped ReadArray and WriteArray calls.