Memleak using Streamreader

Question

I was just trying to programmatically analyze a file when I found something strange (prob. did a mistake somewhere). I got a list of files, opening on after another and analyzing the contents of those files. When I do this I get crazy amounts of memory usage even tho I'm not trying to save anything.

foreach(var file in files)
{
    using(StreamReader sr = File.OpenText(file))
    {
        string s = sr.ReadToEnd();
        //do some checks
        sr.Close(); //do I have to do this?
    }
}

Now the problem is, lets say I'm checking 5 files. Each files size is 500mb. With this method my program would take around 2.5gb memory even tho I'm pretty sure I'm closing the file/stream with the using and also the sr.Close();.

I've tried to not use .ReadToEnd() but that makes it very very slow. Also I tried closing the stream with using aswell as .Close(). I tried StreamReader sr = new StreamReader(file).

I prob. got a thinking mistake somewhere so I would really appreciate anything that puts me in the right direction.

Thanks in advance! Greets Myridor

Answer 1

When I do this I get crazy amounts of memory usage even tho I'm not trying to save anything.

You cannot check for memory leaks by just checking the memory of the process, since this will include garbage that is ready for cleanup. The runtime till do garbage collection when it needs memory. If you have lots of ram in your system it might decide to just request more memory from the OS instead of.

You should use a memory profiler to investigate memory issues, this should have a function to force a GC in an application and monitor the usage. If you think that is to complicated you could just add a GC.Collec()t to your loop, but you should probably remote it after you are done testing.

If you are using using you do not need to explicitly close the file, that should be done by the Dispose method in any reasonable API.

The //do some checks might keep a reference to the string that may cause high usage. That might be intentional, or constitute a memory leak. Use a memory profiler to check for if any large strings are retained.

But if the goal is to minimize the memory usage you should try to rewrite your code to stream data instead. I.e. read the file line by line, or into a reused buffer, and do all the processing you need to do on that line, possibly parsing data into some more compact representation. But this approach is not always feasible.