C# BinaryReader performanceboost alternatives

Question

Maybe a question that has been asked many times, but still struggling with this one.

situation: C# library that contains an interpreter/parser that opens files and interprets/parses them with a huge implementation of logic with heavy use of:

BinaryReader
.PeekChar()
.Read()
.ReadBytes()
.ReadByte()
.BaseStream.Position
.BaseStream.Length

The BinaryReader receives it stream from a FileStream. File size can be several KB's, MB's or even GB's. Pieces of extracted code:

    using (Stream s = File.OpenRead(path)) { // ...
    using (var br = new BinaryReader(context.Input, new ASCIIEncoding())) {  // ...


// context.Input:
public Stream Input { get; set; }

Parsing a file of 3MB takes about 20 seconds, which is very slow. The most time has been lost with the BinaryReader.Read() and PeekChar() functions.

I tried to optimize as much as possible, caching e.g. br.BaseStream.Length when using it in loops. Still the biggest problem is with .Read() and PeekChar() being called so much and I can't change that part of the logic.

Next I thought to increase the buffer of the filestream:

using (Stream s = new FileStream(pclFile, FileMode.Open, FileAccess.Read, FileShare.Read, 4096)) // profiler time: 13784
using (Stream s = new FileStream(pclFile, FileMode.Open, FileAccess.Read, FileShare.Read, 8192)) // profiler time: 13863
using (Stream s = new FileStream(pclFile, FileMode.Open, FileAccess.Read, FileShare.Read, 16384)) // profiler time: 13937
using (Stream s = new FileStream(pclFile, FileMode.Open, FileAccess.Read, FileShare.Read, 32768)) // profiler time: 13776
using (Stream s = new FileStream(pclFile, FileMode.Open, FileAccess.Read, FileShare.Read, 65536)) // profiler time: 13833
using (Stream s = new FileStream(pclFile, FileMode.Open, FileAccess.Read, FileShare.Read, 131072)) // profiler time: 13857
using (Stream s = new FileStream(pclFile, FileMode.Open, FileAccess.Read, FileShare.Read, 1000000)) // profiler time: 13746
using (Stream s = new FileStream(pclFile, FileMode.Open, FileAccess.Read, FileShare.Read, 2000000)) // profiler time: 13814
using (Stream s = new FileStream(pclFile, FileMode.Open, FileAccess.Read, FileShare.Read, 5000000)) // profiler time: 13736

but this didn't do much. I thought that a bigger buffer would require less access to the disk where the BinaryReader.Read(),.PeekChar() could get an advantage. But no luck there.

Next idea was using a MemoryMappedFile and creating a stream out of it:

  long length = new FileInfo(path).Length;      
  using (var mmf = MemoryMappedFile.CreateFromFile(pclFile, FileMode.Open, "pclfile", length))
  {
    using (var viewStream = mmf.CreateViewStream(0, length, MemoryMappedFileAccess.Read))
    { 
      // give this stream to the BinaryReader
    }
  }

A little bit of improvement, but still not good (let's say it is now 19,8seconds instead of 20seconds when profiling).

After this I believe the problem relies with the BinaryReader. What are the best options now? I basically need the .Read(), .ReadBytes(), ReadByte(), PeekChar(), Position and Length from a BinaryReader.

I'm on .NET Framework 4.6.2.

update:

using FileStream (File.OpenRead()) - BinaryReader:

using MemoryMappedViewStream - BinaryReader: