Win64/Linux: Mmap readonly chunks (few MB) of memory at/after the end of a file

Question

Is it possible in Windows and Linux to mmap chunks of memory on a file that isn't as large as the mapping?

The file may contain only a few bytes worth of values, but I want to create multiple 1MB (in that ballpark) memory-mapping and read from this memory as the file grows. The file is written-to by the same program. I will never touch memory beyond the end of the file. But I don't want to redo the mmap calls with every new values that are appended into the file!

• Linux: I checked man 2 mmap and it doesn't appear to forbid this kind of usage.
• Windows: I checked the Windows API. The API devides the task into two functions, CreateFileMapping and MapViewOfFile. The latter is similar to mmap, and wants a handle that is returned by the former. But the size passed to MapViewOfFile is restricted to whatever size was given to CreateFileMapping! Can I simply call CreateFileMapping with whatever my max file size is going to be, let's say 100TB and Windows will let me create views of the file as the file grows till 100TB?

I checked folly's MemoryMap implementation, and it won't allow mappings larger than the file size. Neither does python's mmap. I haven't found a definitive answer for boost.interprocess.

Answer 1

Subsequently I found an answer. I would like to quote my comments made to YangXiaoPo-MSFT's answer who was kind enough to provide me with first-hand MSFT information, after which I did some research on my own.

I subsequently discovered another answer that goes into some detail on how to resize an existing file mapping to be larger. It involves directly calling the Zw / Nt functions and not using the CreateFileMapping wrapper. As far as I know, CreateFileMapping creates a special "section" object that's backed by an arbitrary file - like there are sections .data for example that are too backed by files. Usually these sections are fixed size. But the sepecial Nt functions allow extending the section.

The mentioned answer about NtExtendSection is at https://stackoverflow.com/a/55068177/34509 .

I also uttered the following

Although I will probably not map the entire file into memory - it may grow muiltiple terabytes and even on 64bit, I'm unsure whether windows will find a large enough contiguous VM segment. I will create the file mapping over the entire file, and then map only individual chunks of the file.

Answer 2

The answer is intermediate in Windows. As CreateFileMapping states:

If an application specifies a size for the file mapping object that is larger than the size of the actual named file on disk and if the page protection allows write access (that is, the flProtect parameter specifies PAGE_READWRITE or PAGE_EXECUTE_READWRITE), then the file on disk is increased to match the specified size of the file mapping object. If the file is extended, the contents of the file between the old end of the file and the new end of the file are not guaranteed to be zero; the behavior is defined by the file system. If the file on disk cannot be increased, CreateFileMapping fails and GetLastError returns ERROR_DISK_FULL.

After a file mapping object is created, the size of the file must not exceed the size of the file mapping object; if it does, not all of the file contents are available for sharing.