I am writing a program to read and write a file at the same time. More specifically, all write operations are appending new data to the end of the file and all read operations are reading random positions of the file.
I am thinking of creating memory-mapped file (using
mmap) to achieve efficient read while writing via append (mode
open). However, I don’t think this will work because the memory-mapped file cannot change in size*, unless I
munmap and then
munmap and then
mmap the file again" works, it has many downsides. Not only I need to perform 2 syscalls after every write (or before every read), which hurts performance, the base address returned from the next
mmap call after
munmap could be different from the previous one. Since I am planning to have other in-memory data structure storing pointers to specific offset of this memory mapped file, it could be very inconvenient.
Are there more elegant and efficient ways of doing this? The program will be mostly running on Linux (but solutions with portability to other POSIX systems are preferred). I have read through the following posts, but none of them seems to give a definitive answer.
My intuition is to use
mmap to "reserve" the file with a size that is large enough to accommodate the growth of file, say a few hundred of GiB (that is a very reasonable assumption in my use case). And then somehow reflect the change of file size in this mapped memory without invalidating it with
munmap. However, I am aware that accessing data beyond the real file boundary could result in a bus error. And the documentation isn’t clear about whether changes in file size will get reflected.
*I am not 100% sure about this, but I couldn’t find any source of elegantly changing the size of memory-mapped file.
After some experimentations, I found a way to make it work.
mmap the file with
PROT_NONE and a large enough size. For 64-bit systems, it can be as large
1L << 46 (64TB). This does NOT consume physical memory* (at least on Linux). It will consume address space (virtual memory) for this process.
void* ptr = mmap(NULL, (1L << 40), PROT_NONE, MAP_SHARED, fd, 0);
Then, give read (and/or write) permission to the part of memory within file length using
mprotect. Note that size need to be aligned with page size (which can be obtained by
sysconf(_SC_PAGESIZE), usually 4096).
mprotect(ptr, aligned_size, PROT_READ | PROT_WRITE);
However, if file size is not page-size aligned, reading the portion within mapped region (with
PROT_READ permission) but beyond file length will trigger a bus error, as documented on
Then you can use either file descriptor
fd or the mapped memory to read and write file. Remember to use
msync to persist the data after writing to it. The memory-mapped page with
PROT_READ permission should get the latest file content (if you write to it)**. The newly mapped page with
mprotect will also get the newly updated page.
Depending on the application, you might want to use
ftruncate to make the file size aligned to system page size for the best performance. You might also want to use
MADV_SEQUENTIAL to improve performance when reading those pages.
*This behavior is not mentioned on the manual of
mmap. However, since
PROT_NONE implies those pages are not accessible in anyway, it’s trivial for any OS implementation to not allocating any physical memory to it at all.
**This behavior of memory region mapped before a file write getting updated after the write is completed (
msync) is also not mentioned on the manual (or at least I did not see it). But it seems to be the case at least on recent Linux kernels (4.x onward).
Answered By – lewisxy