To load a large data file (typically larger than the available memory) into memory, the input has to be split into manageable chunks and the corresponding memory blocks allocated and deallocated. The typical way of loading a file into memory to process it could be described in pseudo-code as:
- Allocate a memory buffer of size N (bytes)
- Load the first N bytes of the input file into memory buffer.
- Process the buffer.
- Repeat loading the next N bytes from the input file until
the whole file has been read and processed.
- Free the buffer, close the file.
As for large memory allocations, this process is not very elegant and requires the programmer to think algorithms in advance so that they can be handling the data chunks separately. Cutting an algorithm into smaller parts so that they work on file sections is usually hard to do if not impossible.
Using mmap(), this process can be simpler written as:
- Map the input file to a memory pointer acting as a buffer. - Process the buffer. - Unmap the file.
No need to allocate or free any memory, the whole process is convinced that the whole file is loaded into memory and accessible through the pointer returned by mmap(). This way of loading files into memory does not require slicing the input files into manageable chunks.
From the operating system's point of view, data are loaded from the file as they are needed (paged into memory) where they are truly accessed as in a memory buffers, then paged out when not needed. This handling is invisible for the programmer who has to rely on the operating system to perform the paging efficiently.
In essence, mmap() acts as an interface to files on disk and allows to write programs working on the whole contents of a file as if it were in memory, even if the file size is bigger than the available amount of true memory.