Memory Mapped IO
Learning Objectives
- Review of the C file interface
- Introduction to
mmap
C File Manipulation
- A higher level, more portable interface for interacting with file operations than syscalls.
- Some familiar functions:
fopenandfwrite
Fseek and Ftell
- Lets us move around the position of a
FILE* - New position specified via an
offsetand awhenceorigin - Syscall analog for working with file descriptors:
lseek
Fseek Juggle
MMAP
What is it?
- Syscall- can define a new memory mapping.
- This can apply to files or devices, letting us emulate writing through standard memory writes.
- Files are loaded lazily into memory page by page and writes can be reflected to the underlying file depending on the mapping’s settings.
- On the kernel end, only a memory mapping is created.
- The kernel/CPU is free to do whatever under the hood as long as when a process asks for a memory address a page is available.
How do we implement this? Page faults!
Remember this?
Now, the pages can be tied to file pages, instead of pages backed by physical RAM.
MMAP for IPC
Lazy MMAP
- Laziness in this context means entire files may not be mmapped.
- Parts of files are assigned to memory pages as they are needed.
- When
mmapis called, it’s possible that none of the file is loaded into memory yet.
MMAP Usage
mmap is complicated! Here is the signature:
void *mmap(void *addr, size_t length, int prot, int flags, int fd, off_t offset);
Basic mapping parameters:
-
addr: Page aligned address to start the mapping at orNULLto letmmapchoose -
length: Byte length of mapping
Common access settings:
-
prot: Memory protection (r/w/x/none) -
flags: Update visibility to other processes (ex.MAP_SHARED/MAP_PRIVATE) or define a non-file mapping (MAP_ANONYMOUS)
File mapping options:
-
fd: File descriptor used in a file mapping -
offset: Page-aligined offset in a file to begin a mapping at
See the man page for more!
Mad Mad Access Patterns
- A file search problem: given a query and a formatted file, implement a search to either find the value or report that no such value exists.
- The file size can exceed memory.
- You will solve this problem twice:
- Once with the C file interface (
lookup1.c) - Once with
mmap(lookup2.c)
- Once with the C file interface (
Background
Efficient File Navigation
- If we want to quickly navigate a file’s data, we can encode its contents using a data structure, in our case a tree.
- We translate the tree to a flat array/file by providing a serialization scheme or an encoding.
Here’s a serialization using an inorder traversal:
Another example with a level-ordered traversal:
Our Trees
We will use file offsets:
- The root will always be at offset
4in a correct file. - Each node will hold the file offsets of their children.
- Offset
0is analogous to aNULLptr.
Tree Node Structure
typedef struct {
uint32_t left_child; // offset of node containing left child
uint32_t right_child; // offset of node containing right child
// Offsets are relative to the beginning of the file.
// An offset of zero means the child does not exist.
uint32_t count; // number of times the word occurs in the data set
float price; // price of the word
char word[0]; // contents of the word, null-terminated
} BinaryTreeNode;
How does word[0] work?
Recall malloc!
// allocate 12 bytes for word
BinaryTreeNode* node1 = malloc(sizeof(BinaryTreeNode) + 12);
// allocate 20 bytes for word
BinaryTreeNode* node2 = malloc(sizeof(BinaryTreeNode) + 20);
Implementation Notes
- Use
strcmpon thewordfield of each node to traverse the tree. - You will have to handle certain errors, each with their own specific exit codes – check the docs.
Testing
-
create_datafor making your own BTRE files -
lookup1-reference/lookup2-referencefor comparison
Memory Mapped IO