|Written by: Neil Brown
|Please see MAINTAINERS file for where to send questions.
|This document describes a prototype for a new approach to providing
|overlay-filesystem functionality in Linux (sometimes referred to as
|union-filesystems). An overlay-filesystem tries to present a
|filesystem which is the result over overlaying one filesystem on top
|of the other.
|The result will inevitably fail to look exactly like a normal
|filesystem for various technical reasons. The expectation is that
|many use cases will be able to ignore these differences.
|This approach is 'hybrid' because the objects that appear in the
|filesystem do not all appear to belong to that filesystem. In many
|cases an object accessed in the union will be indistinguishable
|from accessing the corresponding object from the original filesystem.
|This is most obvious from the 'st_dev' field returned by stat(2).
|While directories will report an st_dev from the overlay-filesystem,
|all non-directory objects will report an st_dev from the lower or
|upper filesystem that is providing the object. Similarly st_ino will
|only be unique when combined with st_dev, and both of these can change
|over the lifetime of a non-directory object. Many applications and
|tools ignore these values and will not be affected.
|Upper and Lower
|An overlay filesystem combines two filesystems - an 'upper' filesystem
|and a 'lower' filesystem. When a name exists in both filesystems, the
|object in the 'upper' filesystem is visible while the object in the
|'lower' filesystem is either hidden or, in the case of directories,
|merged with the 'upper' object.
|It would be more correct to refer to an upper and lower 'directory
|tree' rather than 'filesystem' as it is quite possible for both
|directory trees to be in the same filesystem and there is no
|requirement that the root of a filesystem be given for either upper or
|The lower filesystem can be any filesystem supported by Linux and does
|not need to be writable. The lower filesystem can even be another
|overlayfs. The upper filesystem will normally be writable and if it
|is it must support the creation of trusted.* extended attributes, and
|must provide valid d_type in readdir responses, so NFS is not suitable.
|A read-only overlay of two read-only filesystems may use any
|Overlaying mainly involves directories. If a given name appears in both
|upper and lower filesystems and refers to a non-directory in either,
|then the lower object is hidden - the name refers only to the upper
|Where both upper and lower objects are directories, a merged directory
|At mount time, the two directories given as mount options "lowerdir" and
|"upperdir" are combined into a merged directory:
| mount -t overlay overlay -olowerdir=/lower,upperdir=/upper,\
|The "workdir" needs to be an empty directory on the same filesystem
|Then whenever a lookup is requested in such a merged directory, the
|lookup is performed in each actual directory and the combined result
|is cached in the dentry belonging to the overlay filesystem. If both
|actual lookups find directories, both are stored and a merged
|directory is created, otherwise only one is stored: the upper if it
|exists, else the lower.
|Only the lists of names from directories are merged. Other content
|such as metadata and extended attributes are reported for the upper
|directory only. These attributes of the lower directory are hidden.
|whiteouts and opaque directories
|In order to support rm and rmdir without changing the lower
|filesystem, an overlay filesystem needs to record in the upper filesystem
|that files have been removed. This is done using whiteouts and opaque
|directories (non-directories are always opaque).
|A whiteout is created as a character device with 0/0 device number.
|When a whiteout is found in the upper level of a merged directory, any
|matching name in the lower level is ignored, and the whiteout itself
|is also hidden.
|A directory is made opaque by setting the xattr "trusted.overlay.opaque"
|to "y". Where the upper filesystem contains an opaque directory, any
|directory in the lower filesystem with the same name is ignored.
|When a 'readdir' request is made on a merged directory, the upper and
|lower directories are each read and the name lists merged in the
|obvious way (upper is read first, then lower - entries that already
|exist are not re-added). This merged name list is cached in the
|'struct file' and so remains as long as the file is kept open. If the
|directory is opened and read by two processes at the same time, they
|will each have separate caches. A seekdir to the start of the
|directory (offset 0) followed by a readdir will cause the cache to be
|discarded and rebuilt.
|This means that changes to the merged directory do not appear while a
|directory is being read. This is unlikely to be noticed by many
|seek offsets are assigned sequentially when the directories are read.
| - read part of a directory
| - remember an offset, and close the directory
| - re-open the directory some time later
| - seek to the remembered offset
|there may be little correlation between the old and new locations in
|the list of filenames, particularly if anything has changed in the
|Readdir on directories that are not merged is simply handled by the
|underlying directory (upper or lower).
|Objects that are not directories (files, symlinks, device-special
|files etc.) are presented either from the upper or lower filesystem as
|appropriate. When a file in the lower filesystem is accessed in a way
|the requires write-access, such as opening for write access, changing
|some metadata etc., the file is first copied from the lower filesystem
|to the upper filesystem (copy_up). Note that creating a hard-link
|also requires copy_up, though of course creation of a symlink does
|The copy_up may turn out to be unnecessary, for example if the file is
|opened for read-write but the data is not modified.
|The copy_up process first makes sure that the containing directory
|exists in the upper filesystem - creating it and any parents as
|necessary. It then creates the object with the same metadata (owner,
|mode, mtime, symlink-target etc.) and then if the object is a file, the
|data is copied from the lower to the upper filesystem. Finally any
|extended attributes are copied up.
|Once the copy_up is complete, the overlay filesystem simply
|provides direct access to the newly created file in the upper
|filesystem - future operations on the file are barely noticed by the
|overlay filesystem (though an operation on the name of the file such as
|rename or unlink will of course be noticed and handled).
|Multiple lower layers
|Multiple lower layers can now be given using the the colon (":") as a
|separator character between the directory names. For example:
| mount -t overlay overlay -olowerdir=/lower1:/lower2:/lower3 /merged
|As the example shows, "upperdir=" and "workdir=" may be omitted. In
|that case the overlay will be read-only.
|The specified lower directories will be stacked beginning from the
|rightmost one and going left. In the above example lower1 will be the
|top, lower2 the middle and lower3 the bottom layer.
|The copy_up operation essentially creates a new, identical file and
|moves it over to the old name. The new file may be on a different
|filesystem, so both st_dev and st_ino of the file may change.
|Any open files referring to this inode will access the old data and
|metadata. Similarly any file locks obtained before copy_up will not
|apply to the copied up file.
|On a file opened with O_RDONLY fchmod(2), fchown(2), futimesat(2) and
|fsetxattr(2) will fail with EROFS.
|If a file with multiple hard links is copied up, then this will
|"break" the link. Changes will not be propagated to other names
|referring to the same inode.
|Symlinks in /proc/PID/ and /proc/PID/fd which point to a non-directory
|object in overlayfs will not contain valid absolute paths, only
|relative paths leading up to the filesystem's root. This will be
|fixed in the future.
|Some operations are not atomic, for example a crash during copy_up or
|rename will leave the filesystem in an inconsistent state. This will
|be addressed in the future.
|Changes to underlying filesystems
|Offline changes, when the overlay is not mounted, are allowed to either
|the upper or the lower trees.
|Changes to the underlying filesystems while part of a mounted overlay
|filesystem are not allowed. If the underlying filesystem is changed,
|the behavior of the overlay is undefined, though it will not result in
|a crash or deadlock.
|There's testsuite developed by David Howells at:
|Run as root:
| # cd unionmount-testsuite
| # ./run --ov