diff options
author | Clyne Sullivan <tullivan99@gmail.com> | 2019-02-28 17:04:22 -0500 |
---|---|---|
committer | Clyne Sullivan <tullivan99@gmail.com> | 2019-02-28 17:04:22 -0500 |
commit | d6869d1ec4bd24cd2c3eafa534f0849b25ec5607 (patch) | |
tree | 79e54ed27b39c31864895535d11399708d5a45c0 /arduino/libraries/FileSystem/src/littlefs/SPEC.md | |
parent | 614ee97bf3a2270c413527a7f35c54cbecd9e601 (diff) |
added basic code
Diffstat (limited to 'arduino/libraries/FileSystem/src/littlefs/SPEC.md')
-rwxr-xr-x | arduino/libraries/FileSystem/src/littlefs/SPEC.md | 370 |
1 files changed, 370 insertions, 0 deletions
diff --git a/arduino/libraries/FileSystem/src/littlefs/SPEC.md b/arduino/libraries/FileSystem/src/littlefs/SPEC.md new file mode 100755 index 0000000..2a1f9ec --- /dev/null +++ b/arduino/libraries/FileSystem/src/littlefs/SPEC.md @@ -0,0 +1,370 @@ +## The little filesystem technical specification + +This is the technical specification of the little filesystem. This document +covers the technical details of how the littlefs is stored on disk for +introspection and tooling development. This document assumes you are +familiar with the design of the littlefs, for more info on how littlefs +works check out [DESIGN.md](DESIGN.md). + +``` + | | | .---._____ + .-----. | | +--|o |---| littlefs | +--| |---| | + '-----' '----------' + | | | +``` + +## Some important details + +- The littlefs is a block-based filesystem. This is, the disk is divided into + an array of evenly sized blocks that are used as the logical unit of storage + in littlefs. Block pointers are stored in 32 bits. + +- There is no explicit free-list stored on disk, the littlefs only knows what + is in use in the filesystem. + +- The littlefs uses the value of 0xffffffff to represent a null block-pointer. + +- All values in littlefs are stored in little-endian byte order. + +## Directories / Metadata pairs + +Metadata pairs form the backbone of the littlefs and provide a system for +atomic updates. Even the superblock is stored in a metadata pair. + +As their name suggests, a metadata pair is stored in two blocks, with one block +acting as a redundant backup in case the other is corrupted. These two blocks +could be anywhere in the disk and may not be next to each other, so any +pointers to directory pairs need to be stored as two block pointers. + +Here's the layout of metadata blocks on disk: + +| offset | size | description | +|--------|---------------|----------------| +| 0x00 | 32 bits | revision count | +| 0x04 | 32 bits | dir size | +| 0x08 | 64 bits | tail pointer | +| 0x10 | size-16 bytes | dir entries | +| 0x00+s | 32 bits | CRC | + +**Revision count** - Incremented every update, only the uncorrupted +metadata-block with the most recent revision count contains the valid metadata. +Comparison between revision counts must use sequence comparison since the +revision counts may overflow. + +**Dir size** - Size in bytes of the contents in the current metadata block, +including the metadata-pair metadata. Additionally, the highest bit of the +dir size may be set to indicate that the directory's contents continue on the +next metadata-pair pointed to by the tail pointer. + +**Tail pointer** - Pointer to the next metadata-pair in the filesystem. +A null pair-pointer (0xffffffff, 0xffffffff) indicates the end of the list. +If the highest bit in the dir size is set, this points to the next +metadata-pair in the current directory, otherwise it points to an arbitrary +metadata-pair. Starting with the superblock, the tail-pointers form a +linked-list containing all metadata-pairs in the filesystem. + +**CRC** - 32 bit CRC used to detect corruption from power-lost, from block +end-of-life, or just from noise on the storage bus. The CRC is appended to +the end of each metadata-block. The littlefs uses the standard CRC-32, which +uses a polynomial of 0x04c11db7, initialized with 0xffffffff. + +Here's an example of a simple directory stored on disk: +``` +(32 bits) revision count = 10 (0x0000000a) +(32 bits) dir size = 154 bytes, end of dir (0x0000009a) +(64 bits) tail pointer = 37, 36 (0x00000025, 0x00000024) +(32 bits) CRC = 0xc86e3106 + +00000000: 0a 00 00 00 9a 00 00 00 25 00 00 00 24 00 00 00 ........%...$... +00000010: 22 08 00 03 05 00 00 00 04 00 00 00 74 65 61 22 "...........tea" +00000020: 08 00 06 07 00 00 00 06 00 00 00 63 6f 66 66 65 ...........coffe +00000030: 65 22 08 00 04 09 00 00 00 08 00 00 00 73 6f 64 e"...........sod +00000040: 61 22 08 00 05 1d 00 00 00 1c 00 00 00 6d 69 6c a"...........mil +00000050: 6b 31 22 08 00 05 1f 00 00 00 1e 00 00 00 6d 69 k1"...........mi +00000060: 6c 6b 32 22 08 00 05 21 00 00 00 20 00 00 00 6d lk2"...!... ...m +00000070: 69 6c 6b 33 22 08 00 05 23 00 00 00 22 00 00 00 ilk3"...#..."... +00000080: 6d 69 6c 6b 34 22 08 00 05 25 00 00 00 24 00 00 milk4"...%...$.. +00000090: 00 6d 69 6c 6b 35 06 31 6e c8 .milk5.1n. +``` + +A note about the tail pointer linked-list: Normally, this linked-list is +threaded through the entire filesystem. However, after power-loss this +linked-list may become out of sync with the rest of the filesystem. +- The linked-list may contain a directory that has actually been removed +- The linked-list may contain a metadata pair that has not been updated after + a block in the pair has gone bad. + +The threaded linked-list must be checked for these errors before it can be +used reliably. Fortunately, the threaded linked-list can simply be ignored +if littlefs is mounted read-only. + +## Entries + +Each metadata block contains a series of entries that follow a standard +layout. An entry contains the type of the entry, along with a section for +entry-specific data, attributes, and a name. + +Here's the layout of entries on disk: + +| offset | size | description | +|---------|------------------------|----------------------------| +| 0x0 | 8 bits | entry type | +| 0x1 | 8 bits | entry length | +| 0x2 | 8 bits | attribute length | +| 0x3 | 8 bits | name length | +| 0x4 | entry length bytes | entry-specific data | +| 0x4+e | attribute length bytes | system-specific attributes | +| 0x4+e+a | name length bytes | entry name | + +**Entry type** - Type of the entry, currently this is limited to the following: +- 0x11 - file entry +- 0x22 - directory entry +- 0x2e - superblock entry + +Additionally, the type is broken into two 4 bit nibbles, with the upper nibble +specifying the type's data structure used when scanning the filesystem. The +lower nibble clarifies the type further when multiple entries share the same +data structure. + +The highest bit is reserved for marking the entry as "moved". If an entry +is marked as "moved", the entry may also exist somewhere else in the +filesystem. If the entry exists elsewhere, this entry must be treated as +though it does not exist. + +**Entry length** - Length in bytes of the entry-specific data. This does +not include the entry type size, attributes, or name. The full size in bytes +of the entry is 4 + entry length + attribute length + name length. + +**Attribute length** - Length of system-specific attributes in bytes. Since +attributes are system specific, there is not much guarantee on the values in +this section, and systems are expected to work even when it is empty. See the +[attributes](#entry-attributes) section for more details. + +**Name length** - Length of the entry name. Entry names are stored as UTF8, +although most systems will probably only support ASCII. Entry names can not +contain '/' and can not be '.' or '..' as these are a part of the syntax of +filesystem paths. + +Here's an example of a simple entry stored on disk: +``` +(8 bits) entry type = file (0x11) +(8 bits) entry length = 8 bytes (0x08) +(8 bits) attribute length = 0 bytes (0x00) +(8 bits) name length = 12 bytes (0x0c) +(8 bytes) entry data = 05 00 00 00 20 00 00 00 +(12 bytes) entry name = smallavacado + +00000000: 11 08 00 0c 05 00 00 00 20 00 00 00 73 6d 61 6c ........ ...smal +00000010: 6c 61 76 61 63 61 64 6f lavacado +``` + +## Superblock + +The superblock is the anchor for the littlefs. The superblock is stored as +a metadata pair containing a single superblock entry. It is through the +superblock that littlefs can access the rest of the filesystem. + +The superblock can always be found in blocks 0 and 1, however fetching the +superblock requires knowing the block size. The block size can be guessed by +searching the beginning of disk for the string "littlefs", although currently +the filesystems relies on the user providing the correct block size. + +The superblock is the most valuable block in the filesystem. It is updated +very rarely, only during format or when the root directory must be moved. It +is encouraged to always write out both superblock pairs even though it is not +required. + +Here's the layout of the superblock entry: + +| offset | size | description | +|--------|------------------------|----------------------------------------| +| 0x00 | 8 bits | entry type (0x2e for superblock entry) | +| 0x01 | 8 bits | entry length (20 bytes) | +| 0x02 | 8 bits | attribute length | +| 0x03 | 8 bits | name length (8 bytes) | +| 0x04 | 64 bits | root directory | +| 0x0c | 32 bits | block size | +| 0x10 | 32 bits | block count | +| 0x14 | 32 bits | version | +| 0x18 | attribute length bytes | system-specific attributes | +| 0x18+a | 8 bytes | magic string ("littlefs") | + +**Root directory** - Pointer to the root directory's metadata pair. + +**Block size** - Size of the logical block size used by the filesystem. + +**Block count** - Number of blocks in the filesystem. + +**Version** - The littlefs version encoded as a 32 bit value. The upper 16 bits +encodes the major version, which is incremented when a breaking-change is +introduced in the filesystem specification. The lower 16 bits encodes the +minor version, which is incremented when a backwards-compatible change is +introduced. Non-standard Attribute changes do not change the version. This +specification describes version 1.1 (0x00010001), which is the first version +of littlefs. + +**Magic string** - The magic string "littlefs" takes the place of an entry +name. + +Here's an example of a complete superblock: +``` +(32 bits) revision count = 3 (0x00000003) +(32 bits) dir size = 52 bytes, end of dir (0x00000034) +(64 bits) tail pointer = 3, 2 (0x00000003, 0x00000002) +(8 bits) entry type = superblock (0x2e) +(8 bits) entry length = 20 bytes (0x14) +(8 bits) attribute length = 0 bytes (0x00) +(8 bits) name length = 8 bytes (0x08) +(64 bits) root directory = 3, 2 (0x00000003, 0x00000002) +(32 bits) block size = 512 bytes (0x00000200) +(32 bits) block count = 1024 blocks (0x00000400) +(32 bits) version = 1.1 (0x00010001) +(8 bytes) magic string = littlefs +(32 bits) CRC = 0xc50b74fa + +00000000: 03 00 00 00 34 00 00 00 03 00 00 00 02 00 00 00 ....4........... +00000010: 2e 14 00 08 03 00 00 00 02 00 00 00 00 02 00 00 ................ +00000020: 00 04 00 00 01 00 01 00 6c 69 74 74 6c 65 66 73 ........littlefs +00000030: fa 74 0b c5 .t.. +``` + +## Directory entries + +Directories are stored in entries with a pointer to the first metadata pair +in the directory. Keep in mind that a directory may be composed of multiple +metadata pairs connected by the tail pointer when the highest bit in the dir +size is set. + +Here's the layout of a directory entry: + +| offset | size | description | +|--------|------------------------|-----------------------------------------| +| 0x0 | 8 bits | entry type (0x22 for directory entries) | +| 0x1 | 8 bits | entry length (8 bytes) | +| 0x2 | 8 bits | attribute length | +| 0x3 | 8 bits | name length | +| 0x4 | 64 bits | directory pointer | +| 0xc | attribute length bytes | system-specific attributes | +| 0xc+a | name length bytes | directory name | + +**Directory pointer** - Pointer to the first metadata pair in the directory. + +Here's an example of a directory entry: +``` +(8 bits) entry type = directory (0x22) +(8 bits) entry length = 8 bytes (0x08) +(8 bits) attribute length = 0 bytes (0x00) +(8 bits) name length = 3 bytes (0x03) +(64 bits) directory pointer = 5, 4 (0x00000005, 0x00000004) +(3 bytes) name = tea + +00000000: 22 08 00 03 05 00 00 00 04 00 00 00 74 65 61 "...........tea +``` + +## File entries + +Files are stored in entries with a pointer to the head of the file and the +size of the file. This is enough information to determine the state of the +CTZ skip-list that is being referenced. + +How files are actually stored on disk is a bit complicated. The full +explanation of CTZ skip-lists can be found in [DESIGN.md](DESIGN.md#ctz-skip-lists). + +A terribly quick summary: For every nth block where n is divisible by 2^x, +the block contains a pointer to block n-2^x. These pointers are stored in +increasing order of x in each block of the file preceding the data in the +block. + +The maximum number of pointers in a block is bounded by the maximum file size +divided by the block size. With 32 bits for file size, this results in a +minimum block size of 104 bytes. + +Here's the layout of a file entry: + +| offset | size | description | +|--------|------------------------|------------------------------------| +| 0x0 | 8 bits | entry type (0x11 for file entries) | +| 0x1 | 8 bits | entry length (8 bytes) | +| 0x2 | 8 bits | attribute length | +| 0x3 | 8 bits | name length | +| 0x4 | 32 bits | file head | +| 0x8 | 32 bits | file size | +| 0xc | attribute length bytes | system-specific attributes | +| 0xc+a | name length bytes | directory name | + +**File head** - Pointer to the block that is the head of the file's CTZ +skip-list. + +**File size** - Size of file in bytes. + +Here's an example of a file entry: +``` +(8 bits) entry type = file (0x11) +(8 bits) entry length = 8 bytes (0x08) +(8 bits) attribute length = 0 bytes (0x00) +(8 bits) name length = 12 bytes (0x03) +(32 bits) file head = 543 (0x0000021f) +(32 bits) file size = 256 KB (0x00040000) +(12 bytes) name = largeavacado + +00000000: 11 08 00 0c 1f 02 00 00 00 00 04 00 6c 61 72 67 ............larg +00000010: 65 61 76 61 63 61 64 6f eavacado +``` + +## Entry attributes + +Each dir entry can have up to 256 bytes of system-specific attributes. Since +these attributes are system-specific, they may not be portable between +different systems. For this reason, all attributes must be optional. A minimal +littlefs driver must be able to get away with supporting no attributes at all. + +For some level of portability, littlefs has a simple scheme for attributes. +Each attribute is prefixes with an 8-bit type that indicates what the attribute +is. The length of attributes may also be determined from this type. Attributes +in an entry should be sorted based on portability, since attribute parsing +will end when it hits the first attribute it does not understand. + +Each system should choose a 4-bit value to prefix all attribute types with to +avoid conflicts with other systems. Additionally, littlefs drivers that support +attributes should provide a "ignore attributes" flag to users in case attribute +conflicts do occur. + +Attribute types prefixes with 0x0 and 0xf are currently reserved for future +standard attributes. Standard attributes will be added to this document in +that case. + +Here's an example of non-standard time attribute: +``` +(8 bits) attribute type = time (0xc1) +(72 bits) time in seconds = 1506286115 (0x0059c81a23) + +00000000: c1 23 1a c8 59 00 .#..Y. +``` + +Here's an example of non-standard permissions attribute: +``` +(8 bits) attribute type = permissions (0xc2) +(16 bits) permission bits = rw-rw-r-- (0x01b4) + +00000000: c2 b4 01 ... +``` + +Here's what a dir entry may look like with these attributes: +``` +(8 bits) entry type = file (0x11) +(8 bits) entry length = 8 bytes (0x08) +(8 bits) attribute length = 9 bytes (0x09) +(8 bits) name length = 12 bytes (0x0c) +(8 bytes) entry data = 05 00 00 00 20 00 00 00 +(8 bits) attribute type = time (0xc1) +(72 bits) time in seconds = 1506286115 (0x0059c81a23) +(8 bits) attribute type = permissions (0xc2) +(16 bits) permission bits = rw-rw-r-- (0x01b4) +(12 bytes) entry name = smallavacado + +00000000: 11 08 09 0c 05 00 00 00 20 00 00 00 c1 23 1a c8 ........ ....#.. +00000010: 59 00 c2 b4 01 73 6d 61 6c 6c 61 76 61 63 61 64 Y....smallavacad +00000020: 6f o +``` |