Skip to content

Commit 344ce0f

Browse files
jackpot51jackpot51
committed
Merge branch 'convert-readme' into 'master'
Convert the README to Markdown See merge request redox-os/tfs!91
2 parents a43dd55 + b8a0d5f commit 344ce0f

File tree

2 files changed

+186
-163
lines changed

2 files changed

+186
-163
lines changed

‎README.md

+186
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,186 @@
1+
# TFS was replaced by [RedoxFS](https://gitlab.redox-os.org/redox-os/redoxfs) and is no longer maintained, most of the features of TFS have been incorporated into RedoxFS
2+
3+
![TFS](https://rawgit.com/ticki/tfs/master/icon.svg)
4+
5+
# TFS: Next-generation file system
6+
7+
TFS is a modular, fast, and feature rich next-gen file system, employing
8+
modern techniques for high performance, high space efficiency, and high
9+
scalability.
10+
11+
TFS was created out of the need for a modern file system for Redox OS,
12+
as a replacement for ZFS, which proved to be slow to implement because
13+
of its monolithic design.
14+
15+
TFS is inspired by the ideas behind ZFS, but at the same time it aims to
16+
be modular and easier to implement.
17+
18+
TFS is not related to the file system of the same name by
19+
*terminalcloud*.
20+
21+
*While many components are complete, TFS itself is not ready for use.*
22+
23+
[![MIT/X11 permissive license.](https://img.shields.io/github/license/ticki/tfs.svg)](https://en.wikipedia.org/wiki/MIT_License)
24+
25+
![GitHub Stars](https://img.shields.io/github/stars/ticki/tfs.svg?style=social&label=Star)
26+
27+
## Design goals
28+
29+
TFS is designed with the following goals in mind:
30+
31+
- Concurrent
32+
33+
TFS contains very few locks and aims to be as suitable for
34+
multithreaded systems as possible. It makes use of multiple truly
35+
concurrent structures to manage the data, and scales linearly by the
36+
number of cores. **This is perhaps the most important feature
37+
of TFS.**
38+
39+
- Asynchronous
40+
41+
TFS is asynchronous: operations can happen independently; writes and
42+
reads from the disk need not block.
43+
44+
- Full-disk compression
45+
46+
TFS is the first file system to incorporate complete full-disk
47+
compression through a scheme we call RACC (random-access
48+
cluster compression). This means that every cluster is compressed
49+
only affecting performance slightly. It is estimated that you get
50+
60-120% more usable space.
51+
52+
- Revision history
53+
54+
TFS stores a revision history of every file without imposing
55+
extra overhead. This means that you can revert any file into an
56+
earlier version, backing up the system automatically and without
57+
imposed overhead from copying.
58+
59+
- Data integrity
60+
61+
TFS, like ZFS, stores full checksums of the file (not just
62+
metadata), and on top of that, it is done in the parent block. That
63+
means that almost all data corruption will be detected upon read.
64+
65+
- Copy-on-write semantics
66+
67+
Similarly to Btrfs and ZFS, TFS uses CoW semantics, meaning that no
68+
cluster is ever overwritten directly, but instead it is copied and
69+
written to a new cluster.
70+
71+
- O(1) recursive copies
72+
73+
Like some other file systems, TFS can do recursive copies in
74+
constant time, but there is an unique addition: TFS doesn't copy
75+
even after it is mutated. How? It maintains segments of the file
76+
individually, such that only the updated segment needs copying.
77+
78+
- Guaranteed atomicity
79+
80+
The system will never enter an inconsistent state (unless there is
81+
hardware failure), meaning that unexpected power-off won't ever
82+
damage the system.
83+
84+
- Improved caching
85+
86+
TFS puts a lot of effort into caching the disk to speed up
87+
disk accesses. It uses machine learning to learn patterns and
88+
predict future uses to reduce the number of cache misses. TFS also
89+
compresses the in-memory cache, reducing the amount of
90+
memory needed.
91+
92+
- Better file monitoring
93+
94+
CoW is very suitable for high-performance, scalable file monitoring,
95+
but unfortunately only few file systems incorporate that. TFS is one
96+
of those.
97+
98+
- All memory safe
99+
100+
TFS uses only components written in Rust. As such, memory unsafety
101+
is only possible in code marked unsafe, which is checked
102+
extra carefully.
103+
104+
- Full coverage testing
105+
106+
TFS aims to be full coverage with respect to testing. This gives
107+
relatively strong guarantees on correctness by instantly revealing
108+
large classes of bugs.
109+
110+
- SSD friendly
111+
112+
TFS tries to avoid the write limitation in SSD by repositioning
113+
dead sectors.
114+
115+
- Improved garbage collection
116+
117+
TFS uses Bloom filters for space-efficient and fast
118+
garbage collection. TFS allows the FS garbage collector to run in
119+
the background without blocking the rest of the file system.
120+
121+
## FAQ
122+
123+
### Why do you use SPECK as the default cipher?
124+
125+
- SPECK is a relatively young cipher, yet it has been subject to a lot
126+
of (ineffective) cryptanalysis, so it is relatively secure. It has
127+
really good performance and a simple implementation. Portability is
128+
an important part of the TFS design, and truly portable AES
129+
implementations without side-channel attacks is harder than many
130+
think (particularly, there are issues with SubBytes in most
131+
portable implementations). SPECK does not have this issue, and can
132+
thus be securely implemented portably with minimal effort.
133+
134+
### How similar is TFS and ZFS?
135+
136+
- Not that similar, actually. They share many of the basic ideas, but
137+
otherwise they are essentially unconnected. But ZFS' design has
138+
shaped TFS' a lot.
139+
140+
### Is TFS Redox-only?
141+
142+
- No, and it was never planned to be Redox-only.
143+
144+
### How does whole-disk compression work?
145+
146+
- Whole-disk compression is -- to my knowledge -- exclusive to TFS. It
147+
works by collecting as many "pages" (virtual data blocks) into a
148+
"cluster" (allocation unit). By doing this, the pages can be read by
149+
simply decompressing the respective cluster.
150+
151+
### Why is ZMicro so slow? Will it affect the performance of TFS?
152+
153+
- The reason ZMicro is so slow is because it works on a bit level,
154+
giving excellent compression ratio on the cost of performance. This
155+
horribly slow performance is paid back by the reduced number
156+
of writes. In fact, more than 50% of the allocations with ZMicro
157+
will only write one sector, as opposed to 3. Secondly, no matter how
158+
fast your disk is, it will not get anywhere near the performance of
159+
ZMicro because disk operations are inherently slow, and when put in
160+
perspective, the performance of the compression is
161+
really unimportant.
162+
163+
### Extendible hashing or B+ trees?
164+
165+
- Neither. TFS uses a combination of trees and hash tables: Nested hash tables, a form of hash trees. The idea is that instead of reallocating, a new subtable is created in the bucket.
166+
167+
## Resources on design
168+
169+
I've written a number of pieces on the design of TFS:
170+
171+
- [SeaHash: Explained](http://ticki.github.io/blog/seahash-explained/) - This describes the default checksum algorithm designed for TFS.
172+
- [On Random-Access Compression](http://ticki.github.io/blog/on-random-access-compression/) - This post describes the algorithm used for random-access compression.
173+
- [Ternary as a prediction residue code](http://ticki.github.io/blog/ternary-as-a-prediction-residue-code/) - The use of this is related to creating a good adaptive (headerless) entropy compressor.
174+
- [How LZ4 works](http://ticki.github.io/blog/how-lz4-works/) - This describes how the LZ4 compression algorithm works.
175+
- [Collision Resolution with Nested Hash Tables](https://ticki.github.io/blog/collision-resolution-with-nested-hash-tables/) - This describes the method of nested hash tables we use for the directory structure.
176+
- [An Atomic Hash Table](https://ticki.github.io/blog/an-atomic-hash-table/) - This describes the concurrent, in-memory hash table/key-value store.
177+
178+
## Specification
179+
180+
The full specification can be found in `specification.tex`, to render it install `texlive` or another distribution with XeTeX, and run:
181+
182+
```sh
183+
xelatex --shell-escape specification.tex
184+
```
185+
186+
Then open the file named `specification.pdf`

‎README.rst

-163
This file was deleted.

0 commit comments

Comments
 (0)