|
1 |
| -# KiWi |
| 1 | +<p align="center"> |
| 2 | + <img src="./docs/logo.png" alt="KiWi Logo" width="200" height="220"/> |
| 3 | +</p> |
2 | 4 |
|
3 |
| -KiWi is a simple key-value store that supports the following operations: |
| 5 | +# KiWi: A Lightweight RESP Key-Value Store. |
4 | 6 |
|
5 |
| -- `PUT <key> <value>`: Store the given value under the given key. |
6 |
| -- `GET <key>`: Retrieve the value stored under the given key. |
7 |
| -- `DELETE <key>`: Remove the value stored under the given key. |
| 7 | +KiWi is a high-performance, [RESP](https://redis.io/docs/latest/develop/reference/protocol-spec/) |
| 8 | +compliant, key-value store inspired by the Bitcask paper, designed |
| 9 | +for simplicity, reliability, and blazing-fast read/write operations. |
8 | 10 |
|
9 |
| -## Build |
| 11 | +## Features |
10 | 12 |
|
11 |
| -To build the project, run the following command: |
| 13 | +- RESP protocol support for Redis-compatible client interaction. |
| 14 | +- Non-blocking I/O server with Netty. |
| 15 | +- High-performance key-value store based on the Bitcask storage model. |
| 16 | +- In-memory indexing for fast reads. |
| 17 | +- TTL-based key expiration. |
| 18 | +- Checksums for data integrity. |
| 19 | +- Compaction and efficient file merging process. |
| 20 | +- Hint files for quick startup times. |
| 21 | +- Tunable durability. |
12 | 22 |
|
13 |
| -```shell |
14 |
| -./gradlew build |
15 |
| -``` |
| 23 | +## Quick Start |
| 24 | + |
| 25 | +1. Start Docker container |
| 26 | + ```bash |
| 27 | + docker run --rm --name kiwi -p 6379:6379 nemanjam/kiwi:latest |
| 28 | + ``` |
| 29 | + |
| 30 | +2. Connect to the server with `redis-cli` |
| 31 | + ```bash |
| 32 | + redis-cli -h localhost |
| 33 | + ``` |
| 34 | + |
| 35 | +3. Use the server as you would a Redis server. |
| 36 | + ```text |
| 37 | + SET key value |
| 38 | + OK |
| 39 | +
|
| 40 | + GET key |
| 41 | + "value" |
| 42 | +
|
| 43 | + EXISTS key |
| 44 | + (integer) 1 |
| 45 | +
|
| 46 | + DEL key |
| 47 | + OK |
| 48 | +
|
| 49 | + EXISTS key |
| 50 | + (integer) 0 |
| 51 | + ``` |
| 52 | + |
| 53 | +### Supported Commands |
| 54 | + |
| 55 | +- `SET key value` |
| 56 | +- `GET key` |
| 57 | +- `DEL key` |
| 58 | +- `EXISTS key` |
| 59 | +- `FLUSHDB` |
| 60 | +- `PING` |
| 61 | +- `DBSIZE` |
| 62 | +- `INFO` |
| 63 | + |
| 64 | +## Installation |
| 65 | + |
| 66 | +### Prerequisites |
| 67 | + |
| 68 | +- Java 21 |
| 69 | +- Docker (optional, for running via a container) |
| 70 | + |
| 71 | +### From Source |
| 72 | + |
| 73 | +1. Clone the repository: |
| 74 | + ```bash |
| 75 | + git clone https://github.com/nemanjam/kiwi.git |
| 76 | + cd kiwi |
| 77 | + ``` |
| 78 | +2. Build the project: |
| 79 | + ```bash |
| 80 | + ./gradlew assembleDist installDist |
| 81 | + ``` |
| 82 | +3. Run the KiWi server: |
| 83 | + ```bash |
| 84 | + ./kiwi-server/build/install/kiwi-server/bin/kiwi-server |
| 85 | + ``` |
| 86 | + |
| 87 | +### Docker |
| 88 | + |
| 89 | +1. Build the Docker image |
| 90 | + ```bash |
| 91 | + docker build -t kiwi . |
| 92 | + ``` |
| 93 | +2. Run the container: |
| 94 | + ```bash |
| 95 | + docker run --rm --name kiwi -p 6379:6379 kiwi:latest |
| 96 | + ``` |
| 97 | +3. Connect to the server: |
| 98 | + ```bash |
| 99 | + redis-cli -h localhost -p 6379 |
| 100 | + ``` |
| 101 | + |
| 102 | +## Configuration |
| 103 | + |
| 104 | +KiWi can be configured using environment variables or a HOCON configuration file. |
| 105 | +Refer to [Typesafe Config](https://github.com/lightbend/config) for configuration examples. |
| 106 | + |
| 107 | +Default values are: |
| 108 | + |
| 109 | +- Storage [application.conf](kiwi-core/src/main/resources/application.conf) |
| 110 | +- Server [application.conf](kiwi-server/src/main/resources/application.conf) |
| 111 | + |
| 112 | +## Benchmarks |
| 113 | + |
| 114 | +KiWi can be evaluated with [redis-benchmark](https://redis.io/topics/benchmarks) utility command. |
| 115 | + |
| 116 | +Below are the results of running `redis-benchmark` with KiWi and Redis on a local setup (MacBook M3 |
| 117 | +Pro with 18GB RAM and Sequoia 15.1.1). |
16 | 118 |
|
17 |
| -## Test |
| 119 | +```text |
| 120 | +redis-benchmark -h localhost -t set -n 100000 -r 10000000 -d 1024 |
18 | 121 |
|
19 |
| -To execute the tests, use the following command: |
| 122 | +====== SET ====== |
| 123 | + 100000 requests completed in 2.18 seconds |
| 124 | + 50 parallel clients |
| 125 | + 1024 bytes payload |
| 126 | + keep alive: 1 |
| 127 | + host configuration "save": |
| 128 | + host configuration "appendonly": |
| 129 | + multi-thread: no |
20 | 130 |
|
21 |
| -```shell |
22 |
| -./gradlew test |
| 131 | +Summary: |
| 132 | + throughput summary: 45934.77 requests per second |
| 133 | + latency summary (msec): |
| 134 | + avg min p50 p95 p99 max |
| 135 | + 1.016 0.088 0.951 1.863 2.703 30.655 |
23 | 136 | ```
|
24 | 137 |
|
25 |
| -## Run |
| 138 | +```text |
| 139 | +redis-benchmark -h localhost -t get -n 100000 -r 10000000 -d 1024 |
26 | 140 |
|
27 |
| -To run the project, execute the following commands: |
| 141 | +====== GET ====== |
| 142 | + 100000 requests completed in 1.83 seconds |
| 143 | + 50 parallel clients |
| 144 | + 1024 bytes payload |
| 145 | + keep alive: 1 |
| 146 | + host configuration "save": |
| 147 | + host configuration "appendonly": |
| 148 | + multi-thread: no |
28 | 149 |
|
29 |
| -```shell |
30 |
| -./gradlew assembleDist installDist |
| 150 | +Summary: |
| 151 | + throughput summary: 60753.34 requests per second |
| 152 | + latency summary (msec): |
| 153 | + avg min p50 p95 p99 max |
| 154 | + 0.661 0.088 0.639 0.967 1.311 12.255 |
31 | 155 | ```
|
32 | 156 |
|
33 |
| -```shell |
34 |
| -./build/install/kiwi/bin/kiwi |
| 157 | +JVM options: |
| 158 | + |
| 159 | +```text |
| 160 | +-Xms2g -Xmx2g -XX:UseG1GC –XX:+UseStringDeduplication -XX:+AlwaysPreTouch |
| 161 | +```` |
| 162 | +
|
| 163 | +KiWi Configuration: |
| 164 | +
|
| 165 | +```hocon |
| 166 | +kiwi { |
| 167 | + storage { |
| 168 | + log { |
| 169 | + dir = "/tmp/kiwi" |
| 170 | + segment.bytes = 1073741824 // 1GB |
| 171 | + |
| 172 | + sync { |
| 173 | + mode = "periodic" |
| 174 | + periodic { |
| 175 | + interval = 10s |
| 176 | + } |
| 177 | + } |
| 178 | + } |
| 179 | + } |
| 180 | +} |
35 | 181 | ```
|
| 182 | + |
| 183 | +## Design |
| 184 | + |
| 185 | +KiWi combines the simplicity of RESP with the efficient storage model described in the Bitcask |
| 186 | +paper. This architecture is designed for high performance and simplicity. |
| 187 | + |
| 188 | +### Storage Model |
| 189 | + |
| 190 | +- All write operations are appended to a log file, ensuring sequential disk writes for maximum |
| 191 | + performance. |
| 192 | +- When the active log file reaches a configurable size, it is rolled over to a segment file. |
| 193 | +- Periodically, segment files are compacted to remove stale data and reclaim disk space. |
| 194 | +- Crash recovery is achieved by replaying the log files during startup. |
| 195 | +- Disk I/O operations, like log compaction, are handled in background threads to avoid blocking |
| 196 | + client requests. |
| 197 | + |
| 198 | +### In-Memory Index |
| 199 | + |
| 200 | +- All keys are stored in an in-memory hash table, pointing to their location in the log file. |
| 201 | +- This ensures `O(1)` read performance while keeping the storage footprint minimal. |
| 202 | + |
| 203 | +### Non-Blocking I/O Server |
| 204 | + |
| 205 | +- Netty-based event loop for handling client requests. |
| 206 | +- KiWi supports the RESP protocol, making it compatible with Redis clients and tools. |
| 207 | + |
| 208 | +### Durability |
| 209 | + |
| 210 | +- KiWi provides tunable durability options to balance performance and data safety: |
| 211 | + - `periodic` (default): Writes are flushed to disk at regular intervals. |
| 212 | + - `batch`: Writes are batched and flushed when the batch window expires. All writers are blocked |
| 213 | + until the batch is written. |
| 214 | + - `lazy`: Flush is delegated to the operating system, which may delay writes for performance. |
| 215 | + |
| 216 | +### Pros |
| 217 | + |
| 218 | +- Fast writes due to sequential disk I/O. |
| 219 | +- Fast reads with O(1) lookups using the in-memory index. |
| 220 | +- Simple and robust crash recovery with the data and hint files. |
| 221 | +- Incremental crash-safe compaction process. |
| 222 | + |
| 223 | +### Cons |
| 224 | + |
| 225 | +- The in-memory index requires all keys to fit in memory. |
| 226 | +- Log compaction introduces periodic I/O overhead. |
| 227 | + |
| 228 | +## Contributing |
| 229 | + |
| 230 | +We welcome contributions to KiWi! Here’s how you can help: |
| 231 | + |
| 232 | +1. Fork the repository. |
| 233 | +2. Create a new branch for your feature or bugfix. |
| 234 | +3. Submit a pull request with a clear description of your changes. |
| 235 | + |
| 236 | +## License |
| 237 | + |
| 238 | +KiWi is licensed under the MIT License. See [LICENSE](./LICENSE) for details. |
0 commit comments