|
1 | 1 | # 构建工作负载(Building Workloads)
|
2 | 2 |
|
| 3 | +Deterload是一套基于nix开发的工作负载构建系统。 |
| 4 | +构建工作负载主要是使用nix, |
| 5 | +你可能会心头一紧,🙀“我就是想构建一些工作负载,还要需要一套新的编程语言/一个包管理器?”。 |
| 6 | +😺放轻松!不用担心! |
| 7 | +尽管nix的完整学习曲线较陡峭, |
| 8 | +但在本项目中,你只需要掌握少量直观的nix命令和语法即可。 |
3 | 9 |
|
| 10 | +Deterload is a workload building system developed based on nix. |
| 11 | +Building workloads mainly involves using nix. |
| 12 | +You might tense up, 🙀 "I just want to build some workloads, why do I need a new programming language/package manager?" |
| 13 | +😺 Relax! Don't worry! |
| 14 | +Although nix has a steep learning curve overall, |
| 15 | +in this project, you only need to master a few intuitive nix commands and syntax. |
| 16 | + |
| 17 | +## 基础构建(Basic Building) |
| 18 | + |
| 19 | +让我们从最简单的例子开始——构建一套OpenBLAS切片。 |
| 20 | +只需一行命令: |
| 21 | + |
| 22 | +Let's start with the simplest example — building an OpenBLAS checkpoint. |
| 23 | +It only takes one command: |
| 24 | + |
| 25 | +```bash |
| 26 | +nix-build -A openblas.cpt |
| 27 | +``` |
| 28 | + |
| 29 | +这行命令的组成: |
| 30 | + |
| 31 | +* `nix-build`是nix用于构建包的基础命令 |
| 32 | +* `-A openblas.cpt`指定了构建目标 |
| 33 | +* 提示1:如果你想看详细的构建信息(很酷炫的树形依赖图、任务数统计、时间统计等等), |
| 34 | + 你可以将`nix-build`替换为`nom-build`(一个`nix-build`的第三方包装命令`)。 |
| 35 | +* 提示2:其中`openblas`是一个结构体(nix里被成为attribute set,类似python字典), |
| 36 | + 包含了多个OpenBLAS工作负载相关的包,比如`openblas.benchmark`、`openblas.linux`、`openblas.qemu`和`openblas.cpt`等等。 |
| 37 | +* 提示3:如果你的shell有命令补全功能,`nix-build -A openblas.`敲tab键能给你补全出openblas里所有的包。 |
| 38 | + 其中`openblas.cpt`是我们需要的切片。 |
| 39 | + |
| 40 | +This command consists of: |
| 41 | + |
| 42 | +* `nix-build` is nix's basic command for building packages |
| 43 | +* `-A openblas.cpt` specifies the build target |
| 44 | +* Tip 1: If you want to see detailed build information (cool dependency trees, task statistics, time statistics, etc.), |
| 45 | + you can replace `nix-build` with `nom-build` (a third-party wrapper for `nix-build`). |
| 46 | +* Tip 2: Here `openblas` is a structure (called attribute set in nix, similar to Python dictionary), |
| 47 | + containing multiple OpenBLAS workload-related packages, such as `openblas.benchmark`, `openblas.linux`, `openblas.qemu`, and `openblas.cpt`, etc. |
| 48 | +* Tip 3: If your shell has command completion, pressing tab after `nix-build -A openblas.` will show all packages in openblas. |
| 49 | + Among these, `openblas.cpt` is the checkpoint we need. |
| 50 | + |
| 51 | +构建OpenBLAS的切片需要几个小时。 |
| 52 | +构建完成后会输出类似这样的路径: |
| 53 | + |
| 54 | +Building an OpenBLAS checkpoin takes several hours. |
| 55 | +After completion, it outputs a path like this: |
| 56 | + |
| 57 | +``` |
| 58 | +/nix/store/6rbfs8nx9xiv1s7z5xbi7m6djbkn9sgh-openblas_gcc_1410_RISCV64_GENERIC_glibc_qemu_20M_maxK30_1core_cpt |
| 59 | +``` |
| 60 | + |
| 61 | +nix会自动将该路径符号链接到`./result`。 |
| 62 | +你可以通过`-o`选项来改变默认符号链接的目标地址: |
| 63 | + |
| 64 | +nix will automatically create a symbolic link to this path at `./result`. |
| 65 | +You can change the default symbolic link target using the `-o` option: |
| 66 | + |
| 67 | +```bash |
| 68 | +nix-build -A openblas.cpt -o result-openblas.cpt |
| 69 | +``` |
| 70 | + |
| 71 | +值得注意的是,这一次构建`openblas.cpt`会非常快速。 |
| 72 | +这是因为nix采用的确定性构建的机制。 |
| 73 | +这一次构建和上一次构建除了名字以外没啥不同,所以nix直接复用之前的构建结果。 |
| 74 | + |
| 75 | +Notably, this second build of `openblas.cpt` will be very quick. |
| 76 | +This is due to nix's deterministic build mechanism. |
| 77 | +Since this build is identical to the previous one except for the name, nix directly reuses the previous build result. |
| 78 | + |
| 79 | +## 配参数(Configuring Arguments) |
| 80 | + |
| 81 | +构建产物的路径名(如上面的例子)包含了多个标签,例如: |
| 82 | + |
| 83 | +* 编译器版本(gcc 14.1.0) |
| 84 | +* OpenBLAS的目标架构(RISCV64_GENERIC) |
| 85 | +* ... |
| 86 | + |
| 87 | +The build output path (as in the example above) contains multiple tags, such as: |
| 88 | + |
| 89 | +* Compiler version (gcc 14.1.0) |
| 90 | +* OpenBLAS target architecture (RISCV64_GENERIC) |
| 91 | +* ... |
| 92 | + |
| 93 | +这些标签都是默认配置中预设好的参数。 |
| 94 | +我们可以根据自己的需求配置参数。 |
| 95 | +Deterload支持三种配置方式: |
| 96 | + |
| 97 | +* 命令行 |
| 98 | +* 配置文件 |
| 99 | +* 命令行+配置文件 |
| 100 | + |
| 101 | +These tags represent parameters set in the default configuration. |
| 102 | +We can configure these parameters according to our needs. |
| 103 | +Deterload supports three configuration methods: |
| 104 | + |
| 105 | +* Command line |
| 106 | +* Configuration file |
| 107 | +* Command line + Configuration file |
| 108 | + |
| 109 | +### 命令行(Command Line) |
| 110 | + |
| 111 | +使用`--arg key value`的方式配置参数,例如: |
| 112 | + |
| 113 | +Configure parameters using `--arg key value`, for example: |
| 114 | + |
| 115 | +```bash |
| 116 | +nix-build --arg enableVector true --arg cpt-maxK '"10"' -A openblas.cpt |
| 117 | +``` |
| 118 | + |
| 119 | +* `--arg enableVector true`:启用编译器的自动向量化 |
| 120 | +* `--arg cpt-maxK '"10"':设置simpoint的maxK设为10 |
| 121 | + |
| 122 | +注意:nix对参数类型有严格要求。 |
| 123 | +比如`enableVector`是一个bool类型,接收true/false。 |
| 124 | +`cpt-maxK`是一个字符串类型的参数,因此接收的参数需要加双引号(额外加单引号是为了shell不要吞掉双引号)。 |
| 125 | + |
| 126 | + |
| 127 | +* `--arg enableVector true`: Enable compiler auto-vectorization |
| 128 | +* `--arg cpt-maxK '"10"'`: Set simpoint's maxK to 10 |
| 129 | + |
| 130 | +Note: nix has strict type requirements for parameters. |
| 131 | +For instance, `enableVector` is a boolean type, accepting true/false. |
| 132 | +`cpt-maxK` is a string parameter, so it needs double quotes (with extra single quotes to prevent shell from stripping the double quotes). |
| 133 | + |
| 134 | +对于字符串类型的参数,双引号单引号过于麻烦,可以用`--argstr key value`来简化`--arg key '"value"'`: |
| 135 | + |
| 136 | +For string parameters, dealing with double and single quotes is cumbersome, so you can use `--argstr key value` to simplify `--arg key '"value"'`: |
| 137 | + |
| 138 | +```bash |
| 139 | +nix-build --arg enableVector true --argstr cpt-maxK 10 -A openblas.cpt |
| 140 | +``` |
| 141 | + |
| 142 | +### 配置文件(Configuration File) |
| 143 | + |
| 144 | +你可能会想:“我可以把命令行写入写一个shell脚本,岂不是就有了‘配置文件’了嘛”。像这样: |
| 145 | + |
| 146 | +You might think: "I could write these command lines into a shell script, and that would be a 'configuration file', right?" Like this: |
| 147 | + |
| 148 | +```bash |
| 149 | +#!/usr/bin/env bash |
| 150 | +# 这是一个难以保证确定性构建的“配置文件” |
| 151 | +# This is a "configuration file" that can't guarantee deterministic builds |
| 152 | +nix-build \ |
| 153 | + --arg enableVector true \ |
| 154 | + --argstr cpt-maxK 10 \ |
| 155 | + -A openblas.cpt |
| 156 | +``` |
| 157 | + |
| 158 | +这样的“配置文件”并不适合协同开发,因为: |
| 159 | + |
| 160 | +* 不同开发者用的Deterload版本可能不同,构建结果难以一致。 |
| 161 | +* 参数名称和含义可能会因版本变化而有所不同。 |
| 162 | + |
| 163 | +This type of "configuration file" isn't suitable for collaborative development because: |
| 164 | + |
| 165 | +* Different developers might use different Deterload versions, making build results inconsistent. |
| 166 | +* Parameter names and meanings might change between versions. |
| 167 | + |
| 168 | +为了解决这些问题,我们可以使用nix来编写配置文件。 |
| 169 | +例如,以下是一个与上述命令行等价的配置文件: |
| 170 | + |
| 171 | +To solve these issues, we can use nix to write configuration files. |
| 172 | +Here's a configuration file equivalent to the above command line: |
| 173 | + |
| 174 | +```nix |
| 175 | +# vec_maxK10.nix |
| 176 | +{...}@args: import (builtins.fetchTarball { |
| 177 | + url = "https://github.com/OpenXiangShan/Deterload/archive/v0.1.4.tar.gz"; |
| 178 | + # nix-prefetch-url --unpack https://github.com/OpenXiangShan/Deterload/archive/v0.1.4.tar.gz |
| 179 | + sha256 = "0l7bfjqjjlxkg8addgm6gkjv7p1psisv1wy648xwa8nw3nmgaw5d"; |
| 180 | +}) ({ |
| 181 | + enableVector = true; |
| 182 | + cpt-maxK = "10"; |
| 183 | +} // args) |
| 184 | +``` |
| 185 | + |
| 186 | +这段代码主要分成两个部分: |
| 187 | + |
| 188 | +* 固定版本的部分: |
| 189 | + * `url`设定了Deterload的源码来自GitHub,版本为v0.1.4。 |
| 190 | + * `sha256`是Deterload v0.1.4源码的sha256值,这个nix确定性构建的关键部分。 |
| 191 | + 你可以用`nix-prefetch-url`获取此值(见代码注释)。 |
| 192 | +* 配置参数的部分: |
| 193 | + * 配置了`enableVector`和`cpt-maxK`,具体含义与前文一致。 |
| 194 | + |
| 195 | +This code consists of two main parts: |
| 196 | + |
| 197 | +* Version fixing part: |
| 198 | + * `url` specifies that Deterload's source code comes from GitHub, version v0.1.4. |
| 199 | + * `sha256` is the sha256 value of Deterload v0.1.4 source code, crucial for nix's deterministic building. |
| 200 | + You can get this value using `nix-prefetch-url` (see code comment). |
| 201 | +* Parameter configuration part: |
| 202 | + * Configures `enableVector` and `cpt-maxK`, with meanings as explained earlier. |
| 203 | + |
| 204 | +将上述代码保存为文件(例如`vec_maxK10.nix`)。 |
| 205 | +每个开发者只需运行以下命令,就能生成二进制级别一致的`openblas.cpt`切片: |
| 206 | + |
| 207 | +Save this code as a file (e.g., `vec_maxK10.nix`). |
| 208 | +Any developer can run the following command to generate a binary-identical `openblas.cpt` checkpoint: |
| 209 | + |
| 210 | +```bash |
| 211 | +nix-build vec_maxK10.nix -A openblas.cpt |
| 212 | +``` |
| 213 | + |
| 214 | +比如在我的电脑上获得的结果路径,以及第一个切片的md5sum应该和你得到一样: |
| 215 | + |
| 216 | +For example, the checkpoint path of the result, and the md5sum of the first checkpoint on my computer should match yours: |
| 217 | + |
| 218 | +```bash |
| 219 | +# cd /nix/store/s3wxbj9rcxksn22v9ghlhikf1rvi4ybf-openblas_gcc_1410_RISCV64_ZVL128B_glibc_qemu_20M_maxK10_1core_cpt/miao && ls |
| 220 | +2 186 2343 3274 4093 4668 5991 6285 6357 |
| 221 | +# md5sum 2/_2_0.168009.gz |
| 222 | +43305c3b69822ea9fd34b5e08078ad68 result/miao/2/_2_0.168009.gz |
| 223 | +``` |
| 224 | + |
| 225 | +### 命令行+配置文件(Command Line + Configuration File) |
| 226 | + |
| 227 | +Deterload支持命令行+配置文件混合的配置方式。 |
| 228 | +以上述`vec_maxK10.nix`为例,命令行参数的优先级高于配置文件参数: |
| 229 | + |
| 230 | +Deterload supports mixed configuration using command line and configuration files. |
| 231 | +Using the above `vec_maxK10.nix` as an example, command line parameters take precedence over configuration file parameters: |
| 232 | + |
| 233 | +``` |
| 234 | +nix-build vec_maxK10.nix --argstr cpt-maxK 20 --argstr cpt-intervals 1000000 -A openblas.cpt |
| 235 | +``` |
| 236 | + |
| 237 | +上述命令覆盖了原本配置文件的`cpt-maxK`改为了`"20"`,并将`cpt-intervals`设置为了`"1000000"`。 |
| 238 | + |
| 239 | +This command overrides the original `cpt-maxK` in the configuration file to `"20"` and sets `cpt-intervals` to `"1000000"`. |
0 commit comments