Skip to content

Commit 839d1df

Browse files
committed
docs: usages: 1.building_outputs
1 parent bffd4f6 commit 839d1df

File tree

4 files changed

+237
-29
lines changed

4 files changed

+237
-29
lines changed

docs/usages/1.building_outputs.md

+236
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,239 @@
11
# 构建工作负载(Building Workloads)
22

3+
Deterload是一套基于nix开发的工作负载构建系统。
4+
构建工作负载主要是使用nix,
5+
你可能会心头一紧,🙀“我就是想构建一些工作负载,还要需要一套新的编程语言/一个包管理器?”。
6+
😺放轻松!不用担心!
7+
尽管nix的完整学习曲线较陡峭,
8+
但在本项目中,你只需要掌握少量直观的nix命令和语法即可。
39

10+
Deterload is a workload building system developed based on nix.
11+
Building workloads mainly involves using nix.
12+
You might tense up, 🙀 "I just want to build some workloads, why do I need a new programming language/package manager?"
13+
😺 Relax! Don't worry!
14+
Although nix has a steep learning curve overall,
15+
in this project, you only need to master a few intuitive nix commands and syntax.
16+
17+
## 基础构建(Basic Building)
18+
19+
让我们从最简单的例子开始——构建一套OpenBLAS切片。
20+
只需一行命令:
21+
22+
Let's start with the simplest example — building an OpenBLAS checkpoint.
23+
It only takes one command:
24+
25+
```bash
26+
nix-build -A openblas.cpt
27+
```
28+
29+
这行命令的组成:
30+
31+
* `nix-build`是nix用于构建包的基础命令
32+
* `-A openblas.cpt`指定了构建目标
33+
* 提示1:如果你想看详细的构建信息(很酷炫的树形依赖图、任务数统计、时间统计等等),
34+
你可以将`nix-build`替换为`nom-build`(一个`nix-build`的第三方包装命令`)。
35+
* 提示2:其中`openblas`是一个结构体(nix里被成为attribute set,类似python字典),
36+
包含了多个OpenBLAS工作负载相关的包,比如`openblas.benchmark``openblas.linux``openblas.qemu``openblas.cpt`等等。
37+
* 提示3:如果你的shell有命令补全功能,`nix-build -A openblas.`敲tab键能给你补全出openblas里所有的包。
38+
其中`openblas.cpt`是我们需要的切片。
39+
40+
This command consists of:
41+
42+
* `nix-build` is nix's basic command for building packages
43+
* `-A openblas.cpt` specifies the build target
44+
* Tip 1: If you want to see detailed build information (cool dependency trees, task statistics, time statistics, etc.),
45+
you can replace `nix-build` with `nom-build` (a third-party wrapper for `nix-build`).
46+
* Tip 2: Here `openblas` is a structure (called attribute set in nix, similar to Python dictionary),
47+
containing multiple OpenBLAS workload-related packages, such as `openblas.benchmark`, `openblas.linux`, `openblas.qemu`, and `openblas.cpt`, etc.
48+
* Tip 3: If your shell has command completion, pressing tab after `nix-build -A openblas.` will show all packages in openblas.
49+
Among these, `openblas.cpt` is the checkpoint we need.
50+
51+
构建OpenBLAS的切片需要几个小时。
52+
构建完成后会输出类似这样的路径:
53+
54+
Building an OpenBLAS checkpoin takes several hours.
55+
After completion, it outputs a path like this:
56+
57+
```
58+
/nix/store/6rbfs8nx9xiv1s7z5xbi7m6djbkn9sgh-openblas_gcc_1410_RISCV64_GENERIC_glibc_qemu_20M_maxK30_1core_cpt
59+
```
60+
61+
nix会自动将该路径符号链接到`./result`
62+
你可以通过`-o`选项来改变默认符号链接的目标地址:
63+
64+
nix will automatically create a symbolic link to this path at `./result`.
65+
You can change the default symbolic link target using the `-o` option:
66+
67+
```bash
68+
nix-build -A openblas.cpt -o result-openblas.cpt
69+
```
70+
71+
值得注意的是,这一次构建`openblas.cpt`会非常快速。
72+
这是因为nix采用的确定性构建的机制。
73+
这一次构建和上一次构建除了名字以外没啥不同,所以nix直接复用之前的构建结果。
74+
75+
Notably, this second build of `openblas.cpt` will be very quick.
76+
This is due to nix's deterministic build mechanism.
77+
Since this build is identical to the previous one except for the name, nix directly reuses the previous build result.
78+
79+
## 配参数(Configuring Arguments)
80+
81+
构建产物的路径名(如上面的例子)包含了多个标签,例如:
82+
83+
* 编译器版本(gcc 14.1.0)
84+
* OpenBLAS的目标架构(RISCV64_GENERIC)
85+
* ...
86+
87+
The build output path (as in the example above) contains multiple tags, such as:
88+
89+
* Compiler version (gcc 14.1.0)
90+
* OpenBLAS target architecture (RISCV64_GENERIC)
91+
* ...
92+
93+
这些标签都是默认配置中预设好的参数。
94+
我们可以根据自己的需求配置参数。
95+
Deterload支持三种配置方式:
96+
97+
* 命令行
98+
* 配置文件
99+
* 命令行+配置文件
100+
101+
These tags represent parameters set in the default configuration.
102+
We can configure these parameters according to our needs.
103+
Deterload supports three configuration methods:
104+
105+
* Command line
106+
* Configuration file
107+
* Command line + Configuration file
108+
109+
### 命令行(Command Line)
110+
111+
使用`--arg key value`的方式配置参数,例如:
112+
113+
Configure parameters using `--arg key value`, for example:
114+
115+
```bash
116+
nix-build --arg enableVector true --arg cpt-maxK '"10"' -A openblas.cpt
117+
```
118+
119+
* `--arg enableVector true`:启用编译器的自动向量化
120+
* `--arg cpt-maxK '"10"':设置simpoint的maxK设为10
121+
122+
注意:nix对参数类型有严格要求。
123+
比如`enableVector`是一个bool类型,接收true/false。
124+
`cpt-maxK`是一个字符串类型的参数,因此接收的参数需要加双引号(额外加单引号是为了shell不要吞掉双引号)。
125+
126+
127+
* `--arg enableVector true`: Enable compiler auto-vectorization
128+
* `--arg cpt-maxK '"10"'`: Set simpoint's maxK to 10
129+
130+
Note: nix has strict type requirements for parameters.
131+
For instance, `enableVector` is a boolean type, accepting true/false.
132+
`cpt-maxK` is a string parameter, so it needs double quotes (with extra single quotes to prevent shell from stripping the double quotes).
133+
134+
对于字符串类型的参数,双引号单引号过于麻烦,可以用`--argstr key value`来简化`--arg key '"value"'`
135+
136+
For string parameters, dealing with double and single quotes is cumbersome, so you can use `--argstr key value` to simplify `--arg key '"value"'`:
137+
138+
```bash
139+
nix-build --arg enableVector true --argstr cpt-maxK 10 -A openblas.cpt
140+
```
141+
142+
### 配置文件(Configuration File)
143+
144+
你可能会想:“我可以把命令行写入写一个shell脚本,岂不是就有了‘配置文件’了嘛”。像这样:
145+
146+
You might think: "I could write these command lines into a shell script, and that would be a 'configuration file', right?" Like this:
147+
148+
```bash
149+
#!/usr/bin/env bash
150+
# 这是一个难以保证确定性构建的“配置文件”
151+
# This is a "configuration file" that can't guarantee deterministic builds
152+
nix-build \
153+
--arg enableVector true \
154+
--argstr cpt-maxK 10 \
155+
-A openblas.cpt
156+
```
157+
158+
这样的“配置文件”并不适合协同开发,因为:
159+
160+
* 不同开发者用的Deterload版本可能不同,构建结果难以一致。
161+
* 参数名称和含义可能会因版本变化而有所不同。
162+
163+
This type of "configuration file" isn't suitable for collaborative development because:
164+
165+
* Different developers might use different Deterload versions, making build results inconsistent.
166+
* Parameter names and meanings might change between versions.
167+
168+
为了解决这些问题,我们可以使用nix来编写配置文件。
169+
例如,以下是一个与上述命令行等价的配置文件:
170+
171+
To solve these issues, we can use nix to write configuration files.
172+
Here's a configuration file equivalent to the above command line:
173+
174+
```nix
175+
# vec_maxK10.nix
176+
{...}@args: import (builtins.fetchTarball {
177+
url = "https://github.com/OpenXiangShan/Deterload/archive/v0.1.4.tar.gz";
178+
# nix-prefetch-url --unpack https://github.com/OpenXiangShan/Deterload/archive/v0.1.4.tar.gz
179+
sha256 = "0l7bfjqjjlxkg8addgm6gkjv7p1psisv1wy648xwa8nw3nmgaw5d";
180+
}) ({
181+
enableVector = true;
182+
cpt-maxK = "10";
183+
} // args)
184+
```
185+
186+
这段代码主要分成两个部分:
187+
188+
* 固定版本的部分:
189+
* `url`设定了Deterload的源码来自GitHub,版本为v0.1.4。
190+
* `sha256`是Deterload v0.1.4源码的sha256值,这个nix确定性构建的关键部分。
191+
你可以用`nix-prefetch-url`获取此值(见代码注释)。
192+
* 配置参数的部分:
193+
* 配置了`enableVector``cpt-maxK`,具体含义与前文一致。
194+
195+
This code consists of two main parts:
196+
197+
* Version fixing part:
198+
* `url` specifies that Deterload's source code comes from GitHub, version v0.1.4.
199+
* `sha256` is the sha256 value of Deterload v0.1.4 source code, crucial for nix's deterministic building.
200+
You can get this value using `nix-prefetch-url` (see code comment).
201+
* Parameter configuration part:
202+
* Configures `enableVector` and `cpt-maxK`, with meanings as explained earlier.
203+
204+
将上述代码保存为文件(例如`vec_maxK10.nix`)。
205+
每个开发者只需运行以下命令,就能生成二进制级别一致的`openblas.cpt`切片:
206+
207+
Save this code as a file (e.g., `vec_maxK10.nix`).
208+
Any developer can run the following command to generate a binary-identical `openblas.cpt` checkpoint:
209+
210+
```bash
211+
nix-build vec_maxK10.nix -A openblas.cpt
212+
```
213+
214+
比如在我的电脑上获得的结果路径,以及第一个切片的md5sum应该和你得到一样:
215+
216+
For example, the checkpoint path of the result, and the md5sum of the first checkpoint on my computer should match yours:
217+
218+
```bash
219+
# cd /nix/store/s3wxbj9rcxksn22v9ghlhikf1rvi4ybf-openblas_gcc_1410_RISCV64_ZVL128B_glibc_qemu_20M_maxK10_1core_cpt/miao && ls
220+
2 186 2343 3274 4093 4668 5991 6285 6357
221+
# md5sum 2/_2_0.168009.gz
222+
43305c3b69822ea9fd34b5e08078ad68 result/miao/2/_2_0.168009.gz
223+
```
224+
225+
### 命令行+配置文件(Command Line + Configuration File)
226+
227+
Deterload支持命令行+配置文件混合的配置方式。
228+
以上述`vec_maxK10.nix`为例,命令行参数的优先级高于配置文件参数:
229+
230+
Deterload supports mixed configuration using command line and configuration files.
231+
Using the above `vec_maxK10.nix` as an example, command line parameters take precedence over configuration file parameters:
232+
233+
```
234+
nix-build vec_maxK10.nix --argstr cpt-maxK 20 --argstr cpt-intervals 1000000 -A openblas.cpt
235+
```
236+
237+
上述命令覆盖了原本配置文件的`cpt-maxK`改为了`"20"`,并将`cpt-intervals`设置为了`"1000000"`
238+
239+
This command overrides the original `cpt-maxK` in the configuration file to `"20"` and sets `cpt-intervals` to `"1000000"`.

docs/usages/2.configuring.md

-27
This file was deleted.

docs/usages/3.examples.md

-1
This file was deleted.

docs/usages/4.running/index.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,3 @@
1-
# 运行输出的工作负载(Running Output Workloads)
1+
# 运行工作负载(Running Workloads)
22

33
TODO:

0 commit comments

Comments
 (0)