forked from onnx/onnx
-
Notifications
You must be signed in to change notification settings - Fork 2
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Signed-off-by: Yuan Yao <[email protected]>
- Loading branch information
1 parent
d45ad99
commit 74e11de
Showing
83 changed files
with
507 additions
and
52 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,118 @@ | ||
<!-- | ||
Copyright (c) ONNX Project Contributors | ||
SPDX-License-Identifier: Apache-2.0 | ||
--> | ||
|
||
(onnx-detail-float4)= | ||
|
||
# Float stored in 4 bits | ||
|
||
## Papers | ||
|
||
4 bit floating point formats have emerged as a solution to the | ||
rising cost and deployment challenges of large language models. | ||
The S1E2M1 format has been part of the [Open Compute Project (OCP)](https://www.opencompute.org/documents/ocp-microscaling-formats-mx-v1-0-spec-final-pdf) | ||
standard. | ||
|
||
As a result, a new data type was introduced in `onnx==1.18.0` | ||
to support a limited set of operators to enable computation | ||
with float4. | ||
|
||
- `FLOAT4E2M1`: 1 bit for the sign, 2 bits for the exponents, and 1 bit for the mantissa. | ||
No nan or infinities. | ||
|
||
## E2M1 | ||
|
||
$S$ stands for the sign. $10_2$ describe a number base 2. | ||
|
||
```{eval-rst} | ||
.. list-table:: Float4 type | ||
:widths: 10 10 | ||
:header-rows: 1 | ||
* - | ||
- E2M1 | ||
* - Exponent bias | ||
- 1 | ||
* - Infinities | ||
- | ||
* - NaN | ||
- | ||
* - Zeros | ||
- :math:`S.00.0_2` | ||
* - Max | ||
- :math:`S.11.1_2` | ||
* - Min | ||
- :math:`S.00.1_2 = 2^{-1}` | ||
``` | ||
|
||
Let's denote the bit representation as $S.b_2 b_1 b_0$. | ||
The float value is defined by the following expressions: | ||
|
||
```{eval-rst} | ||
.. list-table:: Float4 type values | ||
:widths: 10 10 | ||
:header-rows: 1 | ||
* - | ||
- E2M1 | ||
* - exponent :math:`\neq` 0 | ||
- :math:`(-1)^S 2^{\sum_{i=1}^2 b_i 2^{i-1} - 1} \left( 1 + b_0 2^{-1} \right)` | ||
* - exponent :math:`=` 0 | ||
- :math:`(-1)^S b_0 2^{-1}` | ||
``` | ||
|
||
The following table lists all the representable values by float4 E2M1, ignoring the sign bit: | ||
```{eval-rst} | ||
.. list-table:: Float4 type values | ||
:widths: 10 10 | ||
:header-rows: 1 | ||
* - bits (ignoring sign bit) | ||
- E2M1 | ||
* - 000 | ||
- 0 | ||
* - 001 | ||
- 0.5 | ||
* - 010 | ||
- 1 | ||
* - 011 | ||
- 1.5 | ||
* - 100 | ||
- 2 | ||
* - 101 | ||
- 3 | ||
* - 110 | ||
- 4 | ||
* - 111 | ||
- 6 | ||
``` | ||
|
||
## Cast | ||
|
||
Upcasting from float4 to float32, float16, bfloat16, and float8 is exact. | ||
The behavior for downcasting to float 4 is summarized below | ||
|
||
| x | E2M1 | | ||
| ----------------- | ------------------------------------------------- | | ||
| -6<=x<=6 | E2M1 converted value of x. Round to nearest even. | | ||
| x=+/-0 | +/-0 | | ||
| x>6 | 6 | | ||
| x<-6 | -6 | | ||
| +Inf | 6 | | ||
| -Inf | -6 | | ||
| NaN | 6 | | ||
|
||
## Packing and Unpacking | ||
|
||
Float4 is stored as 2x4bit in a single byte. | ||
The first element is stored in the 4 LSB and the second element is stored in the 4 MSB, | ||
i.e. for elements `x` and `y` that are consecutive elements in the array: | ||
``` | ||
pack(x,y): y << 4 | x & 0x0F | ||
unpack(z): x = z & 0x0F, y = z >> 4 | ||
``` | ||
In case the total number of elements is odd, padding of 4 bits will be appended. | ||
The storage size of a 4 bit tensor of size `N` is `ceil(N/2)`. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -16,4 +16,5 @@ deeper than the code documentation. | |
float8 | ||
int4 | ||
float4 | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Binary file modified
BIN
+0 Bytes
(100%)
onnx/backend/test/data/node/test_acos/test_data_set_0/output_0.pb
Binary file not shown.
2 changes: 1 addition & 1 deletion
2
onnx/backend/test/data/node/test_acosh/test_data_set_0/output_0.pb
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
2 changes: 1 addition & 1 deletion
2
onnx/backend/test/data/node/test_asin/test_data_set_0/output_0.pb
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
2 changes: 1 addition & 1 deletion
2
onnx/backend/test/data/node/test_asinh/test_data_set_0/output_0.pb
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
2 changes: 1 addition & 1 deletion
2
onnx/backend/test/data/node/test_atan/test_data_set_0/output_0.pb
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
4 changes: 2 additions & 2 deletions
4
onnx/backend/test/data/node/test_atanh/test_data_set_0/output_0.pb
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Binary file modified
BIN
+0 Bytes
(100%)
onnx/backend/test/data/node/test_blackmanwindow/test_data_set_0/output_0.pb
Binary file not shown.
Binary file modified
BIN
+0 Bytes
(100%)
onnx/backend/test/data/node/test_blackmanwindow_expanded/test_data_set_0/output_0.pb
Binary file not shown.
Binary file modified
BIN
+0 Bytes
(100%)
onnx/backend/test/data/node/test_blackmanwindow_symmetric/test_data_set_0/output_0.pb
Binary file not shown.
Binary file modified
BIN
+0 Bytes
(100%)
...backend/test/data/node/test_blackmanwindow_symmetric_expanded/test_data_set_0/output_0.pb
Binary file not shown.
2 changes: 1 addition & 1 deletion
2
onnx/backend/test/data/node/test_cosh/test_data_set_0/output_0.pb
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Binary file modified
BIN
+0 Bytes
(100%)
onnx/backend/test/data/node/test_cosh_example/test_data_set_0/output_0.pb
Binary file not shown.
Binary file modified
BIN
+0 Bytes
(100%)
onnx/backend/test/data/node/test_dft/test_data_set_0/output_0.pb
Binary file not shown.
Binary file modified
BIN
+0 Bytes
(100%)
onnx/backend/test/data/node/test_dft_axis/test_data_set_0/output_0.pb
Binary file not shown.
Binary file modified
BIN
+0 Bytes
(100%)
onnx/backend/test/data/node/test_dft_axis_opset19/test_data_set_0/output_0.pb
Binary file not shown.
Binary file modified
BIN
+0 Bytes
(100%)
onnx/backend/test/data/node/test_dft_inverse/test_data_set_0/output_0.pb
Binary file not shown.
Binary file modified
BIN
+0 Bytes
(100%)
onnx/backend/test/data/node/test_dft_inverse_opset19/test_data_set_0/output_0.pb
Binary file not shown.
Binary file modified
BIN
+0 Bytes
(100%)
onnx/backend/test/data/node/test_dft_opset19/test_data_set_0/output_0.pb
Binary file not shown.
Binary file modified
BIN
+0 Bytes
(100%)
onnx/backend/test/data/node/test_gelu_default_1/test_data_set_0/output_0.pb
Binary file not shown.
Binary file modified
BIN
+0 Bytes
(100%)
onnx/backend/test/data/node/test_gelu_default_1_expanded/test_data_set_0/output_0.pb
Binary file not shown.
7 changes: 3 additions & 4 deletions
7
onnx/backend/test/data/node/test_gelu_default_2/test_data_set_0/output_0.pb
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
7 changes: 3 additions & 4 deletions
7
onnx/backend/test/data/node/test_gelu_default_2_expanded/test_data_set_0/output_0.pb
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Binary file modified
BIN
+0 Bytes
(100%)
onnx/backend/test/data/node/test_gelu_tanh_2/test_data_set_0/output_0.pb
Binary file not shown.
Binary file modified
BIN
+0 Bytes
(100%)
onnx/backend/test/data/node/test_gelu_tanh_2_expanded/test_data_set_0/output_0.pb
Binary file not shown.
Binary file modified
BIN
+0 Bytes
(100%)
onnx/backend/test/data/node/test_hammingwindow/test_data_set_0/output_0.pb
Binary file not shown.
Binary file modified
BIN
+0 Bytes
(100%)
onnx/backend/test/data/node/test_hammingwindow_expanded/test_data_set_0/output_0.pb
Binary file not shown.
2 changes: 1 addition & 1 deletion
2
onnx/backend/test/data/node/test_hammingwindow_symmetric/test_data_set_0/output_0.pb
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,2 +1,2 @@ | ||
ByJ(C�=�iF>)��>��E?��x?��x?��E?$��>�iF>C�= | ||
ByJ(@�=�iF>*��>��E?��x?��x?��E?"��>�iF>@�= |
2 changes: 1 addition & 1 deletion
2
.../backend/test/data/node/test_hammingwindow_symmetric_expanded/test_data_set_0/output_0.pb
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,2 +1,2 @@ | ||
ByJ(C�=�iF>)��>��E?��x?��x?��E?$��>�iF>C�= | ||
ByJ(@�=�iF>*��>��E?��x?��x?��E?"��>�iF>@�= |
Binary file modified
BIN
+0 Bytes
(100%)
onnx/backend/test/data/node/test_hannwindow/test_data_set_0/output_0.pb
Binary file not shown.
Binary file modified
BIN
+0 Bytes
(100%)
onnx/backend/test/data/node/test_hannwindow_expanded/test_data_set_0/output_0.pb
Binary file not shown.
Binary file modified
BIN
+0 Bytes
(100%)
onnx/backend/test/data/node/test_hannwindow_symmetric/test_data_set_0/output_0.pb
Binary file not shown.
Binary file modified
BIN
+0 Bytes
(100%)
onnx/backend/test/data/node/test_hannwindow_symmetric_expanded/test_data_set_0/output_0.pb
Binary file not shown.
4 changes: 2 additions & 2 deletions
4
onnx/backend/test/data/node/test_lppool_1d_default/test_data_set_0/output_0.pb
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Binary file modified
BIN
+0 Bytes
(100%)
onnx/backend/test/data/node/test_lppool_2d_default/test_data_set_0/output_0.pb
Binary file not shown.
Binary file modified
BIN
+0 Bytes
(100%)
onnx/backend/test/data/node/test_lppool_2d_pads/test_data_set_0/output_0.pb
Binary file not shown.
Binary file modified
BIN
+0 Bytes
(100%)
onnx/backend/test/data/node/test_lppool_2d_same_lower/test_data_set_0/output_0.pb
Binary file not shown.
Binary file modified
BIN
+0 Bytes
(100%)
onnx/backend/test/data/node/test_lppool_2d_same_upper/test_data_set_0/output_0.pb
Binary file not shown.
Binary file modified
BIN
+0 Bytes
(100%)
onnx/backend/test/data/node/test_lppool_2d_strides/test_data_set_0/output_0.pb
Binary file not shown.
Binary file modified
BIN
+0 Bytes
(100%)
onnx/backend/test/data/node/test_lppool_3d_default/test_data_set_0/output_0.pb
Binary file not shown.
Binary file modified
BIN
+0 Bytes
(100%)
onnx/backend/test/data/node/test_lrn_default/test_data_set_0/output_0.pb
Binary file not shown.
Binary file modified
BIN
+0 Bytes
(100%)
onnx/backend/test/data/node/test_mish/test_data_set_0/output_0.pb
Binary file not shown.
Binary file modified
BIN
+0 Bytes
(100%)
onnx/backend/test/data/node/test_mish_expanded/test_data_set_0/output_0.pb
Binary file not shown.
2 changes: 1 addition & 1 deletion
2
onnx/backend/test/data/node/test_mvn/test_data_set_0/output_0.pb
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1 +1 @@ | ||
BYJl�d�?�;�>:�ſr����d���>D��>6nQ?��[?:���#rc�vyH�E3U?> ����,?�UD?�Pi?�ҿ�o����?��>�2�?�ϗ?��m����=msþ��� | ||
BYJl�d�?�;�>:�ſr����d���>D��>6nQ?��[?:���#rc�vyH�G3U?? ����,?�UD?�Pi?�ҿ�o����?��>�2�?�ϗ?��m����=msþ��� |
2 changes: 1 addition & 1 deletion
2
onnx/backend/test/data/node/test_mvn_expanded/test_data_set_0/output_0.pb
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1 +1 @@ | ||
BYJl�d�?�;�>:�ſr����d���>D��>6nQ?��[?:���#rc�vyH�E3U?> ����,?�UD?�Pi?�ҿ�o����?��>�2�?�ϗ?��m����=msþ��� | ||
BYJl�d�?�;�>:�ſr����d���>D��>6nQ?��[?:���#rc�vyH�G3U?? ����,?�UD?�Pi?�ҿ�o����?��>�2�?�ϗ?��m����=msþ��� |
2 changes: 1 addition & 1 deletion
2
onnx/backend/test/data/node/test_mvn_expanded_ver18/test_data_set_0/output_0.pb
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1 +1 @@ | ||
BYJl�d�?�;�>:�ſr����d���>D��>6nQ?��[?:���#rc�vyH�E3U?> ����,?�UD?�Pi?�ҿ�o����?��>�2�?�ϗ?��m����=msþ��� | ||
BYJl�d�?�;�>:�ſr����d���>D��>6nQ?��[?:���#rc�vyH�G3U?? ����,?�UD?�Pi?�ҿ�o����?��>�2�?�ϗ?��m����=msþ��� |
Binary file modified
BIN
+0 Bytes
(100%)
onnx/backend/test/data/node/test_pow/test_data_set_0/output_0.pb
Binary file not shown.
2 changes: 1 addition & 1 deletion
2
...test/data/node/test_reduce_log_sum_exp_do_not_keepdims_random/test_data_set_0/output_0.pb
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,2 +1,2 @@ | ||
BreducedJ0�eMIz�@�&~X@{��U��忎c;��^@n6���"@�� | ||
BreducedJ0�eMIz�@�&~X@}��U��忎c;��^@n6���"@�� | ||
�1�? |
2 changes: 1 addition & 1 deletion
2
.../node/test_reduce_log_sum_exp_do_not_keepdims_random_expanded/test_data_set_0/output_0.pb
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,2 +1,2 @@ | ||
BreducedJ0�eMIz�@�&~X@{��U��忎c;��^@n6���"@�� | ||
BreducedJ0�eMIz�@�&~X@}��U��忎c;��^@n6���"@�� | ||
�1�? |
2 changes: 1 addition & 1 deletion
2
...ackend/test/data/node/test_reduce_log_sum_exp_keepdims_random/test_data_set_0/output_0.pb
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,2 +1,2 @@ | ||
BreducedJ0�eMIz�@�&~X@{��U��忎c;��^@n6���"@�� | ||
BreducedJ0�eMIz�@�&~X@}��U��忎c;��^@n6���"@�� | ||
�1�? |
2 changes: 1 addition & 1 deletion
2
...st/data/node/test_reduce_log_sum_exp_keepdims_random_expanded/test_data_set_0/output_0.pb
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,2 +1,2 @@ | ||
BreducedJ0�eMIz�@�&~X@{��U��忎c;��^@n6���"@�� | ||
BreducedJ0�eMIz�@�&~X@}��U��忎c;��^@n6���"@�� | ||
�1�? |
2 changes: 1 addition & 1 deletion
2
...ta/node/test_reduce_log_sum_exp_negative_axes_keepdims_random/test_data_set_0/output_0.pb
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,2 +1,2 @@ | ||
BreducedJ0�eMIz�@�&~X@{��U��忎c;��^@n6���"@�� | ||
BreducedJ0�eMIz�@�&~X@}��U��忎c;��^@n6���"@�� | ||
�1�? |
2 changes: 1 addition & 1 deletion
2
...est_reduce_log_sum_exp_negative_axes_keepdims_random_expanded/test_data_set_0/output_0.pb
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,2 +1,2 @@ | ||
BreducedJ0�eMIz�@�&~X@{��U��忎c;��^@n6���"@�� | ||
BreducedJ0�eMIz�@�&~X@}��U��忎c;��^@n6���"@�� | ||
�1�? |
2 changes: 1 addition & 1 deletion
2
onnx/backend/test/data/node/test_sinh/test_data_set_0/output_0.pb
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Binary file modified
BIN
+0 Bytes
(100%)
onnx/backend/test/data/node/test_stft_with_window/test_data_set_0/input_2.pb
Binary file not shown.
Binary file modified
BIN
+0 Bytes
(100%)
onnx/backend/test/data/node/test_stft_with_window/test_data_set_0/output_0.pb
Binary file not shown.
2 changes: 1 addition & 1 deletion
2
onnx/backend/test/data/node/test_tan/test_data_set_0/output_0.pb
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.