You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: SQFT/README.md
+10Lines changed: 10 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -12,6 +12,7 @@ Specifically, the highlights of SQFT include:
12
12
Please refer to our paper for more details.
13
13
14
14
## News
15
+
-**[2025.07.02]** SQFT's NLS algorithm is now supported in [OpenVINO NNCF](https://github.com/openvinotoolkit/nncf) for quantization-aware tuning to improve accuracy on downstream Tasks, including new heuristics. More details can be found [here](https://github.com/openvinotoolkit/nncf/tree/v2.17.0/examples/llm_compression/torch/downstream_qat_with_nls).
15
16
-**[2024.09.24]****SQFT** paper has been released ([link](https://arxiv.org/pdf/2410.03750)) and **accepted at EMNLP 2024 Findings**. :books:
16
17
-**[2024.09.24]** Release the code for **SQFT**. :tada:
17
18
@@ -384,6 +385,15 @@ lm_eval --model hf \
384
385
385
386
</details>
386
387
388
+
## New Heuristic
389
+
390
+
In addition to the super simple heuristic using the median config, we have designed new heuristics based on training loss and activation counts during training. Specifically, we have:
391
+
392
+
- **Loss-based Sub-adapter**: This heuristic selects sub-adapters based on the loss information recorded during training.
393
+
- **Activation-based Sub-adapter**: This heuristic selects sub-adapters based on the most frequently activated configurations during training.
394
+
395
+
For more details, please refer to the script [here](https://github.com/openvinotoolkit/nncf/blob/v2.17.0/examples/llm_compression/torch/downstream_qat_with_nls/main.py).
0 commit comments