Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test (example/llm): Refactor and add basic tests for the LLM entry-point #1002

Merged
merged 53 commits into from
Sep 11, 2024

Conversation

nickfraser
Copy link
Collaborator

@nickfraser nickfraser commented Aug 19, 2024

Create a tests which fully test the end-to-end examples, since the stability of these examples is becoming increasingly important. The idea is to put into a place a structure that can be applied to other entry-points in brevitas_examples. Incorporates #987 - Now rebased on dev.

TODO:

  • Move argument parser to a separate function, to allow calling from tests
  • Update main function to take parsed arguments and an input, return useful output that we want to test as output
  • Create run tests for tiny versions of important models:
    • Llama
    • OPT
    • Mistral
  • Create run tests on all pre-quantization and PTQ techniques on all selected tiny models:
    • --ln-affine-merge
    • --weight-equalization
    • --act-equalization layerwise
    • --act-equalization fx
    • --bias-corr
    • --act-calibration
    • --gptq
  • Test --replace-mha on OPT
  • Test some important datatypes:
    • Weight-only (int4)
    • Int8
    • FP8
    • MXFP8 (fails)
  • Create accuracy tests for tiny versions of important models (should cover all models, datatypes and PTQ techniques):
    • Llama
    • OPT
    • Mistral
  • Create tests for export flows
    • ONNX QCDQ
    • Torchscript QCDQ
    • torchmlir fails and won't fix
    • torchmlir packed weights fails and won't fix
  • Added test to github actions

@nickfraser nickfraser mentioned this pull request Aug 20, 2024
20 tasks
@nickfraser nickfraser marked this pull request as ready for review August 20, 2024 17:02
@nickfraser nickfraser self-assigned this Aug 20, 2024
@nickfraser nickfraser added the next release PRs which should be merged for the next release label Aug 20, 2024
@nickfraser nickfraser requested review from Giuseppe5 and removed request for Giuseppe5 August 21, 2024 17:23
@nickfraser nickfraser requested review from Giuseppe5 and removed request for Giuseppe5 August 26, 2024 10:15
noxfile.py Show resolved Hide resolved
tests/brevitas_examples/test_llm.py Outdated Show resolved Hide resolved
tests/brevitas_examples/test_llm.py Outdated Show resolved Hide resolved
tests/brevitas_examples/test_llm.py Outdated Show resolved Hide resolved
tests/brevitas_examples/test_llm.py Show resolved Hide resolved
tests/brevitas_examples/test_llm.py Show resolved Hide resolved
tests/brevitas_examples/test_llm.py Outdated Show resolved Hide resolved
tests/brevitas_examples/test_llm.py Outdated Show resolved Hide resolved
@nickfraser nickfraser requested review from Giuseppe5 and removed request for Giuseppe5 September 2, 2024 17:06
@nickfraser nickfraser merged commit f58f64b into Xilinx:dev Sep 11, 2024
23 checks passed
@nickfraser nickfraser deleted the test/llm branch September 11, 2024 15:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
next release PRs which should be merged for the next release
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants