[Good First Issue][NNCF]: Use stateful model in LLM compression examples

[alexsu52]

Context

A “stateful model” is a model that implicitly preserves data between two consecutive inference calls such as KV cache for LLMs (more details). Using a stateful model in the inference allows us to minimize the overhead of processing the KV cache, and due to this and additional optimizations, significantly speed up the inference of the model. OpenVINO currently export LLM from PyTorch to OpenVINO IR as stateful model by default. Thus, NNCF should also demonstrate the default flow in its examples.

What needs to be done?

Update the following LLM compression examples to use stateful model:

• Large Language Models FP8 Compression Example
• Find the appropriate hyperparameters to compress the TinyLLama model
• Compress TinyLLama model using synthetic data

Example Pull Requests

#3490

Resources

• Contribution guide - start here!
• Intel DevHub Discord channel - engage in discussions, ask questions and talk to OpenVINO developers
• How to link your Pull Request to an issue

Contact points

@ljaljushkin
@alexsu52

Ticket

No response

[adnaen]

Hey, I'd like to contribute here. can i get this?

[ljaljushkin]

Hey, I'd like to contribute here. can i get this?

Hi! Sure, you can. Assigning you to the issue, but in the future you can simply do that by leaving a comment with the following content: .take

[github-actions]

Thank you for looking into this issue! Please let us know if you have any questions or require any help.