Discussion about potential GSoC projects, Project2 and Project14

[AsVoider]

Hello, mentors @riacheruvu @DimaPastushenkov,

My name is Jianxiang Gao, a year 0 master from SJTU, IPADS lab, and I'm interested in Project2 'OpenVINO Messenger AI-Assistant for an AI PC' for GSoC 25.

Based on my reading of the project materials and my research on RAG technology, I would like to discuss my current understanding of the project and some questions I have to ensure that I am on the right track.

The Prs for prerequisite task is here: #20921, #29028, #20941. These Prs have been all already merged.

Regarding the implementation of the project, I currently have a high-level plan:

1. Research message platforms and complete platform API integration.
2. Processing and storing data in a structured, unstructured, or semi-structured manner.
3. Set up an LLM server on the AI PC using Openvino Backend, select an appropriate model, and optimizing the model's inference speed while ensuring accuracy.
4. Implement data retrieval and prompt engineering for a basic RAG workflow.
5. Build a user-friendly interface with UI, including at least the RAG control, message management, and model configuration sections.
6. Test, optimize, and deploy, while providing user documentation.

Besides, I also have some questions on the details about this project:

1. For data processing and storage, do we need to provide encryption to ensure privacy? Should we consider the performance overhead introduced by encryption and decryption?
2. Considering the timeliness of information, storage space usage, and the decline in retrieval efficiency and quality due to excessive data accumulation, are there any best practices for data storage that I can refer to?
3. I prefer a modular development approach to reduce coupling in the engineering aspect of the project. If this aligns with your engineering development principles, could you share your experience or provide any suggestions?
4. What are the criteria for evaluating the completion of the project? I can use these standards to assess whether I can extend its functionality.

Here are some additional questions:

1. Could you kindly share any specific requirements or insights that would enhance the quality of my proposal?
2. Based on my overall plan, which parts need to be emphasized in the proposal, and which areas are you more interested in?

My research focuses on MLSys, which aims at deploying large language models on edge devices like PCs and mobile phones, and optimizing inference performance. I also have experience deploying RAG on a PC, like GraphRAG, RAGFlow, etc. Right now, I'm working on the following tasks:

1. Using an API to obtain data from a messaging application and process it locally.
2. Converting the processed data into vectors using an embedding model and storing them in a locally deployed vector database.
3. Deploying a model on a PC that relies on the OpenVINO backend.
   I hope that my existing experience can contribute to the project's work.

I'd appreciate your insights and suggestions! Thank you for your valuable time.

Looking forward for your reply.

Best regards,
Jianxiang Gao

[AsVoider]

Hi Mentors of Project 14, @alexsu52 @AlexanderDokuchaev,

I hope this message finds you well. Alongside my interest in Project 2, I am eager to contribute to this project: "Accelerating Inference of NNCF-Compressed LLMs with Triton." I believe my technical background in LLM kernel optimization would be a valuable asset to your team.

Relevant Experience:

1. LLM Kernel Adaptation: I possess a strong understanding of flash attention and matmul operators in llama.cpp, having successfully ported them to a laboratory project using the SYCL framework.
2. Sparse Operator Optimization: I optimized the sparse FFN operator for the open-source version of PowerInfer, efficiently reducing overhead through operator fusion (details in Here).
3. Sparse MoE Specialization: Currently, I am assisting my lab with restructuring PowerInfer, focusing on enhancing the sparse MoE operator.

Here are some questions about this project:

1. In examining open-source low-bit Triton matrix multiplication kernels (such as GemLite, Tinygemm, Marlin, etc.), are there notable differences in terms of operator flexibility, memory layout support, and hardware generalization capability? Which library would you recommend for building and expanding?
2. Are there specific programming considerations or best practices when utilizing Triton for mixed-precision quantization in NNCF? For instance, how should one approach memory alignment, data layout, and the selection of tile/block sizes?
3. Triton's "write once, run anywhere" promise appears to conflict with vendor-specific optimizations (e.g., Tensor Cores for INT4 on Hopper). How should one balance portability with achieving peak hardware performance?
4. What evaluation criteria and metrics should be considered when establishing performance baselines and conducting benchmark testing?
   Additionally, I would like to know which aspects you prioritize most in the proposal.

Although I have not written a Triton kernel myself, I have observed many optimized operators developed with Triton during my research, such as MInference.

I would greatly appreciate your response. If given the opportunity to participate in the project, I am committed to bringing my dedication and passion to the team.

Thank you for your time and guidance.