[Feature] MindSpore features integrate

[lvyufeng]

Motivation

Use good features of MindSpore

Jit

PSJit

PSJit means ast analsis method, which asked the developer to follow the limitted grammar to construct the model. Pytorch do not support such method and the torch implement version often caused grammar error when convert to MindSpore. Therefore we should support the modeling convert to disable not supported grammar.

PIJit

PIJit likes torch dynamo, which use the bytecode analysis to capture the whole graph, if MindSpore detect the grammar like branch/thirdparty libs/unspported grammar will automaticlly split the graph, even fallback to the single kernel dispatch when extreme situations。We need test most of the huggingface hosted models to get the benifits of PIJit

Dataset

Apache Arrow Files + MindSpore Dataset Engine

DVM

DVM(device virtual machine), means a automatic opeator fuse method when operators are dispatching, MindSpore can capture the dispatch queue of operators and anlysis the possible to fuse, and finally replace the operator list with a fused op to speedup execution.

Automatic Modeling Convertion

This is a function to convert huggingface torch implementation to MindSpore native implementation which can be leverage with AutoModel.from_pretrained. Here is the pseudo code:

1. convert model to MindSpore version

import mindnlp
from mindnlp import model_export
from transformers import AutoModel
model = AutoModel.from_pretrained('llama3-8b')

ms_model = model_export(model)

2. use jit to speedup

import mindspore as ms
@ms.jit
def generate(*args):
    return ms_model.generate(*args)

3. save modeling.py+ckpt with mindspore implementation

ms_model.save_pretrained(path_to_save)

4. upload to huggingfacehub

hf_hub.upload(path_to_save)

5. use AutoModel to load the mindspore version of model

model = AutoModel.from_pretrained('llama3-8b-ms', dtype=mindspore.float16, remote_code=True)