MetaSpore is a one-stop end-to-end machine learning development platform that provides a full-cycle framework and development interface for from data preprocessing, model training, offline experiments, online predictions to online experiment traffic bucketization and ab-testing.
MetaSpore is developed and opensourced by DMetaSoul team. You could also join our slack user discussion space.
MetaSpore has the following features:
- One-stop end-to-end development, from offline model training to online prediction and bucketing experiments, with a unified development experience across the entire process;
- Deep learning training framework, compatible with PyTorch ecology, supports distributed large-scale sparse feature learning;
- The training framework is connected with PySpark to seamlessly read the training data from the data lake and data warehouse;
- High-performance online prediction service, supports fast inference for neural network, decision tree, Spark ML, SKLearn and other models; supports heterogeneous hardware inference acceleration;
- In the offline unified feature extraction framework, the online feature reading logic is automatically generated, and the feature extraction logic is unified cross offline and online;
- Online algorithm application framework, providing model prediction, experiment bucketing and traffic splitting, dynamic hot loading of parameters and rich debug functions;
- Rich industry algorithm examples and end-to-end solutions.
We provide precompiled offline training wheel package on pypi, install it via pip:
pip install metasporeThe minimum Python version required is 3.8.
After installation, also install pytorch and pyspark (they are not included as depenencies of metaspore wheel so you could choose pyspark and pytorch versions as needed):
pip install pyspark
pip install torch==1.11.0+cpu -f https://download.pytorch.org/whl/cpu/torch_stable.htmlWe provide prebuilt docker images for MetaSpore Serving Service:
docker pull dmetasoul/metaspore-serving-release:cpu-v1.0.1
docker pull dmetasoul/metaspore-serving-release:gpu-v1.0.1
See Run Serving Service in Docker for details.
For questions about usage, you can post questions in GitHub Discussion, or through GitHub Issue.
Email us at [email protected].
Join our user discussion slack channel: MetaSpore User Discussion
MetaSpore is a completely open source project released under the Apache License 2.0. Participation, feedback, and code contributions are welcome.