Skip to content

Commit 42ac430

Browse files
htahir1htahir1
committed
initial commit
0 parents  commit 42ac430

17 files changed

+1472
-0
lines changed

‎.dockerignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
.venv*
2+
.requirements*

‎.gitignore‎

Lines changed: 151 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,151 @@
1+
# Byte-compiled / optimized / DLL files
2+
__pycache__/
3+
*.py[cod]
4+
*$py.class
5+
6+
# C extensions
7+
*.so
8+
9+
# Distribution / packaging
10+
.Python
11+
build/
12+
develop-eggs/
13+
dist/
14+
downloads/
15+
eggs/
16+
.eggs/
17+
lib/
18+
lib64/
19+
parts/
20+
sdist/
21+
var/
22+
wheels/
23+
pip-wheel-metadata/
24+
share/python-wheels/
25+
*.egg-info/
26+
.installed.cfg
27+
*.egg
28+
MANIFEST
29+
30+
# PyInstaller
31+
# Usually these files are written by a python script from a template
32+
# before PyInstaller builds the exe, so as to inject date/other infos into it.
33+
*.manifest
34+
*.spec
35+
36+
# Installer logs
37+
pip-log.txt
38+
pip-delete-this-directory.txt
39+
40+
# Unit test / coverage reports
41+
htmlcov/
42+
.tox/
43+
.nox/
44+
.coverage
45+
.coverage.*
46+
.cache
47+
nosetests.xml
48+
coverage.xml
49+
*.cover
50+
*.py,cover
51+
.hypothesis/
52+
.pytest_cache/
53+
54+
# Translations
55+
*.mo
56+
*.pot
57+
58+
# Django stuff:
59+
*.log
60+
local_settings.py
61+
db.sqlite3
62+
db.sqlite3-journal
63+
64+
# Flask stuff:
65+
instance/
66+
.webassets-cache
67+
68+
# Scrapy stuff:
69+
.scrapy
70+
71+
# Sphinx documentation
72+
docs/_build/
73+
74+
# PyBuilder
75+
target/
76+
77+
# Jupyter Notebook
78+
.ipynb_checkpoints
79+
80+
# IPython
81+
profile_default/
82+
ipython_config.py
83+
84+
# pyenv
85+
.python-version
86+
87+
# pipenv
88+
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
89+
# However, in case of collaboration, if having platform-specific dependencies or dependencies
90+
# having no cross-platform support, pipenv may install dependencies that don't work, or not
91+
# install all needed dependencies.
92+
#Pipfile.lock
93+
94+
# PEP 582; used by e.g. github.com/David-OConnor/pyflow
95+
__pypackages__/
96+
97+
# Celery stuff
98+
celerybeat-schedule
99+
celerybeat.pid
100+
101+
# SageMath parsed files
102+
*.sage.py
103+
104+
# Environments
105+
.env
106+
.venv*
107+
env/
108+
venv/
109+
ENV/
110+
env.bak/
111+
venv.bak/
112+
113+
# Spyder project settings
114+
.spyderproject
115+
.spyproject
116+
117+
# Rope project settings
118+
.ropeproject
119+
120+
# mkdocs documentation
121+
/site
122+
123+
# mypy
124+
.mypy_cache/
125+
.dmypy.json
126+
dmypy.json
127+
128+
# Pyre type checker
129+
.pyre/
130+
131+
# Zenml
132+
.zen/
133+
134+
# MLflow
135+
mlruns/
136+
137+
# VSCode
138+
.vscode/
139+
140+
# Pycharm
141+
.idea/
142+
143+
zencoder/cloned_public_repos
144+
*wandb*
145+
146+
.DS_Store
147+
.local
148+
149+
150+
# ollama ignores
151+
nohup.out

‎LICENSE‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
Apache Software License 2.0
2+
3+
Copyright (c) ZenML GmbH 2025. All rights reserved.
4+
5+
Licensed under the Apache License, Version 2.0 (the "License");
6+
you may not use this file except in compliance with the License.
7+
You may obtain a copy of the License at
8+
9+
http://www.apache.org/licenses/LICENSE-2.0
10+
11+
Unless required by applicable law or agreed to in writing, software
12+
distributed under the License is distributed on an "AS IS" BASIS,
13+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
See the License for the specific language governing permissions and
15+
limitations under the License.

‎README.md‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
# OpenPipe ZenML Integration
2+
3+
This project integrates OpenPipe's fine-tuning capabilities with ZenML pipelines, allowing for streamlined development and deployment of fine-tuned language models.
4+
5+
## Overview
6+
7+
OpenPipe is a platform that allows for easy fine-tuning of large language models (LLMs) on your custom data. This integration with ZenML provides a structured pipeline approach to:
8+
9+
1. Convert your data to OpenPipe-compatible format
10+
2. Create a dataset in OpenPipe
11+
3. Upload your data to OpenPipe
12+
4. Initiate fine-tuning on your chosen base model
13+
5. Monitor the fine-tuning process
14+
15+
## Prerequisites
16+
17+
- Python 3.8+
18+
- ZenML installed and configured
19+
- An OpenPipe API key
20+
21+
## Installation
22+
23+
1. Clone this repository
24+
2. Install the requirements:
25+
26+
```bash
27+
pip install -r requirements.txt
28+
```
29+
30+
3. Set your OpenPipe API key as an environment variable (optional):
31+
32+
```bash
33+
export OPENPIPE_API_KEY=opk-your-api-key
34+
```
35+
36+
## Usage
37+
38+
Run the pipeline with:
39+
40+
```bash
41+
python run.py --openpipe-api-key=opk-your-api-key
42+
```
43+
44+
### Command Line Options
45+
46+
- `--openpipe-api-key`: Your OpenPipe API key (required if not set as environment variable)
47+
- `--dataset-name`: Name for the OpenPipe dataset (default: "zenml_dataset")
48+
- `--model-name`: Name for the fine-tuned model (default: "zenml_finetuned_model")
49+
- `--base-model`: Base model to fine-tune (default: "meta-llama/Meta-Llama-3-8B-Instruct")
50+
- `--system-prompt`: System prompt to use for all examples (default: "You are a helpful assistant")
51+
- `--wait-for-completion/--no-wait-for-completion`: Whether to wait for the fine-tuning job to complete (default: wait)
52+
- `--no-cache`: Disable caching for the pipeline run
53+
54+
## Customizing Data
55+
56+
The pipeline uses a data loader that loads data into a pandas DataFrame. The default implementation expects columns for 'question' and 'answer'. You may need to modify the data loader to match your data format.
57+
58+
## Contributing
59+
60+
Contributions are welcome! Please feel free to submit a Pull Request.

‎configs/openpipe_finetuning.yaml‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
# environment configuration
2+
settings:
3+
docker:
4+
required_integrations:
5+
- sklearn
6+
- pandas
7+
requirements:
8+
- pyarrow
9+
- requests
10+
- openpipe
11+
12+
# data parameters
13+
random_state: 42
14+
sample_size: 30
15+
data_source: "toy"
16+
user_column: "question"
17+
assistant_column: "answer"
18+
system_prompt: "You are a helpful customer service assistant for Ultra electronics products."
19+
split_ratio: 0.9
20+
metadata_columns: ["product"]
21+
22+
# openPipe parameters
23+
dataset_name: "ultra_customer_service"
24+
model_name: "customer_service_assistant"
25+
base_model: "meta-llama/Meta-Llama-3-8B-Instruct"
26+
enable_sft: true
27+
enable_preference_tuning: false
28+
learning_rate_multiplier: 1.0
29+
num_epochs: 10
30+
batch_size: "auto"
31+
default_temperature: 0
32+
wait_for_completion: true
33+
timeout_minutes: 120
34+
verbose_logs: true
35+
auto_rename: true
36+
force_overwrite: false

‎openpipe_data.jsonl‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
{"messages": [{"role": "system", "content": "You are a helpful customer service assistant for Ultra electronics products."}, {"role": "user", "content": "What is the price of the UltraPhone X?"}, {"role": "assistant", "content": "The UltraPhone X is available for $999. Would you like to know about our financing options?"}], "split": "TRAIN", "metadata": {"product": "UltraPhone X"}}
2+
{"messages": [{"role": "system", "content": "You are a helpful customer service assistant for Ultra electronics products."}, {"role": "user", "content": "Is the UltraTablet Pro compatible with my other devices?"}, {"role": "assistant", "content": "Yes, the UltraTablet Pro is compatible with all other Ultra devices and most third-party products. It supports standard connectivity options."}], "split": "TRAIN", "metadata": {"product": "UltraPhone SE"}}
3+
{"messages": [{"role": "system", "content": "You are a helpful customer service assistant for Ultra electronics products."}, {"role": "user", "content": "How does the UltraPhone SE compare to other models?"}, {"role": "assistant", "content": "The UltraPhone SE ($699) has 4G, 5.8-inch display, 8MP camera, 128GB storage. Compared to other models, it offers a great balance of performance and value."}], "split": "TRAIN", "metadata": {"product": "UltraTablet Pro"}}
4+
{"messages": [{"role": "system", "content": "You are a helpful customer service assistant for Ultra electronics products."}, {"role": "user", "content": "Can I trade in my old device for a new UltraPhone X?"}, {"role": "assistant", "content": "Absolutely! You can trade in your eligible device for credit toward your new UltraPhone X, potentially saving up to $300."}], "split": "TRAIN", "metadata": {"product": "UltraWatch"}}
5+
{"messages": [{"role": "system", "content": "You are a helpful customer service assistant for Ultra electronics products."}, {"role": "user", "content": "What accessories are available for the UltraPhone X?"}, {"role": "assistant", "content": "We offer a range of accessories for the UltraPhone X, including cases, screen protectors, chargers, and adapters."}], "split": "TRAIN", "metadata": {"product": "UltraBook Air"}}
6+
{"messages": [{"role": "system", "content": "You are a helpful customer service assistant for Ultra electronics products."}, {"role": "user", "content": "What is the price of the UltraWatch?"}, {"role": "assistant", "content": "The UltraWatch is available for $349. Would you like to know about our financing options?"}], "split": "TRAIN", "metadata": {"product": "UltraPhone X"}}
7+
{"messages": [{"role": "system", "content": "You are a helpful customer service assistant for Ultra electronics products."}, {"role": "user", "content": "Tell me about the features of UltraPhone X"}, {"role": "assistant", "content": "The UltraPhone X offers 5G, 6.7-inch display, 12MP camera, 256GB storage. Is there a specific feature you're interested in?"}], "split": "TRAIN", "metadata": {"product": "UltraPhone SE"}}
8+
{"messages": [{"role": "system", "content": "You are a helpful customer service assistant for Ultra electronics products."}, {"role": "user", "content": "Is the UltraPhone SE compatible with my other devices?"}, {"role": "assistant", "content": "Yes, the UltraPhone SE is compatible with all other Ultra devices and most third-party products. It supports standard connectivity options."}], "split": "TRAIN", "metadata": {"product": "UltraTablet Pro"}}
9+
{"messages": [{"role": "system", "content": "You are a helpful customer service assistant for Ultra electronics products."}, {"role": "user", "content": "What accessories are available for the UltraBook Air?"}, {"role": "assistant", "content": "We offer a range of accessories for the UltraBook Air, including cases, screen protectors, chargers, and adapters."}], "split": "TRAIN", "metadata": {"product": "UltraWatch"}}
10+
{"messages": [{"role": "system", "content": "You are a helpful customer service assistant for Ultra electronics products."}, {"role": "user", "content": "Can I trade in my old device for a new UltraPhone X?"}, {"role": "assistant", "content": "Absolutely! You can trade in your eligible device for credit toward your new UltraPhone X, potentially saving up to $300."}], "split": "TRAIN", "metadata": {"product": "UltraBook Air"}}
11+
{"messages": [{"role": "system", "content": "You are a helpful customer service assistant for Ultra electronics products."}, {"role": "user", "content": "Can I trade in my old device for a new UltraPhone SE?"}, {"role": "assistant", "content": "Absolutely! You can trade in your eligible device for credit toward your new UltraPhone SE, potentially saving up to $300."}], "split": "TRAIN", "metadata": {"product": "UltraPhone X"}}
12+
{"messages": [{"role": "system", "content": "You are a helpful customer service assistant for Ultra electronics products."}, {"role": "user", "content": "Is the UltraWatch compatible with my other devices?"}, {"role": "assistant", "content": "Yes, the UltraWatch is compatible with all other Ultra devices and most third-party products. It supports standard connectivity options."}], "split": "TRAIN", "metadata": {"product": "UltraPhone SE"}}
13+
{"messages": [{"role": "system", "content": "You are a helpful customer service assistant for Ultra electronics products."}, {"role": "user", "content": "What accessories are available for the UltraWatch?"}, {"role": "assistant", "content": "We offer a range of accessories for the UltraWatch, including cases, screen protectors, chargers, and adapters."}], "split": "TRAIN", "metadata": {"product": "UltraTablet Pro"}}
14+
{"messages": [{"role": "system", "content": "You are a helpful customer service assistant for Ultra electronics products."}, {"role": "user", "content": "What is the price of the UltraTablet Pro?"}, {"role": "assistant", "content": "The UltraTablet Pro is available for $1299. Would you like to know about our financing options?"}], "split": "TRAIN", "metadata": {"product": "UltraWatch"}}
15+
{"messages": [{"role": "system", "content": "You are a helpful customer service assistant for Ultra electronics products."}, {"role": "user", "content": "What colors does the UltraPhone SE come in?"}, {"role": "assistant", "content": "The UltraPhone SE is available in Space Gray, Silver, and Midnight Blue colors. The Midnight Blue is our newest addition."}], "split": "TRAIN", "metadata": {"product": "UltraBook Air"}}
16+
{"messages": [{"role": "system", "content": "You are a helpful customer service assistant for Ultra electronics products."}, {"role": "user", "content": "When will the UltraTablet Pro be back in stock?"}, {"role": "assistant", "content": "We expect to receive more stock of the UltraTablet Pro within the next 2 weeks. Would you like to be notified when it's available?"}], "split": "TRAIN", "metadata": {"product": "UltraPhone X"}}
17+
{"messages": [{"role": "system", "content": "You are a helpful customer service assistant for Ultra electronics products."}, {"role": "user", "content": "Is the UltraPhone SE compatible with my other devices?"}, {"role": "assistant", "content": "Yes, the UltraPhone SE is compatible with all other Ultra devices and most third-party products. It supports standard connectivity options."}], "split": "TRAIN", "metadata": {"product": "UltraPhone SE"}}
18+
{"messages": [{"role": "system", "content": "You are a helpful customer service assistant for Ultra electronics products."}, {"role": "user", "content": "Tell me about the features of UltraTablet Pro"}, {"role": "assistant", "content": "The UltraTablet Pro offers 10.5-inch display, 512GB storage, keyboard included. Is there a specific feature you're interested in?"}], "split": "TRAIN", "metadata": {"product": "UltraTablet Pro"}}
19+
{"messages": [{"role": "system", "content": "You are a helpful customer service assistant for Ultra electronics products."}, {"role": "user", "content": "What colors does the UltraPhone X come in?"}, {"role": "assistant", "content": "The UltraPhone X is available in Space Gray, Silver, and Midnight Blue colors. The Midnight Blue is our newest addition."}], "split": "TRAIN", "metadata": {"product": "UltraWatch"}}
20+
{"messages": [{"role": "system", "content": "You are a helpful customer service assistant for Ultra electronics products."}, {"role": "user", "content": "Do you offer any discounts on the UltraPhone X?"}, {"role": "assistant", "content": "We currently have a 10% discount for students and military personnel on the UltraPhone X, bringing the price from $999 to a reduced rate."}], "split": "TRAIN", "metadata": {"product": "UltraBook Air"}}
21+
{"messages": [{"role": "system", "content": "You are a helpful customer service assistant for Ultra electronics products."}, {"role": "user", "content": "What accessories are available for the UltraTablet Pro?"}, {"role": "assistant", "content": "We offer a range of accessories for the UltraTablet Pro, including cases, screen protectors, chargers, and adapters."}], "split": "TRAIN", "metadata": {"product": "UltraPhone X"}}
22+
{"messages": [{"role": "system", "content": "You are a helpful customer service assistant for Ultra electronics products."}, {"role": "user", "content": "What is the price of the UltraTablet Pro?"}, {"role": "assistant", "content": "The UltraTablet Pro is available for $1299. Would you like to know about our financing options?"}], "split": "TRAIN", "metadata": {"product": "UltraPhone SE"}}
23+
{"messages": [{"role": "system", "content": "You are a helpful customer service assistant for Ultra electronics products."}, {"role": "user", "content": "Can I trade in my old device for a new UltraWatch?"}, {"role": "assistant", "content": "Absolutely! You can trade in your eligible device for credit toward your new UltraWatch, potentially saving up to $300."}], "split": "TRAIN", "metadata": {"product": "UltraTablet Pro"}}
24+
{"messages": [{"role": "system", "content": "You are a helpful customer service assistant for Ultra electronics products."}, {"role": "user", "content": "What colors does the UltraPhone X come in?"}, {"role": "assistant", "content": "The UltraPhone X is available in Space Gray, Silver, and Midnight Blue colors. The Midnight Blue is our newest addition."}], "split": "TRAIN", "metadata": {"product": "UltraWatch"}}
25+
{"messages": [{"role": "system", "content": "You are a helpful customer service assistant for Ultra electronics products."}, {"role": "user", "content": "Can I trade in my old device for a new UltraPhone X?"}, {"role": "assistant", "content": "Absolutely! You can trade in your eligible device for credit toward your new UltraPhone X, potentially saving up to $300."}], "split": "TRAIN", "metadata": {"product": "UltraBook Air"}}
26+
{"messages": [{"role": "system", "content": "You are a helpful customer service assistant for Ultra electronics products."}, {"role": "user", "content": "What accessories are available for the UltraTablet Pro?"}, {"role": "assistant", "content": "We offer a range of accessories for the UltraTablet Pro, including cases, screen protectors, chargers, and adapters."}], "split": "TRAIN", "metadata": {"product": "UltraPhone X"}}
27+
{"messages": [{"role": "system", "content": "You are a helpful customer service assistant for Ultra electronics products."}, {"role": "user", "content": "What accessories are available for the UltraTablet Pro?"}, {"role": "assistant", "content": "We offer a range of accessories for the UltraTablet Pro, including cases, screen protectors, chargers, and adapters."}], "split": "TRAIN", "metadata": {"product": "UltraPhone SE"}}
28+
{"messages": [{"role": "system", "content": "You are a helpful customer service assistant for Ultra electronics products."}, {"role": "user", "content": "Tell me about the features of UltraPhone SE"}, {"role": "assistant", "content": "The UltraPhone SE offers 4G, 5.8-inch display, 8MP camera, 128GB storage. Is there a specific feature you're interested in?"}], "split": "TEST", "metadata": {"product": "UltraTablet Pro"}}
29+
{"messages": [{"role": "system", "content": "You are a helpful customer service assistant for Ultra electronics products."}, {"role": "user", "content": "Is the UltraPhone X compatible with my other devices?"}, {"role": "assistant", "content": "Yes, the UltraPhone X is compatible with all other Ultra devices and most third-party products. It supports standard connectivity options."}], "split": "TEST", "metadata": {"product": "UltraWatch"}}
30+
{"messages": [{"role": "system", "content": "You are a helpful customer service assistant for Ultra electronics products."}, {"role": "user", "content": "Tell me about the features of UltraTablet Pro"}, {"role": "assistant", "content": "The UltraTablet Pro offers 10.5-inch display, 512GB storage, keyboard included. Is there a specific feature you're interested in?"}], "split": "TEST", "metadata": {"product": "UltraBook Air"}}

‎pipelines/__init__.py‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
# Apache Software License 2.0
2+
#
3+
# Copyright (c) ZenML GmbH 2025. All rights reserved.
4+
#
5+
# Licensed under the Apache License, Version 2.0 (the "License");
6+
# you may not use this file except in compliance with the License.
7+
# You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
#
17+
18+
from .openpipe_finetuning import openpipe_finetuning

0 commit comments

Comments
 (0)