Skip to content

Commit 5f0fc69

Browse files
TimPietruskyTimPietruskyRunPod
TimPietrusky
and
TimPietruskyRunPod
authored
feat: prepare worker-vllm for the hub (#214)
* docs: remove outdated video; remove old info; added missing config for tools * ci: use proper release for dev (pr only) and production (release only) * ci(hub): added openai example; use smollm2 as base model * docs: added conventions to be able to work with ai ide's * chore: remove outdated stuff * chore: update copyright to 2025 * ci: added github permissions * feat: added gpuIds, gputCount and allowedCudaVersions; removed default value for LOAD_FORMAT to check which influence this has on the ui --------- Co-authored-by: Tim Pietrusky <[email protected]>
1 parent aef1187 commit 5f0fc69

File tree

12 files changed

+942
-474
lines changed

12 files changed

+942
-474
lines changed

‎.github/CONTRIBUTING.md‎

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
# Contributing to worker-vllm
2+
3+
## 🚀 Release Process
4+
5+
### Development Workflow
6+
7+
1. **Feature Development**
8+
9+
```bash
10+
git checkout -b feature/your-feature-name
11+
# Make your changes
12+
git push origin feature/your-feature-name
13+
```
14+
15+
- Creates pull request → triggers dev build: `runpod/worker-v1-vllm:dev-feature-your-feature-name`
16+
17+
2. **Main Branch**
18+
```bash
19+
git checkout main
20+
git merge feature/your-feature-name
21+
git push origin main
22+
```
23+
- No automatic builds on main (staging area)
24+
25+
### Creating Releases
26+
27+
**Method 1: GitHub UI (Recommended)**
28+
29+
1. Go to [Releases](https://github.com/runpod-workers/worker-vllm/releases)
30+
2. Click **"Create a new release"**
31+
3. **Tag version**: `v2.8.0` (with "v" prefix, semantic versioning)
32+
4. **Target**: `main` branch
33+
5. **Title**: `Release 2.8.0`
34+
6. **Description**: Brief changelog
35+
7. Click **"Publish release"**
36+
37+
**Method 2: Git CLI**
38+
39+
```bash
40+
git checkout main
41+
git tag v2.8.0
42+
git push origin v2.8.0
43+
```
44+
45+
### What Happens Automatically
46+
47+
✅ **GitHub Release** created (if using Method 1)
48+
✅ **Docker Image** built and pushed: `runpod/worker-v1-vllm:v2.8.0`
49+
✅ **Documentation** updated with new version references
50+
51+
## 📋 Version Format
52+
53+
- **Format**: `vMAJOR.MINOR.PATCH` (e.g., `v2.8.0`)
54+
- **With "v" prefix**: Use `v2.8.0` for git tags
55+
- **Semantic Versioning**: Follow [SemVer](https://semver.org/)
56+
57+
## 🐛 Development
58+
59+
### Running Tests
60+
61+
```bash
62+
# Update test configuration in .runpod/tests.json
63+
# Tests run automatically via RunPod platform
64+
```
65+
66+
### Model Updates
67+
68+
- Update `MODEL_NAME` in `.runpod/tests.json` and `worker-config.json`
69+
- Ensure model has vLLM support and chat template (for OpenAI compatibility)
70+
71+
### Environment Variables
72+
73+
See [README.md](../README.md) for full list of supported environment variables.
74+
75+
## 🔧 CI/CD Workflows
76+
77+
- **Dev builds**: All pull requests → `dev-<branch-name>` images
78+
- **Release builds**: Git tags → versioned images + GitHub releases
79+
- **Manual triggers**: Available in GitHub Actions for emergency releases

‎.github/workflows/dev.yml‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
name: Development
2+
3+
on:
4+
pull_request:
5+
branches:
6+
- "**"
7+
8+
permissions:
9+
contents: read
10+
11+
jobs:
12+
dev:
13+
runs-on: [blacksmith-8vcpu-ubuntu-2204, linux]
14+
steps:
15+
- name: Checkout
16+
uses: actions/checkout@v3
17+
18+
- name: Clear space to remove unused folders
19+
run: |
20+
rm -rf /usr/share/dotnet
21+
rm -rf /opt/ghc
22+
rm -rf "/usr/local/share/boost"
23+
rm -rf "$AGENT_TOOLSDIRECTORY"
24+
25+
- name: Set up QEMU
26+
uses: docker/setup-qemu-action@v3
27+
28+
- name: Set up Docker Buildx
29+
uses: docker/setup-buildx-action@v3
30+
31+
- name: Login to Docker Hub
32+
uses: docker/login-action@v3
33+
with:
34+
username: ${{ secrets.DOCKERHUB_USERNAME }}
35+
password: ${{ secrets.DOCKERHUB_TOKEN }}
36+
37+
- name: blacksmith docker layer cache
38+
uses: useblacksmith/build-push-action@v1
39+
with:
40+
setup-only: true
41+
42+
- name: Set environment variables
43+
run: |
44+
echo "DOCKERHUB_REPO=${{ vars.DOCKERHUB_REPO || 'runpod' }}" >> $GITHUB_ENV
45+
echo "DOCKERHUB_IMG=${{ vars.DOCKERHUB_IMG || 'worker-v1-vllm' }}" >> $GITHUB_ENV
46+
echo "HUGGINGFACE_ACCESS_TOKEN=${{ secrets.HUGGINGFACE_ACCESS_TOKEN }}" >> $GITHUB_ENV
47+
# Convert branch name to safe docker tag (replace / with -)
48+
BRANCH_NAME="${GITHUB_REF##refs/heads/}"
49+
SAFE_BRANCH_NAME=$(echo "$BRANCH_NAME" | sed 's/[^a-zA-Z0-9._-]/-/g' | sed 's/--*/-/g')
50+
echo "RELEASE_VERSION=dev-${SAFE_BRANCH_NAME}" >> $GITHUB_ENV
51+
52+
- name: Build and push the images to Docker Hub
53+
uses: docker/bake-action@v2
54+
with:
55+
push: true
56+
set: |
57+
*.args.DOCKERHUB_REPO=${{ env.DOCKERHUB_REPO }}
58+
*.args.DOCKERHUB_IMG=${{ env.DOCKERHUB_IMG }}
59+
*.args.RELEASE_VERSION=${{ env.RELEASE_VERSION }}
60+
*.args.HUGGINGFACE_ACCESS_TOKEN=${{ env.HUGGINGFACE_ACCESS_TOKEN }}

‎.github/workflows/release.yml‎

Lines changed: 102 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
name: Release
2+
3+
on:
4+
push:
5+
tags:
6+
- "v[0-9]+.[0-9]+.[0-9]+*" # Trigger on version tags like v1.0.0, v2.1.0, etc.
7+
workflow_dispatch:
8+
inputs:
9+
version:
10+
description: "Version to release (e.g., v2.8.0)"
11+
required: true
12+
type: string
13+
14+
permissions:
15+
contents: write # Required for creating GitHub releases
16+
packages: write # Required for pushing Docker images (if using GitHub packages)
17+
18+
jobs:
19+
release:
20+
runs-on: [blacksmith-8vcpu-ubuntu-2204, linux]
21+
steps:
22+
- name: Checkout
23+
uses: actions/checkout@v3
24+
25+
- name: Clear space to remove unused folders
26+
run: |
27+
rm -rf /usr/share/dotnet
28+
rm -rf /opt/ghc
29+
rm -rf "/usr/local/share/boost"
30+
rm -rf "$AGENT_TOOLSDIRECTORY"
31+
32+
- name: Set up QEMU
33+
uses: docker/setup-qemu-action@v3
34+
35+
- name: Set up Docker Buildx
36+
uses: docker/setup-buildx-action@v3
37+
38+
- name: Login to Docker Hub
39+
uses: docker/login-action@v3
40+
with:
41+
username: ${{ secrets.DOCKERHUB_USERNAME }}
42+
password: ${{ secrets.DOCKERHUB_TOKEN }}
43+
44+
- name: blacksmith docker layer cache
45+
uses: useblacksmith/build-push-action@v1
46+
with:
47+
setup-only: true
48+
49+
- name: Set environment variables
50+
run: |
51+
echo "DOCKERHUB_REPO=${{ vars.DOCKERHUB_REPO || 'runpod' }}" >> $GITHUB_ENV
52+
echo "DOCKERHUB_IMG=${{ vars.DOCKERHUB_IMG || 'worker-v1-vllm' }}" >> $GITHUB_ENV
53+
echo "HUGGINGFACE_ACCESS_TOKEN=${{ secrets.HUGGINGFACE_ACCESS_TOKEN }}" >> $GITHUB_ENV
54+
55+
# Determine version based on trigger type
56+
if [[ "${{ github.event_name }}" == "workflow_dispatch" ]]; then
57+
# Manual trigger: use input version
58+
VERSION="${{ github.event.inputs.version }}"
59+
echo "RELEASE_VERSION=${VERSION}" >> $GITHUB_ENV
60+
echo "IS_MANUAL_RELEASE=true" >> $GITHUB_ENV
61+
else
62+
# Tag trigger: use tag name (remove refs/tags/ prefix)
63+
VERSION=${GITHUB_REF#refs/tags/}
64+
echo "RELEASE_VERSION=${VERSION}" >> $GITHUB_ENV
65+
echo "IS_MANUAL_RELEASE=false" >> $GITHUB_ENV
66+
fi
67+
68+
- name: Build and push the images to Docker Hub
69+
uses: docker/bake-action@v2
70+
with:
71+
push: true
72+
set: |
73+
*.args.DOCKERHUB_REPO=${{ env.DOCKERHUB_REPO }}
74+
*.args.DOCKERHUB_IMG=${{ env.DOCKERHUB_IMG }}
75+
*.args.RELEASE_VERSION=${{ env.RELEASE_VERSION }}
76+
*.args.HUGGINGFACE_ACCESS_TOKEN=${{ env.HUGGINGFACE_ACCESS_TOKEN }}
77+
78+
- name: Create GitHub Release
79+
if: env.IS_MANUAL_RELEASE == 'false'
80+
uses: actions/create-release@v1
81+
env:
82+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
83+
with:
84+
tag_name: ${{ github.ref_name }}
85+
release_name: Release ${{ github.ref_name }}
86+
body: |
87+
Release ${{ github.ref_name }}
88+
89+
Docker Image: `${{ env.DOCKERHUB_REPO }}/${{ env.DOCKERHUB_IMG }}:${{ env.RELEASE_VERSION }}`
90+
91+
## Changes
92+
See [commit history](https://github.com/${{ github.repository }}/commits/${{ github.ref_name }}) for detailed changes.
93+
draft: false
94+
prerelease: false
95+
96+
- name: Manual Release Summary
97+
if: env.IS_MANUAL_RELEASE == 'true'
98+
run: |
99+
echo "🚀 Manual release completed!"
100+
echo "Version: ${{ env.RELEASE_VERSION }}"
101+
echo "Docker Image: ${{ env.DOCKERHUB_REPO }}/${{ env.DOCKERHUB_IMG }}:${{ env.RELEASE_VERSION }}"
102+
echo "Note: No GitHub release created for manual triggers"

‎.gitmodules‎

Lines changed: 0 additions & 3 deletions
This file was deleted.

‎.runpod/hub.json‎

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,19 @@
77
"config": {
88
"runsOn": "GPU",
99
"containerDiskInGb": 200,
10+
"gpuIds": "ADA_80_PRO, AMPERE_80",
11+
"gpuCount": 1,
12+
"allowedCudaVersions": [
13+
"12.9",
14+
"12.8",
15+
"12.7",
16+
"12.6",
17+
"12.5",
18+
"12.4",
19+
"12.3",
20+
"12.2",
21+
"12.1"
22+
],
1023
"presets": [
1124
{
1225
"name": "deepseek-ai/deepseek-r1-distill-llama-8b",
@@ -129,7 +142,6 @@
129142
"value": "bitsandbytes"
130143
}
131144
],
132-
"default": "auto",
133145
"advanced": true
134146
}
135147
},

‎.runpod/tests.json‎

Lines changed: 46 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -1,32 +1,52 @@
11
{
2-
"tests": [
3-
{
4-
"name": "basic_inference_test",
5-
"input": {
6-
"prompt": "Write a short poem about artificial intelligence."
2+
"tests": [
3+
{
4+
"name": "basic_inference_test",
5+
"input": {
6+
"prompt": "Write a short poem about artificial intelligence."
7+
},
8+
"timeout": 30000
9+
},
10+
{
11+
"name": "openai_messages_test",
12+
"input": {
13+
"openai_route": "/v1/chat/completions",
14+
"openai_input": {
15+
"model": "HuggingFaceTB/SmolLM2-135M-Instruct",
16+
"messages": [
17+
{
18+
"role": "system",
19+
"content": "You are a helpful assistant that writes concise responses."
720
},
8-
"timeout": 30000
9-
}
10-
],
11-
"config": {
12-
"gpuTypeId": "NVIDIA GeForce RTX 4090",
13-
"gpuCount": 1,
14-
"env": [
1521
{
16-
"key": "MODEL_NAME",
17-
"value": "facebook/opt-350m"
22+
"role": "user",
23+
"content": "Explain what a neural network is in one sentence."
1824
}
19-
],
20-
"allowedCudaVersions": [
21-
"12.7",
22-
"12.6",
23-
"12.5",
24-
"12.4",
25-
"12.3",
26-
"12.2",
27-
"12.1",
28-
"12.0",
29-
"11.7"
30-
]
25+
],
26+
"max_tokens": 50,
27+
"temperature": 0.7
28+
}
29+
},
30+
"timeout": 30000
3131
}
32+
],
33+
"config": {
34+
"gpuTypeId": "NVIDIA GeForce RTX 4090",
35+
"gpuCount": 1,
36+
"env": [
37+
{
38+
"key": "MODEL_NAME",
39+
"value": "HuggingFaceTB/SmolLM2-135M-Instruct"
40+
}
41+
],
42+
"allowedCudaVersions": [
43+
"12.7",
44+
"12.6",
45+
"12.5",
46+
"12.4",
47+
"12.3",
48+
"12.2",
49+
"12.1"
50+
]
51+
}
3252
}

‎LICENSE‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
MIT License
22

3-
Copyright (c) 2023 runpod-workers
3+
Copyright (c) 2025 Runpod
44

55
Permission is hereby granted, free of charge, to any person obtaining a copy
66
of this software and associated documentation files (the "Software"), to deal

0 commit comments

Comments
 (0)