Skip to content

Refactor: use shared dependencies across crates #153

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 68 commits into
base: develop
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
68 commits
Select commit Hold shift + click to select a range
c42793f
ability to automatically set shm size based on sys memory
JannikSt Feb 6, 2025
f1f7538
clippy
JannikSt Feb 6, 2025
6d4aa01
Merge pull request #107 from PrimeIntellect-ai/improvement/shm-size-s…
JannikSt Feb 6, 2025
ca57e74
bump version
JannikSt Feb 6, 2025
faf4375
Merge pull request #108 from PrimeIntellect-ai/improvement/shm-size-s…
JannikSt Feb 6, 2025
ad90427
Feature/disable ejection (#111)
JannikSt Feb 6, 2025
bf52611
support ability to restore metrics via orchestrator (#112)
JannikSt Feb 6, 2025
8e8808a
bump version
JannikSt Feb 6, 2025
535e0c3
merge
JannikSt Feb 6, 2025
b698e32
improve resiliency of validator (#114)
JannikSt Feb 6, 2025
7ddf448
bump version
JannikSt Feb 6, 2025
bbabc26
resolve conflicts
JannikSt Feb 6, 2025
b7adeeb
add improvement to metrics reporting (#119)
JannikSt Feb 12, 2025
519307a
resolve conflicts
JannikSt Feb 12, 2025
0d3d5cd
fmt
JannikSt Feb 12, 2025
8cb71b4
very basic validator functionality
mattdf Feb 4, 2025
bf88c80
fmt
mattdf Feb 4, 2025
c58574b
add signature ...
mattdf Feb 4, 2025
92629cd
use sign_request
mattdf Feb 4, 2025
75e055d
use custom serializer for consistency
mattdf Feb 5, 2025
61a482b
add validator arg to miner, implement rounding robust partial eq
mattdf Feb 5, 2025
36349d6
fmt, clippy fix
mattdf Feb 5, 2025
7a47331
fix remote makefile entry
mattdf Feb 5, 2025
9a85c57
misc makefile fixes
mattdf Feb 5, 2025
58f97ba
fix clippy
mattdf Feb 12, 2025
9e801d3
fix fmt...
mattdf Feb 12, 2025
7255a74
make app_state unused
mattdf Feb 12, 2025
e172fbd
Merge pull request #96 from PrimeIntellect-ai/feature/validator-chall…
mattdf Feb 12, 2025
7350607
minor readme adjustment
JannikSt Feb 13, 2025
c3480c9
remove redundant files
JannikSt Feb 13, 2025
07b440b
add license file (#124)
mattdf Feb 13, 2025
463ec6c
basic adjustment of readme, add security, add contributing and docs (…
JannikSt Feb 13, 2025
888810f
fix worker naming in readme
JannikSt Feb 13, 2025
593520f
add readme images (#125)
JannikSt Feb 14, 2025
5849a61
resolve conflicts
JannikSt Feb 14, 2025
12596a9
add disclosure
JannikSt Feb 14, 2025
4c6e15b
add video to remote gpu setup in docs (#128)
JannikSt Feb 26, 2025
047a514
fix submodule setup (#129)
JannikSt Feb 27, 2025
a46c301
Fix/submodule path (#130)
JannikSt Feb 27, 2025
10ea9d7
do not run format on xl-runner
JannikSt Mar 7, 2025
e217951
fix runner setup
JannikSt Mar 7, 2025
b2fe49d
add shared volume to containers to support model downloads (#134)
JannikSt Mar 7, 2025
65753cc
- Readme improvements (#131)
burnpiro Mar 7, 2025
36bb005
feature/toploc-integration (#127)
JannikSt Mar 10, 2025
82a9b4d
align smart contract module (#135)
JannikSt Mar 12, 2025
40fcaf8
support dynamic staking system and ability to automatically increase …
JannikSt Mar 12, 2025
c76d601
fix worker unit formatting (#138)
JannikSt Mar 15, 2025
541430f
remove s3 from release pipeline (#141)
JannikSt Mar 15, 2025
b72760d
support new env vars in validator docker (#143)
JannikSt Mar 15, 2025
ea1da66
improve orchestrator docker image (#144)
JannikSt Mar 15, 2025
4094f5e
fix stake setting approach (#142)
JannikSt Mar 15, 2025
7f306a6
Feature/docker compose setup (#145)
JannikSt Mar 17, 2025
9420a14
add basic ability to generate new wallets from worker cli (#147)
JannikSt Mar 18, 2025
a5716fc
support toploc-server auth (#146)
JannikSt Mar 19, 2025
624128f
move private keys into env files rather than cli args (#150)
JannikSt Mar 19, 2025
089b8dc
only allow discovery svc upload if you have ai token (#149)
JannikSt Mar 19, 2025
d2576a9
support setting penalty in validator arg (#137)
JannikSt Mar 19, 2025
f503fb7
Feature/discovery blacklist detection (#139)
JannikSt Mar 19, 2025
5980e8d
Improvement/discovery security check (#151)
JannikSt Mar 19, 2025
527d7dd
update email in security.md (#154)
JohannesHa Mar 20, 2025
f0bce1c
add install scripts (#155)
JannikSt Mar 20, 2025
914aaf9
Feature/node chain sync (#148)
JannikSt Mar 21, 2025
fb7c9e3
improve boot sequence (#157)
JannikSt Mar 21, 2025
5721523
Improvement/manual transcation approval (#152)
JannikSt Mar 21, 2025
f5e40f4
fix typo (#158)
samsja Mar 21, 2025
0fb0bba
Refactor: use shared dependencies across crates
gsegatti Mar 19, 2025
d16d312
Apply clippy suggestion
gsegatti Mar 20, 2025
9f39afd
add suggestions
gsegatti Mar 20, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 10 additions & 3 deletions .env.example
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
RPC_URL=
NETWORK_ID=
MIN_STAKE_AMOUNT=
WORKER_COMPUTE_POOL_ID=
WORKER_EXTERNAL_IP=

# Private keys of privileged accounts
PRIVATE_KEY_FEDERATOR=
Expand All @@ -9,9 +11,9 @@ PRIVATE_KEY_VALIDATOR=
VALIDATOR_ADDRESS=

# Provider with their node
PROVIDER_PRIVATE_KEY=
PRIVATE_KEY_PROVIDER=
PROVIDER_ADDRESS=
NODE_PRIVATE_KEY=
PRIVATE_KEY_NODE=
NODE_ADDRESS=

# Pool Owner
Expand All @@ -24,4 +26,9 @@ AI_TOKEN_ADDRESS=
COMPUTE_REGISTRY_ADDRESS=
DOMAIN_REGISTRY_ADDRESS=
STAKE_MANAGER_ADDRESS=
COMPUTE_POOL_ADDRESS=
COMPUTE_POOL_ADDRESS=

WORK_VALIDATION_CONTRACT=

LEVITICUS_URL=
S3_CREDENTIALS=
2 changes: 1 addition & 1 deletion .github/workflows/checks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ env:
jobs:
check:
name: Format & Lint
runs-on: xl-runner
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@master
Expand Down
25 changes: 9 additions & 16 deletions .github/workflows/dev-release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -52,8 +52,8 @@ jobs:
- name: Prepare binaries
run: |
mkdir -p release-artifacts
if [ -f target/release/miner ]; then
cp target/release/miner release-artifacts/miner-linux-x86_64
if [ -f target/release/worker ]; then
cp target/release/worker release-artifacts/worker-linux-x86_64
fi
if [ -f target/release/validator ]; then
cp target/release/validator release-artifacts/validator-linux-x86_64
Expand Down Expand Up @@ -96,7 +96,7 @@ jobs:
- Linux x86_64

Components:
- Miner
- worker
- Validator
- Orchestrator
- Discovery svc
Expand All @@ -117,13 +117,6 @@ jobs:
- name: Set up Google Cloud SDK
uses: google-github-actions/setup-gcloud@v1

# Upload to Google Cloud Storage
- name: Upload to GCS
run: |
gsutil -m cp -r release-artifacts/* gs://prime-miner/${{ steps.tag.outputs.tag_name }}/
gsutil -m cp -r gs://prime-miner/${{ steps.tag.outputs.tag_name }}/* gs://prime-miner/latest/
gsutil -m setmeta -h "Cache-Control:no-cache, max-age=0" gs://prime-miner/latest/**/*

- name: Generate Docker metadata
id: meta
run: |
Expand Down Expand Up @@ -153,8 +146,8 @@ jobs:
tags: |
ghcr.io/${{ steps.meta.outputs.repo_lower }}/discovery:dev
ghcr.io/${{ steps.meta.outputs.repo_lower }}/discovery:${{ steps.tag.outputs.tag_name }}
us-east1-docker.pkg.dev/${{ secrets.GCP_PROJECT_ID }}/prime-miner/discovery:dev
us-east1-docker.pkg.dev/${{ secrets.GCP_PROJECT_ID }}/prime-miner/discovery:${{ steps.tag.outputs.tag_name }}
us-east1-docker.pkg.dev/${{ secrets.GCP_PROJECT_ID }}/prime-protocol/discovery:dev
us-east1-docker.pkg.dev/${{ secrets.GCP_PROJECT_ID }}/prime-protocol/discovery:${{ steps.tag.outputs.tag_name }}

- name: Build and push Validator image
uses: docker/build-push-action@v4
Expand All @@ -165,8 +158,8 @@ jobs:
tags: |
ghcr.io/${{ steps.meta.outputs.repo_lower }}/validator:dev
ghcr.io/${{ steps.meta.outputs.repo_lower }}/validator:${{ steps.tag.outputs.tag_name }}
us-east1-docker.pkg.dev/${{ secrets.GCP_PROJECT_ID }}/prime-miner/validator:dev
us-east1-docker.pkg.dev/${{ secrets.GCP_PROJECT_ID }}/prime-miner/validator:${{ steps.tag.outputs.tag_name }}
us-east1-docker.pkg.dev/${{ secrets.GCP_PROJECT_ID }}/prime-protocol/validator:dev
us-east1-docker.pkg.dev/${{ secrets.GCP_PROJECT_ID }}/prime-protocol/validator:${{ steps.tag.outputs.tag_name }}

- name: Build and push Orchestrator image
uses: docker/build-push-action@v4
Expand All @@ -177,5 +170,5 @@ jobs:
tags: |
ghcr.io/${{ steps.meta.outputs.repo_lower }}/orchestrator:dev
ghcr.io/${{ steps.meta.outputs.repo_lower }}/orchestrator:${{ steps.tag.outputs.tag_name }}
us-east1-docker.pkg.dev/${{ secrets.GCP_PROJECT_ID }}/prime-miner/orchestrator:dev
us-east1-docker.pkg.dev/${{ secrets.GCP_PROJECT_ID }}/prime-miner/orchestrator:${{ steps.tag.outputs.tag_name }}
us-east1-docker.pkg.dev/${{ secrets.GCP_PROJECT_ID }}/prime-protocol/orchestrator:dev
us-east1-docker.pkg.dev/${{ secrets.GCP_PROJECT_ID }}/prime-protocol/orchestrator:${{ steps.tag.outputs.tag_name }}
27 changes: 10 additions & 17 deletions .github/workflows/prod-release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ jobs:
if: steps.check_tag.outputs.exists == 'false'
run: |
mkdir -p release-artifacts
for binary in miner validator orchestrator discovery; do
for binary in worker validator orchestrator discovery; do
if [ -f "target/release/$binary" ]; then
cp "target/release/$binary" "release-artifacts/$binary-linux-x86_64"
fi
Expand Down Expand Up @@ -95,7 +95,7 @@ jobs:
- Linux x86_64

Components:
- Miner
- worker
- Validator
- Orchestrator
- Discovery service
Expand All @@ -106,8 +106,8 @@ jobs:
Download the appropriate binary for your system and verify the checksum.

```bash
# Verify checksum (example for miner)
sha256sum -c miner-linux-x86_64.checksum
# Verify checksum (example for worker)
sha256sum -c worker-linux-x86_64.checksum
```

## Changes
Expand All @@ -128,13 +128,6 @@ jobs:
if: steps.check_tag.outputs.exists == 'false'
uses: google-github-actions/setup-gcloud@v1

- name: Upload to GCS
if: steps.check_tag.outputs.exists == 'false'
run: |
gsutil -m cp -r release-artifacts/* gs://prime-miner/v${{ steps.get_version.outputs.version }}/
gsutil -m cp -r release-artifacts/* gs://prime-miner/stable/
gsutil -m setmeta -h "Cache-Control:no-cache, max-age=0" gs://prime-miner/stable/**/*

- name: Configure Docker for Artifact Registry
if: steps.check_tag.outputs.exists == 'false'
run: |
Expand Down Expand Up @@ -169,8 +162,8 @@ jobs:
tags: |
ghcr.io/${{ steps.meta.outputs.repo_lower }}/discovery:latest
ghcr.io/${{ steps.meta.outputs.repo_lower }}/discovery:v${{ steps.get_version.outputs.version }}
us-east1-docker.pkg.dev/${{ secrets.GCP_PROJECT_ID }}/prime-miner/discovery:latest
us-east1-docker.pkg.dev/${{ secrets.GCP_PROJECT_ID }}/prime-miner/discovery:v${{ steps.get_version.outputs.version }}
us-east1-docker.pkg.dev/${{ secrets.GCP_PROJECT_ID }}/prime-protocol/discovery:latest
us-east1-docker.pkg.dev/${{ secrets.GCP_PROJECT_ID }}/prime-protocol/discovery:v${{ steps.get_version.outputs.version }}

- name: Build and push Validator image
if: steps.check_tag.outputs.exists == 'false'
Expand All @@ -182,8 +175,8 @@ jobs:
tags: |
ghcr.io/${{ steps.meta.outputs.repo_lower }}/validator:latest
ghcr.io/${{ steps.meta.outputs.repo_lower }}/validator:v${{ steps.get_version.outputs.version }}
us-east1-docker.pkg.dev/${{ secrets.GCP_PROJECT_ID }}/prime-miner/validator:latest
us-east1-docker.pkg.dev/${{ secrets.GCP_PROJECT_ID }}/prime-miner/validator:v${{ steps.get_version.outputs.version }}
us-east1-docker.pkg.dev/${{ secrets.GCP_PROJECT_ID }}/prime-protocol/validator:latest
us-east1-docker.pkg.dev/${{ secrets.GCP_PROJECT_ID }}/prime-protocol/validator:v${{ steps.get_version.outputs.version }}

- name: Build and push Orchestrator image
if: steps.check_tag.outputs.exists == 'false'
Expand All @@ -195,5 +188,5 @@ jobs:
tags: |
ghcr.io/${{ steps.meta.outputs.repo_lower }}/orchestrator:latest
ghcr.io/${{ steps.meta.outputs.repo_lower }}/orchestrator:v${{ steps.get_version.outputs.version }}
us-east1-docker.pkg.dev/${{ secrets.GCP_PROJECT_ID }}/prime-miner/orchestrator:latest
us-east1-docker.pkg.dev/${{ secrets.GCP_PROJECT_ID }}/prime-miner/orchestrator:v${{ steps.get_version.outputs.version }}
us-east1-docker.pkg.dev/${{ secrets.GCP_PROJECT_ID }}/prime-protocol/orchestrator:latest
us-east1-docker.pkg.dev/${{ secrets.GCP_PROJECT_ID }}/prime-protocol/orchestrator:v${{ steps.get_version.outputs.version }}
4 changes: 3 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -24,4 +24,6 @@ Thumbs.db
/private/

.env
.env.remote
.env.remote

*.pem
2 changes: 1 addition & 1 deletion .gitmodules
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
[submodule "smart-contracts"]
path = smart-contracts
url = git@github.com:PrimeIntellect-ai/smart-contracts.git
url = https://github.com/PrimeIntellect-ai/smart-contracts.git
9 changes: 4 additions & 5 deletions .tmuxinator.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,11 @@ windows:
- services:
layout: even-horizontal
panes:
- bash -c 'tmux select-pane -T "Miner" && sleep 5 && cd smart-contracts && sh deploy.sh && cd .. && make setup && clear'
- bash -c 'tmux select-pane -T "Worker" && sleep 5 && cd smart-contracts && sh deploy.sh && sh deploy_work_validation.sh && cd .. && make setup && clear'
- bash -c 'tmux select-pane -T "Discovery" && sleep 10 && make watch-discovery'
- bash -c 'tmux select-pane -T "Validator" && sleep 15 && make watch-validator'
- bash -c 'tmux select-pane -T "Orchestrator" && sleep 20 && make watch-orchestrator'
- bash -c 'tmux select-pane -T "Orchestrator" && sleep 20 && make watch-orchestrator'
- background:
layout: even-horizontal
layout: even-horizontal
panes:
- bash -c 'tmux rename-window "Prime Dev Services" && docker-compose up'
- bash -c 'while true; do make whitelist-provider; sleep 10; done'
- bash -c 'tmux rename-window "Prime Dev Services" && docker compose up anvil redis'
68 changes: 68 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
# Contributing Guidelines

We love your input! We want to make contributing to this project as easy and transparent as possible, whether it's:

- Reporting a bug
- Discussing the current state of the code
- Submitting a fix
- Proposing new features
- Becoming a maintainer

## We Develop with Github
We use GitHub to host code, to track issues and feature requests, as well as accept pull requests.

## Branch Strategy
We use a two-branch strategy for development:

1. `develop` - This is our main development branch where all feature branches are merged for nightly builds and testing
2. `main` - This is our stable production branch that contains reviewed and tested code

### Development Process

1. Create a new feature branch from `develop`
2. Make your changes and commit them
3. Submit a pull request to merge into `develop`
4. After review and testing in `develop`, changes will be merged into `main` for production releases

## Pull Request Process

1. Fork the repo and create your feature branch from `develop`
2. If you've added code that should be tested, add tests
3. If you've changed APIs, update the documentation
4. Ensure the test suite passes
5. Make sure your code lints
6. Submit a pull request to merge into `develop`

## Report bugs using Github's issue tracker
We use GitHub issues to track public bugs. Report a bug by opening a new issue; it's that easy!

## Write bug reports with detail, background, and sample code

**Great Bug Reports** tend to have:

- A quick summary and/or background
- Steps to reproduce
- Be specific!
- Give sample code if you can.
- What you expected would happen
- What actually happens
- Notes (possibly including why you think this might be happening, or stuff you tried that didn't work)

## Development Process

1. Create a new branch from `develop` for your work
2. Make your changes
3. Write or update tests as needed
4. Update documentation as needed
5. Submit a pull request to `develop`
6. Address any review feedback

### Commit Messages

- Use the present tense ("Add feature" not "Added feature")
- Use the imperative mood ("Move cursor to..." not "Moves cursor to...")
- Limit the first line to 72 characters or less
- Reference issues and pull requests liberally after the first line

## References
This document was adapted from the open-source contribution guidelines for [Facebook's Draft](https://github.com/facebook/draft-js/blob/a9316a723f9e918afde44dea68b5f9f39b7d9b00/CONTRIBUTING.md).
Loading