Skip to content

Commit cb02722

Browse files
HumairAKHumairAK
committed
feat(backend): replace MLMD with native artifact/task storage
Remove ML Metadata (MLMD) service dependency and implement artifact and task tracking directly in the KFP database via the API server. This architectural change eliminates the external MLMD service (metadata-grpc, metadata-writer) and consolidates all metadata operations through the KFP API. Major changes: - Add v2beta1 artifact service API with storage layer implementation - Extend run service with task CRUD endpoints and ViewMode - Extend run response object with detailed task information - Refactor driver/launcher to use KFP API client instead of MLMD client - Remove all MLMD-related deployments and manifests - Remove object store session info storage in metadata layer - Add comprehensive test coverage for new storage and API layers This simplifies deployment, reduces operational complexity, and provides better control over metadata storage performance and schema. Signed-off-by: Humair Khan <[email protected]> # Conflicts: # backend/api/v2beta1/go_client/run.pb.go # backend/api/v2beta1/go_client/run.pb.gw.go # backend/api/v2beta1/go_client/run_grpc.pb.go # backend/api/v2beta1/swagger/kfp_api_single_file.swagger.json # backend/metadata_writer/src/metadata_helpers.py # backend/src/apiserver/resource/resource_manager.go # backend/src/v2/cmd/driver/main.go # backend/src/v2/compiler/argocompiler/container.go # backend/src/v2/compiler/argocompiler/importer.go # backend/src/v2/driver/driver.go # backend/src/v2/driver/driver_test.go # backend/src/v2/metadata/env.go # manifests/kustomize/env/cert-manager/base-tls-certs/kfp-api-cert.yaml # manifests/kustomize/env/cert-manager/platform-agnostic-standalone-tls/patches/metadata-writer-deployment.yaml # test_data/compiled-workflows/components_with_optional_artifacts.yaml # test_data/compiled-workflows/modelcar.yaml # test_data/compiled-workflows/pipeline_with_dynamic_importer_metadata.yaml # test_data/compiled-workflows/pipeline_with_google_artifact_type.yaml # test_data/compiled-workflows/pipeline_with_importer.yaml # test_data/compiled-workflows/pipeline_with_importer_and_gcpc_types.yaml # test_data/compiled-workflows/pipeline_with_string_machine_fields_task_output.yaml # test_data/compiled-workflows/pythonic_artifact_with_single_return.yaml # test_data/compiled-workflows/ray_job_integration_compiled.yaml
1 parent 0302125 commit cb02722

File tree

589 files changed

+64948
-19550
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

589 files changed

+64948
-19550
lines changed

‎.github/actions/deploy/action.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -64,7 +64,7 @@ runs:
6464
- name: Load Docker Images
6565
shell: bash
6666
run: |
67-
APPS=("apiserver" "driver" "launcher" "scheduledworkflow" "persistenceagent" "frontend" "metadata-writer")
67+
APPS=("apiserver" "driver" "launcher" "scheduledworkflow" "persistenceagent" "frontend")
6868
for app in "${APPS[@]}"; do
6969
docker image load -i ${{ inputs.image_path }}/$app/$app.tar
7070
docker push ${{ inputs.image_registry }}/$app:${{ inputs.image_tag }}

‎.github/actions/kfp-k8s/action.yml‎

Lines changed: 13 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
1-
name: "Install kfp & kfp-kubernetes"
2-
description: "Install kfp & kfp-kubernetes"
1+
name: "Install kfp-server-api, kfp & kfp-kubernetes"
2+
description: "Install kfp-server-api, kfp & kfp-kubernetes from source"
33
inputs:
44
build_version:
55
required: true
@@ -16,6 +16,13 @@ runs:
1616
shell: bash
1717
run: pip install build==${{inputs.build_version}}
1818

19+
- name: Build kfp-server-api dist
20+
id: build-kfp-server-api
21+
shell: bash
22+
working-directory: backend/api/v2beta1/python_http_client
23+
run: |
24+
python -m build .
25+
1926
- name: Build kfp dist
2027
id: install-kfp
2128
shell: bash
@@ -37,15 +44,15 @@ runs:
3744
working-directory: ./kubernetes_platform
3845
run: make golang
3946

40-
# kfp is installed transitively
41-
# --find-links ensures pip first looks in the sdk/python/dist folder
42-
# outputted from generate-kfp-kubernetes-proto-files step before looking at pypi
47+
# kfp and kfp-server-api are installed transitively
48+
# --find-links ensures pip first looks in the dist folders before looking at pypi
49+
# for kfp-server-api and kfp packages
4350
- name: Install kfp & kfp-kubernetes from source
4451
id: install-kfp-kubernetes
4552
shell: bash
4653
if: ${{ steps.generate-kfp-kubernetes-proto-files.outcome == 'success' }}
4754
run: |
48-
pip install -e ./kubernetes_platform/python[dev] --find-links=sdk/python/dist
55+
pip install -e ./kubernetes_platform/python[dev] --find-links=backend/api/v2beta1/python_http_client/dist --find-links=sdk/python/dist
4956
5057
# testing reinstalling kfp package from source with no deps
5158
- name: Reinstall kfp from source with no deps

‎.github/resources/manifests/base/grpc-specs.yaml‎

Lines changed: 0 additions & 38 deletions
This file was deleted.

‎.github/resources/manifests/kubernetes-native/default/kustomization.yaml‎

Lines changed: 0 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -18,19 +18,12 @@ images:
1818
- name: ghcr.io/kubeflow/kfp-frontend
1919
newName: kind-registry:5000/frontend
2020
newTag: latest
21-
- name: ghcr.io/kubeflow/kfp-metadata-writer
22-
newName: kind-registry:5000/metadata-writer
23-
newTag: latest
2421

2522
patches:
2623
- path: ../../base/apiserver-env.yaml
2724
target:
2825
kind: Deployment
2926
name: ml-pipeline
30-
- path: ../../base/grpc-specs.yaml
31-
target:
32-
kind: Deployment
33-
name: metadata-grpc-deployment
3427
- path: ../../base/cache-specs.yaml
3528
target:
3629
kind: Deployment
@@ -47,11 +40,6 @@ replacements:
4740
name: ml-pipeline
4841
fieldPaths:
4942
- spec.template.spec.dnsConfig.searches.[=NAMESPACE.svc.cluster.local]
50-
- select:
51-
kind: Deployment
52-
name: metadata-grpc-deployment
53-
fieldPaths:
54-
- spec.template.spec.dnsConfig.searches.[=NAMESPACE.svc.cluster.local]
5543
- select:
5644
kind: Deployment
5745
name: cache-server

‎.github/resources/manifests/multiuser/artifact-proxy/kustomization.yaml‎

Lines changed: 0 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -18,19 +18,12 @@ images:
1818
- name: ghcr.io/kubeflow/kfp-frontend
1919
newName: kind-registry:5000/frontend
2020
newTag: latest
21-
- name: ghcr.io/kubeflow/kfp-metadata-writer
22-
newName: kind-registry:5000/metadata-writer
23-
newTag: latest
2421

2522
patches:
2623
- path: ../../base/apiserver-env.yaml
2724
target:
2825
kind: Deployment
2926
name: ml-pipeline
30-
- path: ../../base/grpc-specs.yaml
31-
target:
32-
kind: Deployment
33-
name: metadata-grpc-deployment
3427
- path: ../../base/cache-specs.yaml
3528
target:
3629
kind: Deployment
@@ -63,11 +56,6 @@ replacements:
6356
name: ml-pipeline
6457
fieldPaths:
6558
- spec.template.spec.dnsConfig.searches.[=NAMESPACE.svc.cluster.local]
66-
- select:
67-
kind: Deployment
68-
name: metadata-grpc-deployment
69-
fieldPaths:
70-
- spec.template.spec.dnsConfig.searches.[=NAMESPACE.svc.cluster.local]
7159
- select:
7260
kind: Deployment
7361
name: cache-server

‎.github/resources/manifests/multiuser/cache-disabled/kustomization.yaml‎

Lines changed: 0 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -18,10 +18,6 @@ images:
1818
- name: ghcr.io/kubeflow/kfp-frontend
1919
newName: kind-registry:5000/frontend
2020
newTag: latest
21-
- name: ghcr.io/kubeflow/kfp-metadata-writer
22-
newName: kind-registry:5000/metadata-writer
23-
newTag: latest
24-
2521
patches:
2622
- path: ../../base/apiserver-env.yaml
2723
target:
@@ -31,10 +27,6 @@ patches:
3127
target:
3228
kind: Deployment
3329
name: ml-pipeline
34-
- path: ../../base/grpc-specs.yaml
35-
target:
36-
kind: Deployment
37-
name: metadata-grpc-deployment
3830
- path: ../../base/cache-specs.yaml
3931
target:
4032
kind: Deployment
@@ -51,11 +43,6 @@ replacements:
5143
name: ml-pipeline
5244
fieldPaths:
5345
- spec.template.spec.dnsConfig.searches.[=NAMESPACE.svc.cluster.local]
54-
- select:
55-
kind: Deployment
56-
name: metadata-grpc-deployment
57-
fieldPaths:
58-
- spec.template.spec.dnsConfig.searches.[=NAMESPACE.svc.cluster.local]
5946
- select:
6047
kind: Deployment
6148
name: cache-server

‎.github/resources/manifests/multiuser/default/kustomization.yaml‎

Lines changed: 0 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -18,19 +18,12 @@ images:
1818
- name: ghcr.io/kubeflow/kfp-frontend
1919
newName: kind-registry:5000/frontend
2020
newTag: latest
21-
- name: ghcr.io/kubeflow/kfp-metadata-writer
22-
newName: kind-registry:5000/metadata-writer
23-
newTag: latest
2421

2522
patches:
2623
- path: ../../base/apiserver-env.yaml
2724
target:
2825
kind: Deployment
2926
name: ml-pipeline
30-
- path: ../../base/grpc-specs.yaml
31-
target:
32-
kind: Deployment
33-
name: metadata-grpc-deployment
3427
- path: ../../base/cache-specs.yaml
3528
target:
3629
kind: Deployment
@@ -47,11 +40,6 @@ replacements:
4740
name: ml-pipeline
4841
fieldPaths:
4942
- spec.template.spec.dnsConfig.searches.[=NAMESPACE.svc.cluster.local]
50-
- select:
51-
kind: Deployment
52-
name: metadata-grpc-deployment
53-
fieldPaths:
54-
- spec.template.spec.dnsConfig.searches.[=NAMESPACE.svc.cluster.local]
5543
- select:
5644
kind: Deployment
5745
name: cache-server

‎.github/resources/manifests/multiuser/minio/kustomization.yaml‎

Lines changed: 0 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -18,19 +18,12 @@ images:
1818
- name: ghcr.io/kubeflow/kfp-frontend
1919
newName: kind-registry:5000/frontend
2020
newTag: latest
21-
- name: ghcr.io/kubeflow/kfp-metadata-writer
22-
newName: kind-registry:5000/metadata-writer
23-
newTag: latest
2421

2522
patches:
2623
- path: ../../base/apiserver-env.yaml
2724
target:
2825
kind: Deployment
2926
name: ml-pipeline
30-
- path: ../../base/grpc-specs.yaml
31-
target:
32-
kind: Deployment
33-
name: metadata-grpc-deployment
3427
- path: ../../base/cache-specs.yaml
3528
target:
3629
kind: Deployment
@@ -47,11 +40,6 @@ replacements:
4740
name: ml-pipeline
4841
fieldPaths:
4942
- spec.template.spec.dnsConfig.searches.[=NAMESPACE.svc.cluster.local]
50-
- select:
51-
kind: Deployment
52-
name: metadata-grpc-deployment
53-
fieldPaths:
54-
- spec.template.spec.dnsConfig.searches.[=NAMESPACE.svc.cluster.local]
5543
- select:
5644
kind: Deployment
5745
name: cache-server

‎.github/resources/manifests/standalone/cache-disabled-proxy-minio/apiserver-env.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,4 +15,4 @@ spec:
1515
- name: HTTPS_PROXY
1616
value: "http://squid.squid.svc.cluster.local:3128"
1717
- name: NO_PROXY
18-
value: "localhost,127.0.0.1,.svc.cluster.local,kubernetes.default.svc,minio-service.kubeflow,metadata-grpc-service,metadata-grpc-service.kubeflow,ml-pipeline.kubeflow"
18+
value: "localhost,127.0.0.1,.svc.cluster.local,kubernetes.default.svc,minio-service.kubeflow,ml-pipeline.kubeflow"

‎.github/resources/manifests/standalone/cache-disabled-proxy/apiserver-env.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,6 @@ spec:
1515
- name: HTTPS_PROXY
1616
value: "http://squid.squid.svc.cluster.local:3128"
1717
- name: NO_PROXY
18-
value: "localhost,127.0.0.1,.svc.cluster.local,kubernetes.default.svc,minio-service.kubeflow,metadata-grpc-service,metadata-grpc-service.kubeflow,ml-pipeline.kubeflow"
18+
value: "localhost,127.0.0.1,.svc.cluster.local,kubernetes.default.svc,minio-service.kubeflow,ml-pipeline.kubeflow"
1919
- name: OBJECTSTORECONFIG_HOST
2020
value: "minio-service.kubeflow.svc.cluster.local"

0 commit comments

Comments
 (0)