ci: add istio to e2e #646

andyatmiami · 2025-10-10T20:19:23Z

ℹ️ Refactors work originally submitted by @yashpal2104 to be more aligned with the structure of cert-manager and prometheus.

This commit ensures istiod (and required supporting resources) are available in the k8s cluster used for the test-e2e command. Following behavior is being added:

ISTIO_INSTALL_SKIP to opt out of e2e trying to manage istio entirely
new Makefile targets to support downloading the istioctl binary
InstallXXX + UninstallXXX + IsXXXInstalled + WaitIXXX(Ready|Available) functions implemented both for istiod as well as istio-ingressgateway
Any required (but missing) Istio resources are installed during the BeforeSuite hook of e2e
- istioctl is used to install any required resources
Any Istio resources installed during BeforeSuite are then cleaned up in AfterSuite
- If an Istio resource already existed on the cluster prior to e2e - it is NOT cleaned up
- istioctl is used to uninstall any eligible resources
Refactors e2e/utils package to have separate *.go files based on areas of interest for better maintainability
helper function LabelNamespaceForIstioInjection added for when we go to truly leverage Istio as part of e2e

Please note while the Istio resources are now made availabe on the e2e cluster - no tests are actually using these resources (yet). They will get leveraged as part of #500 .

- Add InstallIstioctl() function to download and install istioctl binary - Add InstallIstioMinimalWithIngress() to set up Istio with minimal profile - Add IsIstioInstalled() and WaitIstioctlAvailable() helper functions - Use positional formatting in URL template for istioctl downloads - Support configurable Istio namespace for installation - Add error handling and proper command output redirection This enables e2e tests to automatically set up Istio service mesh components required for workspace HTTP proxy functionality. Signed-off-by: Yash Pal <[email protected]>

google-oss-prow · 2025-10-10T20:19:52Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from andyatmiami. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

workspaces/controller/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

andyatmiami · 2025-10-10T20:20:59Z

/ok-to-test

andyatmiami · 2025-10-10T20:38:19Z

Verification

"Empty" `kind` cluster

➜ controller/ git:(chore/istio-in-e2e) $ cat <<EOF | KIND_EXPERIMENTAL_PROVIDER=podman kind create cluster --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  image: kindest/node:v1.32.0@sha256:c48c62eac5da28cdadcf560d1d8616cfa6783b58f0d94cf63ad1bf49600cb027
  kubeadmConfigPatches:
  - |
    kind: ClusterConfiguration
    apiServer:
      extraArgs:
        "service-account-issuer": "https://kubernetes.default.svc"
        "service-account-signing-key-file": "/etc/kubernetes/pki/sa.key"
EOF

➜ controller/ git:(chore/istio-in-e2e) $ make test-e2e
...
  STEP: checking if istio is installed already @ 10/10/25 15:44:44.23
  running: kubectl get crd gateways.networking.istio.io
  Installing istio...
  running: /Users/astonebe/Development/Code/GitHub/kubeflow-notebooks/workspaces/controller/bin/istioctl install --set profile=default --revision=default -y
  running: kubectl get crd gateways.networking.istio.io
  running: kubectl get deploy -A -l app=istiod -o jsonpath={.items[*].metadata.namespace}
  STEP: checking that istio is available @ 10/10/25 15:44:57.897
  running: kubectl wait --for=condition=Ready pods -l app=istiod -n istio-system --timeout=300s
  running: kubectl get service istiod -n istio-system -o jsonpath={.metadata.name}
  running: kubectl get mutatingwebhookconfiguration istio-sidecar-injector -o jsonpath={.metadata.name}
  running: kubectl get endpoints istiod -n istio-system -o jsonpath={.subsets[*].addresses[*].ip}
  STEP: checking if istio ingress gateway is installed already @ 10/10/25 15:44:58.158
  running: kubectl get deployment -n istio-system istio-ingressgateway --ignore-not-found
  WARNING: istio ingress gateway is already installed. Skipping installation...
  STEP: checking that istio ingress gateway is available @ 10/10/25 15:44:58.198
  running: kubectl wait --for=condition=Ready pods -l app=istio-ingressgateway -n istio-system --timeout=300s
[BeforeSuite] PASSED [34.060 seconds]
...
------------------------------
[AfterSuite] 
/Users/astonebe/Development/Code/GitHub/kubeflow-notebooks/workspaces/controller/test/e2e/e2e_suite_test.go:126
...
  STEP: uninstalling Istio @ 10/10/25 15:46:27.008
  Uninstalling Istio...
  running: /Users/astonebe/Development/Code/GitHub/kubeflow-notebooks/workspaces/controller/bin/istioctl uninstall --set profile=default --revision=default -y
  running: kubectl delete namespace istio-system --wait=true
[AfterSuite] PASSED [19.143 seconds]
------------------------------

Ran 1 of 1 Specs in 129.425 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 0 Skipped
--- PASS: TestE2E (129.43s)
PASS
ok  	github.com/kubeflow/notebooks/workspaces/controller/test/e2e	130.261s

`kind` cluster with `istiod` (no `ingressgateway`)

➜ controller/ git:(chore/istio-in-e2e) $ cat <<EOF | KIND_EXPERIMENTAL_PROVIDER=podman kind create cluster --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  image: kindest/node:v1.32.0@sha256:c48c62eac5da28cdadcf560d1d8616cfa6783b58f0d94cf63ad1bf49600cb027
  kubeadmConfigPatches:
  - |
    kind: ClusterConfiguration
    apiServer:
      extraArgs:
        "service-account-issuer": "https://kubernetes.default.svc"
        "service-account-signing-key-file": "/etc/kubernetes/pki/sa.key"
EOF

➜ controller/ git:(chore/istio-in-e2e) $ istioctl install --set profile=minimal --revision=default -y
        |\          
        | \         
        |  \        
        |   \       
      /||    \      
     / ||     \     
    /  ||      \    
   /   ||       \   
  /    ||        \  
 /     ||         \ 
/______||__________\
____________________
  \__       _____/  
     \_____/        

✔ Istio core installed ⛵️                                                                                                                                                                      
✔ Istiod installed 🧠                                                                                                                                                                          
✔ Installation complete

➜ controller/ git:(chore/istio-in-e2e) $ make test-e2e
...
[BeforeSuite] 
/Users/astonebe/Development/Code/GitHub/kubeflow-notebooks/workspaces/controller/test/e2e/e2e_suite_test.go:60
...
  STEP: checking if istio is installed already @ 10/10/25 15:48:01.136
  running: kubectl get crd gateways.networking.istio.io
  running: kubectl get deploy -A -l app=istiod -o jsonpath={.items[*].metadata.namespace}
  running: kubectl get deployment istiod -n istio-system
  running: /Users/astonebe/Development/Code/GitHub/kubeflow-notebooks/workspaces/controller/bin/istioctl version --short --remote --istioNamespace=istio-system
  WARNING: istio is already installed in namespace istio-system. Skipping installation...
  STEP: checking that istio is available @ 10/10/25 15:48:01.554
  running: kubectl wait --for=condition=Ready pods -l app=istiod -n istio-system --timeout=300s
  running: kubectl get service istiod -n istio-system -o jsonpath={.metadata.name}
  running: kubectl get mutatingwebhookconfiguration istio-sidecar-injector -o jsonpath={.metadata.name}
  running: kubectl get endpoints istiod -n istio-system -o jsonpath={.subsets[*].addresses[*].ip}
  STEP: checking if istio ingress gateway is installed already @ 10/10/25 15:48:01.823
  running: kubectl get deployment -n istio-system istio-ingressgateway --ignore-not-found
  Installing istio ingress gateway...
  running: /Users/astonebe/Development/Code/GitHub/kubeflow-notebooks/workspaces/controller/bin/istioctl install --set profile=empty --set components.ingressGateways[0].name=istio-ingressgateway --set components.ingressGateways[0].enabled=true --revision=default -y --set values.global.istioNamespace=istio-system
Istio ingress gateway installation completed
  STEP: checking that istio ingress gateway is available @ 10/10/25 15:48:04.079
  running: kubectl wait --for=condition=Ready pods -l app=istio-ingressgateway -n istio-system --timeout=300s
[BeforeSuite] PASSED [12.996 seconds]
...
[AfterSuite] 
/Users/astonebe/Development/Code/GitHub/kubeflow-notebooks/workspaces/controller/test/e2e/e2e_suite_test.go:126
...
  STEP: uninstalling Istio ingress gateway @ 10/10/25 15:48:56.258
  Uninstalling Istio ingress gateway...
  running: /Users/astonebe/Development/Code/GitHub/kubeflow-notebooks/workspaces/controller/bin/istioctl uninstall --set profile=empty --set components.ingressGateways[0].name=istio-ingressgateway --set components.ingressGateways[0].enabled=true --revision=default -y --set values.global.istioNamespace=istio-system
...
Ran 1 of 1 Specs in 65.414 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 0 Skipped
--- PASS: TestE2E (65.41s)
PASS
ok  	github.com/kubeflow/notebooks/workspaces/controller/test/e2e	65.668s

`kind` cluster with `istiod` + `ingressgateway`

➜ controller/ git:(chore/istio-in-e2e) $ cat <<EOF | KIND_EXPERIMENTAL_PROVIDER=podman kind create cluster --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  image: kindest/node:v1.32.0@sha256:c48c62eac5da28cdadcf560d1d8616cfa6783b58f0d94cf63ad1bf49600cb027
  kubeadmConfigPatches:
  - |
    kind: ClusterConfiguration
    apiServer:
      extraArgs:
        "service-account-issuer": "https://kubernetes.default.svc"
        "service-account-signing-key-file": "/etc/kubernetes/pki/sa.key"
EOF

➜ controller/ git:(chore/istio-in-e2e) $ istioctl install --set profile=default --revision=default -y
        |\          
        | \         
        |  \        
        |   \       
      /||    \      
     / ||     \     
    /  ||      \    
   /   ||       \   
  /    ||        \  
 /     ||         \ 
/______||__________\
____________________
  \__       _____/  
     \_____/        

✔ Istio core installed ⛵️                                                                                                                                                                      
✔ Istiod installed 🧠                                                                                                                                                                          
✔ Ingress gateways installed 🛬                                                                                                                                                                
✔ Installation complete

➜ controller/ git:(chore/istio-in-e2e) $ make test-e2e
...
[BeforeSuite] 
/Users/astonebe/Development/Code/GitHub/kubeflow-notebooks/workspaces/controller/test/e2e/e2e_suite_test.go:60
...
  STEP: checking if istio is installed already @ 10/10/25 15:59:09.575
  running: kubectl get crd gateways.networking.istio.io
  running: kubectl get deploy -A -l app=istiod -o jsonpath={.items[*].metadata.namespace}
  running: kubectl get deployment istiod -n istio-system
  running: /Users/astonebe/Development/Code/GitHub/kubeflow-notebooks/workspaces/controller/bin/istioctl version --short --remote --istioNamespace=istio-system
  WARNING: istio is already installed in namespace istio-system. Skipping installation...
  STEP: checking that istio is available @ 10/10/25 15:59:09.905
  running: kubectl wait --for=condition=Ready pods -l app=istiod -n istio-system --timeout=300s
  running: kubectl get service istiod -n istio-system -o jsonpath={.metadata.name}
  running: kubectl get mutatingwebhookconfiguration istio-sidecar-injector -o jsonpath={.metadata.name}
  running: kubectl get endpoints istiod -n istio-system -o jsonpath={.subsets[*].addresses[*].ip}
  STEP: checking if istio ingress gateway is installed already @ 10/10/25 15:59:10.166
  running: kubectl get deployment -n istio-system istio-ingressgateway --ignore-not-found
  WARNING: istio ingress gateway is already installed. Skipping installation...
  STEP: checking that istio ingress gateway is available @ 10/10/25 15:59:10.206
  running: kubectl wait --for=condition=Ready pods -l app=istio-ingressgateway -n istio-system --timeout=300s
[BeforeSuite] PASSED [16.415 seconds]
...
------------------------------
[AfterSuite] 
/Users/astonebe/Development/Code/GitHub/kubeflow-notebooks/workspaces/controller/test/e2e/e2e_suite_test.go:126
  STEP: uninstalling CertManager @ 10/10/25 16:00:22.369
  Uninstalling CertManager...
  running: kubectl delete -f https://github.com/jetstack/cert-manager/releases/download/v1.12.13/cert-manager.yaml
[AfterSuite] PASSED [12.437 seconds]
------------------------------

Ran 1 of 1 Specs in 100.873 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 0 Skipped
--- PASS: TestE2E (100.87s)
PASS
ok  	github.com/kubeflow/notebooks/workspaces/controller/test/e2e	101.141s

@yashpal2104

Refactors work submitted by @yashpal2104 to be more aligned with the structure of cert-manager and prometheus. - Note I also rebased the branch off the latest upstream/notebooks-v2 branch. Includes a commented out block of code that adds istio-injection label on the namespaces created by e2e. - VirtualService objects must exist for this code to be uncommented - otherwise the Workspace connect endpoint is unreachable. Moved handling of `istioctl` dependency to the `Makefile` as that code was unwiedly/intimidating to manage in a `.go` file. Additionally, refactored utils module to be decomposed into multiple files as it was becoming unwieldy to manage everything in utils.go. Signed-off-by: Andy Stoneberg <[email protected]>

github-project-automation bot added this to Kubeflow Notebooks Oct 10, 2025

github-project-automation bot moved this to Needs Triage in Kubeflow Notebooks Oct 10, 2025

google-oss-prow bot added size/XL area/controller area - related to controller components labels Oct 10, 2025

google-oss-prow bot requested a review from kimwnasptd October 10, 2025 20:19

google-oss-prow bot requested a review from thesuperzapper October 10, 2025 20:19

google-oss-prow bot added the area/v2 area - version - kubeflow notebooks v2 label Oct 10, 2025

andyatmiami mentioned this pull request Oct 10, 2025

test: Istio e2e test suite #570

Closed

google-oss-prow bot added the ok-to-test label Oct 10, 2025

andyatmiami force-pushed the chore/istio-in-e2e branch from 5e2b571 to 103e4a6 Compare October 10, 2025 20:30

andyatmiami force-pushed the chore/istio-in-e2e branch from 103e4a6 to 8065402 Compare October 10, 2025 20:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

ci: add istio to e2e #646

ci: add istio to e2e #646

andyatmiami commented Oct 10, 2025 •

edited

Loading

Uh oh!

google-oss-prow bot commented Oct 10, 2025

Uh oh!

andyatmiami commented Oct 10, 2025

Uh oh!

andyatmiami commented Oct 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ci: add istio to e2e #646

Are you sure you want to change the base?

ci: add istio to e2e #646

Conversation

andyatmiami commented Oct 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

google-oss-prow bot commented Oct 10, 2025

Uh oh!

andyatmiami commented Oct 10, 2025

Uh oh!

andyatmiami commented Oct 10, 2025

Verification

"Empty" kind cluster

kind cluster with istiod (no ingressgateway)

kind cluster with istiod + ingressgateway

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

andyatmiami commented Oct 10, 2025 •

edited

Loading

"Empty" `kind` cluster

`kind` cluster with `istiod` (no `ingressgateway`)

`kind` cluster with `istiod` + `ingressgateway`