Skip to content

aws only public subnets coverage #62226

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Mar 5, 2025

Conversation

jianlinliu
Copy link
Contributor

@jianlinliu jianlinliu commented Feb 28, 2025

Installer supports OPENSHIFT_INSTALL_AWS_PUBLIC_ONLY, once it enabled, that means the vpc would be NAT-less, installer would only depends on public subnets. The feature can help save budget for nat gateway cost.

@openshift-ci openshift-ci bot requested review from asood-rh and gpei February 28, 2025 11:54
@openshift-ci-robot
Copy link
Contributor

@jianlinliu, pj-rehearse: unable to determine affected jobs. This could be due to a branch that needs to be rebased. ERROR:

could not determine changed registry steps: could not load step registry: file /var/tmp/gitrepo656459136/ci-operator/step-registry/cucushift/installer/rehearse/aws/ipi/byo-subnets/deprovision/cucushift-installer-rehearse-aws-ipi-private-deprovision-chain.yaml has incorrect prefix. Prefix should be cucushift-installer-rehearse-aws-ipi-byo-subnets-deprovision
Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@jianlinliu jianlinliu force-pushed the publicsubnets branch 3 times, most recently from b237ec8 to ba8158a Compare February 28, 2025 12:40
@jianlinliu
Copy link
Contributor Author

/pj-rehearse periodic-ci-openshift-verification-tests-master-installation-nightly-4.18-aws-ipi-byo-subnets-only-public-arm-f14

@openshift-ci-robot
Copy link
Contributor

@jianlinliu: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@jianlinliu
Copy link
Contributor Author

/pj-rehearse periodic-ci-openshift-verification-tests-master-installation-nightly-4.18-aws-ipi-byo-subnets-only-public-arm-f14

@openshift-ci-robot
Copy link
Contributor

@jianlinliu: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@jianlinliu
Copy link
Contributor Author

/pj-rehearse periodic-ci-openshift-verification-tests-master-installation-nightly-4.18-aws-ipi-byo-subnets-only-public-arm-f14

@openshift-ci-robot
Copy link
Contributor

@jianlinliu: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@jianlinliu
Copy link
Contributor Author

/pj-rehearse periodic-ci-openshift-verification-tests-master-installation-nightly-4.18-aws-ipi-byo-subnets-only-public-arm-f14

@openshift-ci-robot
Copy link
Contributor

@jianlinliu: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@jianlinliu
Copy link
Contributor Author

/pj-rehearse periodic-ci-openshift-verification-tests-master-installation-nightly-4.18-aws-ipi-byo-subnets-only-public-arm-f14

@openshift-ci-robot
Copy link
Contributor

@jianlinliu: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@jianlinliu
Copy link
Contributor Author

/pj-rehearse periodic-ci-openshift-verification-tests-master-installation-nightly-4.19-aws-ipi-shared-phz-sts-f14

@openshift-ci-robot
Copy link
Contributor

@jianlinliu: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@jianlinliu
Copy link
Contributor Author

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.19-amd64-nightly-aws-ipi-workers-marketplace-public-subnets-mini-perm-f7

@openshift-ci-robot
Copy link
Contributor

@jianlinliu: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@jianlinliu
Copy link
Contributor Author

rehearse testing jobs get passed.

When OPENSHIFT_INSTALL_AWS_PUBLIC_ONLY enabled, job is here.
When OPENSHIFT_INSTALL_AWS_PUBLIC_ONLY disabled (by default), job is here.

cc @yunjiang29 to review.

@jianlinliu jianlinliu force-pushed the publicsubnets branch 2 times, most recently from 2ec5a2f to dcd3744 Compare March 3, 2025 08:57
@jianlinliu
Copy link
Contributor Author

/pj-rehearse periodic-ci-openshift-verification-tests-master-installation-nightly-4.18-aws-ipi-byo-subnets-only-public-arm-f14

@openshift-ci-robot
Copy link
Contributor

@jianlinliu: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-ci-robot
Copy link
Contributor

@jianlinliu: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-ci-robot
Copy link
Contributor

@jianlinliu: job(s): periodic-ci-openshift-verification-tests-master-installation-nightly-4.18-aws-ipi-byo-subnets-only-public-arm-f14 either don't exist or were not found to be affected, and cannot be rehearsed

@jianlinliu
Copy link
Contributor Author

/pj-rehearse periodic-ci-openshift-verification-tests-master-installation-nightly-4.18-aws-ipi-byo-subnets-only-public-mini-perm-arm-f14

@openshift-ci-robot
Copy link
Contributor

@jianlinliu: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

Copy link
Contributor

openshift-ci bot commented Mar 4, 2025

@jianlinliu: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/rehearse/periodic-ci-openshift-verification-tests-master-installation-nightly-4.18-aws-ipi-byo-subnets-only-public-arm-f14 b994771 link unknown /pj-rehearse periodic-ci-openshift-verification-tests-master-installation-nightly-4.18-aws-ipi-byo-subnets-only-public-arm-f14
ci/rehearse/periodic-ci-openshift-openshift-tests-private-release-4.19-amd64-nightly-aws-ipi-workers-marketplace-public-subnets-mini-perm-f7 ff69bee link unknown /pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.19-amd64-nightly-aws-ipi-workers-marketplace-public-subnets-mini-perm-f7

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@yunjiang29
Copy link
Contributor

/assign @yunjiang29

Copy link
Contributor

@yunjiang29 yunjiang29 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jianlinliu some comments below

private_subnet_ids=$(yq-go r -j ${SHARED_DIR}/private_subnet_ids | jq -r '[ . | join(" ") ] | @csv' | sed "s/\"//g")

if [[ "${OPENSHIFT_INSTALL_AWS_PUBLIC_ONLY}" == "true" ]]; then
private_subnet_ids=$(yq-go r -j ${SHARED_DIR}/public_subnet_ids | jq -r '[ . | join(" ") ] | @csv' | sed "s/\"//g")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To avoid confusion, I'd suggest replacing private_subnet_ids with subnet_ids

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@@ -0,0 +1,9 @@
workflow:
as: cucushift-installer-rehearse-aws-ipi-workers-marketplace-byo-subnets
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this workflow used in the jobs?

Copy link
Contributor Author

@jianlinliu jianlinliu Mar 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was planing to use this install workflow to cover marketplace ci jobs, in the last minutes, I decided to do that later, just developed the new install workflow here for later reference.

- name: OPENSHIFT_INSTALL_AWS_PUBLIC_ONLY
default: ""
documentation: |-
Whether to use only public subnets for AWS. Implies no NAT Gateways.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggest to add a note that indicates OPENSHIFT_INSTALL_AWS_PUBLIC_ONLY will be used by create permissions-policy validation.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@yunjiang29
Copy link
Contributor

Additional node, once OPENSHIFT_INSTALL_AWS_PUBLIC_ONLY enabled:

  1. The stack output PrivateRouteTableIds and PrivateSubnetIds is empty.
  2. The file ${SHARED_DIR}/private_subnet_ids is empty: []

This works well on current PR/jobs, but I'm not sure if it will cause other potential issues in other job configs with OPENSHIFT_INSTALL_AWS_PUBLIC_ONLY enabled, we need to pay more attentions while enable OPENSHIFT_INSTALL_AWS_PUBLIC_ONLY on other jobs.

@jianlinliu
Copy link
Contributor Author

jianlinliu commented Mar 4, 2025

This works well on current PR/jobs, but I'm not sure if it will cause other potential issues in other job configs with OPENSHIFT_INSTALL_AWS_PUBLIC_ONLY enabled, we need to pay more attentions while enable OPENSHIFT_INSTALL_AWS_PUBLIC_ONLY on other jobs.

So far, there is no many other jobs enabled OPENSHIFT_INSTALL_AWS_PUBLIC_ONLY beside openshift-e2e-aws-publicsubnets. The install workflow is different from the ones in this PR, no much overlap with the ones in this PR. So I guess it should be low risky.

@jianlinliu
Copy link
Contributor Author

/pj-rehearse periodic-ci-openshift-verification-tests-master-installation-nightly-4.18-aws-ipi-byo-subnets-only-public-mini-perm-arm-f14

@openshift-ci-robot
Copy link
Contributor

@jianlinliu: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@jianlinliu
Copy link
Contributor Author

/pj-rehearse periodic-ci-openshift-verification-tests-master-installation-nightly-4.19-aws-ipi-shared-phz-sts-f14

@openshift-ci-robot
Copy link
Contributor

@jianlinliu: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@jianlinliu
Copy link
Contributor Author

rehearse testing jobs get passed.

When OPENSHIFT_INSTALL_AWS_PUBLIC_ONLY enabled, job is here.
When OPENSHIFT_INSTALL_AWS_PUBLIC_ONLY disabled (by default), job is here.

@openshift-ci-robot
Copy link
Contributor

[REHEARSALNOTIFIER]
@jianlinliu: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name Repo Type Reason
pull-ci-openshift-local-storage-operator-release-4.9-e2e-operator openshift/local-storage-operator presubmit Registry content changed
pull-ci-openshift-local-storage-operator-release-4.8-e2e-operator openshift/local-storage-operator presubmit Registry content changed
pull-ci-openshift-local-storage-operator-release-4.7-e2e-operator openshift/local-storage-operator presubmit Registry content changed
pull-ci-openshift-local-storage-operator-release-4.6-e2e-operator openshift/local-storage-operator presubmit Registry content changed
pull-ci-openshift-local-storage-operator-release-4.5-e2e-operator openshift/local-storage-operator presubmit Registry content changed
pull-ci-openshift-local-storage-operator-release-4.4-e2e-operator openshift/local-storage-operator presubmit Registry content changed
pull-ci-openshift-local-storage-operator-release-4.3-e2e-operator openshift/local-storage-operator presubmit Registry content changed
pull-ci-openshift-local-storage-operator-release-4.2-e2e-operator openshift/local-storage-operator presubmit Registry content changed
pull-ci-openshift-local-storage-operator-main-e2e-operator openshift/local-storage-operator presubmit Registry content changed
pull-ci-openshift-local-storage-operator-main-e2e-operator-extended openshift/local-storage-operator presubmit Registry content changed
pull-ci-openshift-local-storage-operator-release-4.20-e2e-operator openshift/local-storage-operator presubmit Registry content changed
pull-ci-openshift-local-storage-operator-release-4.20-e2e-operator-extended openshift/local-storage-operator presubmit Registry content changed
pull-ci-openshift-local-storage-operator-release-4.19-e2e-operator openshift/local-storage-operator presubmit Registry content changed
pull-ci-openshift-local-storage-operator-release-4.19-e2e-operator-extended openshift/local-storage-operator presubmit Registry content changed
pull-ci-openshift-local-storage-operator-release-4.18-e2e-operator openshift/local-storage-operator presubmit Registry content changed
pull-ci-openshift-local-storage-operator-release-4.18-e2e-operator-extended openshift/local-storage-operator presubmit Registry content changed
pull-ci-openshift-local-storage-operator-release-4.17-e2e-operator openshift/local-storage-operator presubmit Registry content changed
pull-ci-openshift-local-storage-operator-release-4.17-e2e-operator-extended openshift/local-storage-operator presubmit Registry content changed
pull-ci-openshift-local-storage-operator-release-4.16-e2e-operator openshift/local-storage-operator presubmit Registry content changed
pull-ci-openshift-local-storage-operator-release-4.16-e2e-operator-extended openshift/local-storage-operator presubmit Registry content changed
pull-ci-openshift-local-storage-operator-release-4.15-e2e-operator openshift/local-storage-operator presubmit Registry content changed
pull-ci-openshift-local-storage-operator-release-4.15-e2e-operator-extended openshift/local-storage-operator presubmit Registry content changed
pull-ci-openshift-local-storage-operator-release-4.14-e2e-operator openshift/local-storage-operator presubmit Registry content changed
pull-ci-openshift-local-storage-operator-release-4.14-e2e-operator-extended openshift/local-storage-operator presubmit Registry content changed
pull-ci-openshift-local-storage-operator-release-4.13-e2e-operator openshift/local-storage-operator presubmit Registry content changed

A total of 9664 jobs have been affected by this change. The above listing is non-exhaustive and limited to 25 jobs.

A full list of affected jobs can be found here

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@yunjiang29
Copy link
Contributor

/lgtm
/pj-rehearse ack

@openshift-ci-robot
Copy link
Contributor

@yunjiang29: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-ci-robot openshift-ci-robot added the rehearsals-ack Signifies that rehearsal jobs have been acknowledged label Mar 5, 2025
@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Mar 5, 2025
@yunjiang29
Copy link
Contributor

@liangxia please review and approve, thanks

@liangxia
Copy link
Member

liangxia commented Mar 5, 2025

Installer supports OPENSHIFT_INSTALL_AWS_PUBLIC_ONLY, once it enabled, that means the vpc would be NAT-less, installer would only depends on public subnets. The feature can help save budget for nat gateway cost.

@jianlinliu What's the first version that we start to support this ? My team has a card for this and we'd like to apply it to our infrastructures, too.

@liangxia
Copy link
Member

liangxia commented Mar 5, 2025

/approve

Copy link
Contributor

openshift-ci bot commented Mar 5, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jianlinliu, liangxia, yunjiang29

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 5, 2025
@openshift-merge-bot openshift-merge-bot bot merged commit 103788e into openshift:master Mar 5, 2025
17 checks passed
@jianlinliu
Copy link
Contributor Author

Installer supports OPENSHIFT_INSTALL_AWS_PUBLIC_ONLY, once it enabled, that means the vpc would be NAT-less, installer would only depends on public subnets. The feature can help save budget for nat gateway cost.

@jianlinliu What's the first version that we start to support this ? My team has a card for this and we'd like to apply it to our infrastructures, too.

From the existing ci jobs to cover the option, sounds it starts from 4.15. After talked with bear, sounds it is not enough to meet your team's requirement, so there is a followup user story.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged. rehearsals-ack Signifies that rehearsal jobs have been acknowledged
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants