OTA-209: CVO: Add DevPreviewNoUpgrade job

[DavidHurta]

The Cluster Version Operator repository is starting to develop
DevPreview functionalities.

We do have the e2e-agnostic-usc-devpreview job, which configures
a running Default cluster to a configured DevPreview cluster
using [1] script. However, other being developed features may even
impact the cluster installation; thus, add a job to verify the
changes on DevPreview installed clusters. Run CVO integration
tests to check against respective regressions in a reliable way
instead of running the default step.

The job will help us to verify small changes on DevPreview PRs quickly.
The job will help us to detect respective regressions sooner.

[DavidHurta]

/pj-rehearse pull-ci-openshift-cluster-version-operator-main-e2e-aws-ovn-devpreview

[openshift-ci-robot]

@DavidHurta: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

[DavidHurta]

/pj-rehearse pull-ci-openshift-cluster-version-operator-main-e2e-agnostic-devpreview

[openshift-ci-robot]

@DavidHurta: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

[openshift-ci-robot]

@DavidHurta: This pull request references OTA-209 which is a valid jira issue.

In response to this:

The Cluster Version Operator repository is starting to develop DevPreview functionalities. Introduce a DevPreview job to test out these functionalities (such as openshift/cluster-version-operator#1163). We do have the e2e-agnostic-usc-devpreview job, which configures a running Default cluster to a configured DevPreview cluster using [1] script. However, other being developed features may even impact the cluster installation; thus, add a job to verify the changes on DevPreview installed clusters without any further configuration.

The job will also help us to verify small changes on such PRs quickly. Otherwise, a cluster created by a Cluster Bot is needed.

Make the job optional, as we do not want to block PRs on this job.
Make the job triggerable by a command as we do not have to run the change
every time (at least as of now).

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[DavidHurta]

/pj-rehearse pull-ci-openshift-cluster-version-operator-main-e2e-agnostic-devpreview

Let's learn how (very roughly) stable the job would be.

[openshift-ci-robot]

@DavidHurta: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

[DavidHurta]

Failure not related to the DevPreview feature set.

https://issues.redhat.com/browse/OCPBUGS-52477

[DavidHurta]

/pj-rehearse pull-ci-openshift-cluster-version-operator-main-e2e-agnostic-devpreview

[openshift-ci-robot]

@DavidHurta: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

[hongkailiu]

Failure not related to the DevPreview feature set.

https://issues.redhat.com/browse/OCPBUGS-52477

Nice catch.
Let us assume it will be fixed and the rehearsal turns green.
If we do not always run it, it would still be hard to tell the cause if a failure occurs.
I feel we should always run it (at least for the relevant resources or *.go changes if we can filter them out).

About the e2e suite in test step, I feels the org should have a periodic somewhere to ensure dev-preview works, instead of running it in our repo.
But we can use it as a placeholder for now while we are figuring out what we should verify (i guess features from us) on a dev-preview cluster.
Alternatively if "the CVO is correctly applying DevPreview manifests." is your current focus, you can write the bash to check if the expected log shows up in the test step.
It can grow if you have more things to check in the feature.

[DavidHurta]

/hold

[openshift-ci]

@DavidHurta: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/rehearse/openshift/cluster-version-operator/main/e2e-aws-ovn-devpreview	5bca80e	link	unknown	/pj-rehearse pull-ci-openshift-cluster-version-operator-main-e2e-aws-ovn-devpreview
ci/rehearse/openshift/cluster-version-operator/main/e2e-agnostic-devpreview	8d356eb	link	unknown	/pj-rehearse pull-ci-openshift-cluster-version-operator-main-e2e-agnostic-devpreview

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[DavidHurta]

/pj-rehearse pull-ci-openshift-cluster-version-operator-main-e2e-agnostic-operator-devpreview

[openshift-ci-robot]

@DavidHurta: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

[DavidHurta]

I agree with concerns regarding the noise and unreliability of the e2e test step in DevPreview clusters to provide a valuable signal. The intent was to have some tests rather than none on a manually triggered job to provide additional insights as a developer may assess the results. However, I now see that some of the tests that I was interested in (for example, checking for panics or cluster installation success) seem to not even be a part of the step, and the job seems unstable. I appreciate the comments and feedback.

To provide a job that installs a DevPreview cluster (my main goal), tests some relevant logic, and provides a reliable signal, I am proposing to run the CVO integration tests as the e2e-test. The job passes, tests CVO integration, and runs a DevPreview cluster. The goal is not to check for any possible regression but rather to run a DevPreview cluster on PRs and check for some regressions in a reliable way.

Regarding:

If we do not always run it, it would still be hard to tell the cause if a failure occurs.
I feel we should always run it (at least for the relevant resources or *.go changes if we can filter them out).

That is true. I guess we can run such a job always till the feature is promoted to the TechPreview. Should only be a couple of weeks. However, I am fine with both options as we can evaluate the state of the CVOConfiguration DevPreview feature gate upon promotion into TechPreview.

[hongkailiu]

to run the CVO integration tests as the e2e-test

Nice. I did not notice such a thing exists.
I still feel always-run is better but the excuse of "Should only be a couple of weeks" is convincing enough to me. 😄

/lgtm

[hongkailiu]

/test ci-operator-config

[DavidHurta]

Thanks for having a look!
I can set it to always, no problem. We can then remove the job (or set it to always=false) after the TechPreview promotion of the currently relevant feature gate.
The PR is on hold; I was not sure of the feedback. Let me rebase the PR quickly.

[DavidHurta]

/unhold

[openshift-ci-robot]

[REHEARSALNOTIFIER]
@DavidHurta: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name	Repo	Type	Reason
pull-ci-openshift-cluster-version-operator-main-e2e-agnostic-operator-devpreview	openshift/cluster-version-operator	presubmit	Presubmit changed

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

[hongkailiu]

/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: DavidHurta, hongkailiu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• ci-operator/config/openshift/cluster-version-operator/OWNERS [DavidHurta]
• ci-operator/jobs/openshift/cluster-version-operator/OWNERS [DavidHurta]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[DavidHurta]

/pj-rehearse ack

The newest change has solely removed the always_run field and updated the messages of the commits. Not needed to rerun the job.

[openshift-ci-robot]

@DavidHurta: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.