IntegrationSource CRD #8238

matzew · 2024-10-10T15:16:09Z

Related to #8185

Proposed Changes

Adding new IntegrationSource CRD and tests. Each commit has details on the implementation step or the needs (e.g. as in addressing comments too and enhancing). As alwasys, there is lot of boilerplate code when on-bording new component

Pre-review Checklist

At least 80% unit test coverage
E2E tests for any new behavior
Docs PR for any user-facing impact
Spec PR for any new API feature
Conformance test for any change to the spec

Release Note

Add IntegrationSource for supporting generic event sources based on Apache Camel Kamelets.

Docs

codecov · 2024-10-10T15:23:18Z

Codecov Report

Attention: Patch coverage is 63.10433% with 145 lines in your changes missing coverage. Please review.

Project coverage is 64.58%. Comparing base (c859efd) to head (05417aa).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
...ler/integrationsource/resources/containersource.go	53.20%	68 Missing and 5 partials ⚠️
pkg/reconciler/integrationsource/controller.go	56.14%	22 Missing and 3 partials ⚠️
...kg/apis/sources/v1alpha1/integration_validation.go	77.77%	9 Missing and 3 partials ⚠️
pkg/apis/sources/v1alpha1/register.go	0.00%	11 Missing ⚠️
.../reconciler/integrationsource/integrationsource.go	71.05%	8 Missing and 3 partials ⚠️
pkg/apis/sources/v1alpha1/integration_lifecycle.go	76.00%	6 Missing ⚠️
pkg/apis/sources/v1alpha1/integration_defaults.go	0.00%	3 Missing ⚠️
cmd/controller/main.go	0.00%	2 Missing ⚠️
pkg/apis/sources/v1alpha1/integration_types.go	77.77%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8238      +/-   ##
==========================================
- Coverage   64.61%   64.58%   -0.03%     
==========================================
  Files         362      373      +11     
  Lines       22226    22619     +393     
==========================================
+ Hits        14361    14609     +248     
- Misses       7113     7244     +131     
- Partials      752      766      +14

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

matzew · 2024-10-10T15:37:02Z

pkg/reconciler/integrationsource/resources/containersource.go

+
+func selectImage(source *v1alpha1.IntegrationSource) string {
+	if source.Spec.Timer != nil {
+		return "quay.io/openshift-knative/kn-connector-timer-source:1.0-SNAPSHOT"


@dsimansk for this to get rid of, let's see if we can get the integrations repo setup w/ image push etc.

and yes, this code will be changing too. but is first draft. I am currently more focused on having things work

pkg/apis/sources/v1alpha1/integration_lifecycle.go

matzew · 2024-10-28T14:26:57Z

pkg/reconciler/integrationsource/resources/containersource.go

+
+func selectImage(source *v1alpha1.IntegrationSource) string {
+	if source.Spec.Timer != nil {
+		return "gcr.io/knative-nightly/timer-source:latest"


In a soon/future PR image will be read from config map. But for now this works

matzew · 2024-10-28T14:43:51Z

/test reconciler-tests

pierDipi

Overall it looks good as a start

pkg/apis/sources/v1alpha1/integration_types.go

pkg/reconciler/integrationsource/controller.go

pkg/reconciler/integrationsource/integrationsource.go

pkg/reconciler/integrationsource/resources/containersource.go

pkg/reconciler/integrationsource/integrationsource.go

test/rekt/integrationsource_test.go

…lly the container source for the handling of the kamelet workloads. the logic for applying the env-vars is a bit verbose, but can be improved. Tests/lifeccyle and some other improvements are still missing. but this works. Signed-off-by: Matthias Wessendorf <[email protected]>

… to extend... Signed-off-by: Matthias Wessendorf <[email protected]>

Signed-off-by: Matthias Wessendorf <[email protected]>

Co-authored-by: Pierangelo Di Pilato <[email protected]>

Signed-off-by: Matthias Wessendorf <[email protected]>

pkg/reconciler/integrationsource/integrationsource.go

Signed-off-by: Matthias Wessendorf <[email protected]>

matzew · 2024-10-31T14:12:22Z

@pierDipi thanks for the review, I've addressed your comments

pierDipi

/lgtm
/approve

knative-prow · 2024-10-31T14:14:52Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: matzew, pierDipi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [matzew,pierDipi]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

matzew · 2024-10-31T14:15:03Z

/unhold

* Add initial raw API types and controller/reconciler. Levering internally the container source for the handling of the kamelet workloads. the logic for applying the env-vars is a bit verbose, but can be improved. Tests/lifeccyle and some other improvements are still missing. but this works. Signed-off-by: Matthias Wessendorf <[email protected]> * adding reflection to make the mapping less verbose, and more flexible to extend... Signed-off-by: Matthias Wessendorf <[email protected]> * Add DDB Streams Signed-off-by: Matthias Wessendorf <[email protected]> * polish and add minimal tests Signed-off-by: Matthias Wessendorf <[email protected]> * Update pkg/apis/sources/v1alpha1/integration_lifecycle.go Co-authored-by: Pierangelo Di Pilato <[email protected]> * Update pkg/apis/sources/v1alpha1/integration_lifecycle.go Co-authored-by: Pierangelo Di Pilato <[email protected]> * Fix compiler warnings Signed-off-by: Matthias Wessendorf <[email protected]> * use other image coordinates Signed-off-by: Matthias Wessendorf <[email protected]> * first test for source Signed-off-by: Matthias Wessendorf <[email protected]> * Adding initial rekt test Signed-off-by: Matthias Wessendorf <[email protected]> * Adding SSL support and test for IntegrationSource Signed-off-by: Matthias Wessendorf <[email protected]> * Formatting fixes Signed-off-by: Matthias Wessendorf <[email protected]> * Disable OIDC tests for now Signed-off-by: Matthias Wessendorf <[email protected]> * use knative nightly images Signed-off-by: Matthias Wessendorf <[email protected]> * Fixing linters Signed-off-by: Matthias Wessendorf <[email protected]> * addressing comments, Part1 Signed-off-by: Matthias Wessendorf <[email protected]> --------- Signed-off-by: Matthias Wessendorf <[email protected]> Co-authored-by: Pierangelo Di Pilato <[email protected]>

* IntegrationSource CRD (knative#8238) * Add initial raw API types and controller/reconciler. Levering internally the container source for the handling of the kamelet workloads. the logic for applying the env-vars is a bit verbose, but can be improved. Tests/lifeccyle and some other improvements are still missing. but this works. Signed-off-by: Matthias Wessendorf <[email protected]> * adding reflection to make the mapping less verbose, and more flexible to extend... Signed-off-by: Matthias Wessendorf <[email protected]> * Add DDB Streams Signed-off-by: Matthias Wessendorf <[email protected]> * polish and add minimal tests Signed-off-by: Matthias Wessendorf <[email protected]> * Update pkg/apis/sources/v1alpha1/integration_lifecycle.go Co-authored-by: Pierangelo Di Pilato <[email protected]> * Update pkg/apis/sources/v1alpha1/integration_lifecycle.go Co-authored-by: Pierangelo Di Pilato <[email protected]> * Fix compiler warnings Signed-off-by: Matthias Wessendorf <[email protected]> * use other image coordinates Signed-off-by: Matthias Wessendorf <[email protected]> * first test for source Signed-off-by: Matthias Wessendorf <[email protected]> * Adding initial rekt test Signed-off-by: Matthias Wessendorf <[email protected]> * Adding SSL support and test for IntegrationSource Signed-off-by: Matthias Wessendorf <[email protected]> * Formatting fixes Signed-off-by: Matthias Wessendorf <[email protected]> * Disable OIDC tests for now Signed-off-by: Matthias Wessendorf <[email protected]> * use knative nightly images Signed-off-by: Matthias Wessendorf <[email protected]> * Fixing linters Signed-off-by: Matthias Wessendorf <[email protected]> * addressing comments, Part1 Signed-off-by: Matthias Wessendorf <[email protected]> --------- Signed-off-by: Matthias Wessendorf <[email protected]> Co-authored-by: Pierangelo Di Pilato <[email protected]> * Add missing copyright boilerplate (knative#8305) :lipstick: add missing copyright boilerplate Signed-off-by: Matthias Wessendorf <[email protected]> * [chore] Use a generic access/secret key in the secret for AWS connections (knative#8312) :key: Use a generic access/secret key in the secret for AWS connections Signed-off-by: Matthias Wessendorf <[email protected]> * [chore] Do not generate SSL env vars for each struct entry, just once per containersource/image (knative#8315) :lipstick: Do not generate SSL env vars for each struct entry, just once per containersource/image Signed-off-by: Matthias Wessendorf <[email protected]> * Update Integration API and use custom tags on structs for better readability (knative#8321) Update API and use custom tags on structs Signed-off-by: Matthias Wessendorf <[email protected]> * Integration Api moved to versioned common (knative#8325) Moving API to a versioned common/integration folder Signed-off-by: Matthias Wessendorf <[email protected]> * Introducing common integration helper and change package name (knative#8327) * create new integration/source folder and factor common code into helper.go Signed-off-by: Matthias Wessendorf <[email protected]> * 💄 improve formatting... Signed-off-by: Matthias Wessendorf <[email protected]> * fixing import Signed-off-by: Matthias Wessendorf <[email protected]> --------- Signed-off-by: Matthias Wessendorf <[email protected]> * chore: correct comments (knative#8336) :lipstick: correct comments ... Signed-off-by: Matthias Wessendorf <[email protected]> * Add IntegrationSink CRD (knative#8304) * create new integration/source folder and factor common code into helper.go Signed-off-by: Matthias Wessendorf <[email protected]> * 💄 improve formatting... Signed-off-by: Matthias Wessendorf <[email protected]> * fixing import Signed-off-by: Matthias Wessendorf <[email protected]> * First Sink Signed-off-by: Matthias Wessendorf <[email protected]> * Updates on sink Signed-off-by: Matthias Wessendorf <[email protected]> * starting common types for source Signed-off-by: Matthias Wessendorf <[email protected]> * starting common types for sink Signed-off-by: Matthias Wessendorf <[email protected]> * some code share and refactorings * use other image coordinates Signed-off-by: Matthias Wessendorf <[email protected]> * Add missing api and tests for sink and run codegen Signed-off-by: Matthias Wessendorf <[email protected]> * Fixes Signed-off-by: Matthias Wessendorf <[email protected]> * Adding cert Signed-off-by: Matthias Wessendorf <[email protected]> * Adding test Signed-off-by: Matthias Wessendorf <[email protected]> * Rekt test for sink Signed-off-by: Matthias Wessendorf <[email protected]> * Rekt test for sink Signed-off-by: Matthias Wessendorf <[email protected]> * correct status code Signed-off-by: Matthias Wessendorf <[email protected]> * More... Signed-off-by: Matthias Wessendorf <[email protected]> * re-add service Signed-off-by: Matthias Wessendorf <[email protected]> * delete unversioned common api Signed-off-by: Matthias Wessendorf <[email protected]> * Updates, latest conflicts and formatting Clean ups Adding tests for reconciler Signed-off-by: Matthias Wessendorf <[email protected]> --------- Signed-off-by: Matthias Wessendorf <[email protected]> conflict Signed-off-by: Matthias Wessendorf <[email protected]> * IntegrationSink: rek-test templating support for bool annotations (knative#8342) bool annotations for represented as string, to avoid parsing errors Signed-off-by: Matthias Wessendorf <[email protected]> * Update integrationsink crd (knative#8347) * Align IntegrationSink CRD w/ IntegrationSource CRD Signed-off-by: Matthias Wessendorf <[email protected]> * 💄 Remove old job-sink references Signed-off-by: Matthias Wessendorf <[email protected]> * Docs change Signed-off-by: Matthias Wessendorf <[email protected]> --------- Signed-off-by: Matthias Wessendorf <[email protected]> * Avoid using twice hardcoded strings for the names of the Kube Secret keys, for AWS access/secret key vals (knative#8345) :lipstick: Avoid using twice hardcoded strings for the names of the Kubescret keys, for AWS access/secret key vals Signed-off-by: Matthias Wessendorf <[email protected]> * Add integrationsinks-addressable-resolver cluster role (knative#8356) :passport_control: Add integrationsinks-addressable-resolver cluster role Signed-off-by: Matthias Wessendorf <[email protected]> * Align label generation and apply on container-source as well (knative#8355) Signed-off-by: Matthias Wessendorf <[email protected]> * We changed the event type on the images (knative#8364) :fire: We changed the event type on the images Signed-off-by: Matthias Wessendorf <[email protected]> * Move image to map (knative#8362) Signed-off-by: Matthias Wessendorf <[email protected]> * Adding sns sink to Integration Sink (knative#8365) adding sns sink Signed-off-by: Matthias Wessendorf <[email protected]> * Clean up remove session token (knative#8369) Remove session token and profile credential name Signed-off-by: Matthias Wessendorf <[email protected]> * Chore: field semantic correction (knative#8370) :lipstick: We renamed those fields to , and we reflect that now in validation Signed-off-by: Matthias Wessendorf <[email protected]> * Remove unused filter function (knative#8375) :broom: Remove unused filter function Signed-off-by: Matthias Wessendorf <[email protected]> * add new integration CRDs to storage version migrator tool (knative#8386) Signed-off-by: Matthias Wessendorf <[email protected]> * Going OIDC for Integration Source (knative#8383) Going OIDC for Integration Source: - Generating OIDC specific evn_vars for Knative client of camel - Adding rekt-test for OIDC feature of the source Signed-off-by: Matthias Wessendorf <[email protected]> * Adding reconciler test for OIDC of the Integration Source (knative#8404) Signed-off-by: Matthias Wessendorf <[email protected]> * Not just render oidc env-vars when cfg is on, we need also a present audience to ensure the camel containers work correct (knative#8448) Signed-off-by: Matthias Wessendorf <[email protected]> * run `make generate-release` --------- Signed-off-by: Matthias Wessendorf <[email protected]> Co-authored-by: Pierangelo Di Pilato <[email protected]> Co-authored-by: Christoph Stäbler <[email protected]>

knative-prow bot requested review from Cali0707 and mgencur October 10, 2024 15:16

matzew commented Oct 10, 2024

View reviewed changes

pierDipi reviewed Oct 11, 2024

View reviewed changes

pkg/apis/sources/v1alpha1/integration_lifecycle.go Outdated Show resolved Hide resolved

pierDipi reviewed Oct 11, 2024

View reviewed changes

pkg/apis/sources/v1alpha1/integration_lifecycle.go Outdated Show resolved Hide resolved

This was referenced Oct 11, 2024

WIP: IntegrationSink CRD #8239

Closed

WIP: Integration sink and source: all together #8254

Closed

matzew force-pushed the poc_integration_source branch from 86eca45 to 96df997 Compare October 16, 2024 07:27

matzew force-pushed the poc_integration_source branch 5 times, most recently from dd6df79 to f0bae00 Compare October 28, 2024 13:11

matzew commented Oct 28, 2024

View reviewed changes

pierDipi reviewed Oct 28, 2024

View reviewed changes

matzew and others added 8 commits October 31, 2024 10:10

adding reflection to make the mapping less verbose, and more flexible…

9332005

… to extend... Signed-off-by: Matthias Wessendorf <[email protected]>

Add DDB Streams

44ad47a

Signed-off-by: Matthias Wessendorf <[email protected]>

polish and add minimal tests

f456fac

Signed-off-by: Matthias Wessendorf <[email protected]>

Update pkg/apis/sources/v1alpha1/integration_lifecycle.go

57a5961

Co-authored-by: Pierangelo Di Pilato <[email protected]>

Update pkg/apis/sources/v1alpha1/integration_lifecycle.go

6843be2

Co-authored-by: Pierangelo Di Pilato <[email protected]>

Fix compiler warnings

691605e

Signed-off-by: Matthias Wessendorf <[email protected]>

use other image coordinates

323b6ab

Signed-off-by: Matthias Wessendorf <[email protected]>

matzew added 7 commits October 31, 2024 10:10

first test for source

e59f7bb

Signed-off-by: Matthias Wessendorf <[email protected]>

Adding initial rekt test

e235724

Signed-off-by: Matthias Wessendorf <[email protected]>

Adding SSL support and test for IntegrationSource

52f223e

Signed-off-by: Matthias Wessendorf <[email protected]>

Formatting fixes

16d2432

Signed-off-by: Matthias Wessendorf <[email protected]>

Disable OIDC tests for now

6a51222

Signed-off-by: Matthias Wessendorf <[email protected]>

use knative nightly images

a49d36e

Signed-off-by: Matthias Wessendorf <[email protected]>

Fixing linters

ee8d39d

Signed-off-by: Matthias Wessendorf <[email protected]>

matzew force-pushed the poc_integration_source branch from f0bae00 to 839abc5 Compare October 31, 2024 10:36

matzew changed the title ~~WIP: IntegrationSource CRD~~ IntegrationSource CRD Oct 31, 2024

knative-prow bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 31, 2024

github-advanced-security bot found potential problems Oct 31, 2024

View reviewed changes

pkg/reconciler/integrationsource/integrationsource.go Fixed Show fixed Hide fixed

addressing comments, Part1

05417aa

Signed-off-by: Matthias Wessendorf <[email protected]>

matzew force-pushed the poc_integration_source branch from 839abc5 to 05417aa Compare October 31, 2024 12:00

pierDipi approved these changes Oct 31, 2024

View reviewed changes

knative-prow bot assigned pierDipi Oct 31, 2024

knative-prow bot added the lgtm Indicates that a PR is ready to be merged. label Oct 31, 2024

knative-prow bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 31, 2024

knative-prow bot merged commit 7bca815 into knative:main Oct 31, 2024
33 of 36 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

IntegrationSource CRD #8238

IntegrationSource CRD #8238

matzew commented Oct 10, 2024 •

edited

Loading

codecov bot commented Oct 10, 2024 •

edited

Loading

matzew Oct 10, 2024

matzew Oct 28, 2024

matzew commented Oct 28, 2024

pierDipi left a comment

matzew commented Oct 31, 2024

pierDipi left a comment

knative-prow bot commented Oct 31, 2024

matzew commented Oct 31, 2024

IntegrationSource CRD #8238

IntegrationSource CRD #8238

Conversation

matzew commented Oct 10, 2024 • edited Loading

Proposed Changes

Pre-review Checklist

codecov bot commented Oct 10, 2024 • edited Loading

Codecov Report

matzew Oct 10, 2024

Choose a reason for hiding this comment

matzew Oct 28, 2024

Choose a reason for hiding this comment

matzew commented Oct 28, 2024

pierDipi left a comment

Choose a reason for hiding this comment

matzew commented Oct 31, 2024

pierDipi left a comment

Choose a reason for hiding this comment

knative-prow bot commented Oct 31, 2024

matzew commented Oct 31, 2024

matzew commented Oct 10, 2024 •

edited

Loading

codecov bot commented Oct 10, 2024 •

edited

Loading