Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add IntegrationSink CRD #8304

Merged
merged 20 commits into from
Nov 22, 2024
Merged

Conversation

matzew
Copy link
Member

@matzew matzew commented Nov 5, 2024

Related to #8185

Proposed Changes

  • Adding new IntegrationSink CRD and tests. As alwasys, there is lot of boilerplate code when on-bording new component

Pre-review Checklist

  • At least 80% unit test coverage
  • E2E tests for any new behavior
  • Docs PR for any user-facing impact
  • Spec PR for any new API feature
  • Conformance test for any change to the spec

Release Note

Add IntegrationSink for supporting generic event sinks based on Apache Camel Kamelets.

Docs

@knative-prow knative-prow bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. area/test-and-release Test infrastructure, tests or release size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Nov 5, 2024
@knative-prow knative-prow bot requested review from aslom and evankanderson November 5, 2024 10:10
@knative-prow-robot knative-prow-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 7, 2024
@matzew matzew force-pushed the add_integration_sink_rework branch from be6d2a3 to 44151ad Compare November 13, 2024 15:35
@knative-prow-robot knative-prow-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 13, 2024
@matzew matzew force-pushed the add_integration_sink_rework branch from 44151ad to d378de7 Compare November 13, 2024 15:49
@knative-prow-robot knative-prow-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 19, 2024
@matzew matzew force-pushed the add_integration_sink_rework branch from d378de7 to 1283b65 Compare November 20, 2024 10:59
@knative-prow-robot knative-prow-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 20, 2024
@matzew matzew force-pushed the add_integration_sink_rework branch 2 times, most recently from ba47495 to 376a3e8 Compare November 20, 2024 14:25
Signed-off-by: Matthias Wessendorf <[email protected]>
Signed-off-by: Matthias Wessendorf <[email protected]>
Signed-off-by: Matthias Wessendorf <[email protected]>
Signed-off-by: Matthias Wessendorf <[email protected]>
Signed-off-by: Matthias Wessendorf <[email protected]>
Signed-off-by: Matthias Wessendorf <[email protected]>
Signed-off-by: Matthias Wessendorf <[email protected]>
Signed-off-by: Matthias Wessendorf <[email protected]>
Signed-off-by: Matthias Wessendorf <[email protected]>
Signed-off-by: Matthias Wessendorf <[email protected]>
Signed-off-by: Matthias Wessendorf <[email protected]>
Signed-off-by: Matthias Wessendorf <[email protected]>
Signed-off-by: Matthias Wessendorf <[email protected]>
Signed-off-by: Matthias Wessendorf <[email protected]>
Signed-off-by: Matthias Wessendorf <[email protected]>
Signed-off-by: Matthias Wessendorf <[email protected]>
@matzew matzew force-pushed the add_integration_sink_rework branch from 376a3e8 to b6fd865 Compare November 21, 2024 14:43
Copy link

codecov bot commented Nov 21, 2024

Codecov Report

Attention: Patch coverage is 49.78992% with 239 lines in your changes missing coverage. Please review.

Project coverage is 64.23%. Comparing base (5ad7dab) to head (9d7e051).
Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
pkg/reconciler/integration/sink/integrationsink.go 38.46% 79 Missing and 17 partials ⚠️
pkg/reconciler/integration/sink/controller.go 0.00% 58 Missing ⚠️
.../apis/sinks/v1alpha1/integration_sink_lifecycle.go 9.30% 39 Missing ⚠️
...iler/integration/sink/resources/container_image.go 72.80% 31 Missing ⚠️
...apis/sinks/v1alpha1/integration_sink_validation.go 79.54% 7 Missing and 2 partials ⚠️
...g/apis/sinks/v1alpha1/integration_sink_defaults.go 0.00% 3 Missing ⚠️
pkg/apis/sinks/v1alpha1/integration_sink_types.go 66.66% 2 Missing ⚠️
cmd/controller/main.go 0.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #8304      +/-   ##
==========================================
- Coverage   64.53%   64.23%   -0.31%     
==========================================
  Files         380      390      +10     
  Lines       22772    23248     +476     
==========================================
+ Hits        14697    14934     +237     
- Misses       7300     7520     +220     
- Partials      775      794      +19     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.


🚨 Try these New Features:

@matzew matzew force-pushed the add_integration_sink_rework branch 2 times, most recently from 00b2d0b to 0a3f64e Compare November 22, 2024 12:03
@matzew matzew changed the title WIP: Add integration sink Add integration sink Nov 22, 2024
@knative-prow knative-prow bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 22, 2024
@matzew matzew changed the title Add integration sink Add IntegrationSink CRD Nov 22, 2024
Clean ups
Adding tests for reconciler

Signed-off-by: Matthias Wessendorf <[email protected]>
@matzew matzew force-pushed the add_integration_sink_rework branch from 0a3f64e to 9d7e051 Compare November 22, 2024 12:06
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: integration-sink-server-tls
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we are actually not using this.

In a follow up PR, we will generate these Certificate manifest for each IntegrationSource instance (their Service).

Copy link
Member

@pierDipi pierDipi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
/approve

namespace: {{ .namespace }}
{{ if .annotations }}
{{ range $key, $value := .annotations }}
{{ $key }}: {{ $value }}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this helps when having annotations that are "boolean" but need to be represented as strings

Suggested change
{{ $key }}: {{ $value }}
{{ $key }}: "{{ $value }}"

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah ... I fixed similar before, last week.

Will update it. eventually

@knative-prow knative-prow bot added the lgtm Indicates that a PR is ready to be merged. label Nov 22, 2024
Copy link

knative-prow bot commented Nov 22, 2024

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: matzew, pierDipi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@knative-prow knative-prow bot merged commit 7176ce6 into knative:main Nov 22, 2024
32 of 36 checks passed
matzew added a commit to matzew/eventing that referenced this pull request Feb 13, 2025
* create new integration/source folder and factor common code into helper.go

Signed-off-by: Matthias Wessendorf <[email protected]>

* 💄 improve formatting...

Signed-off-by: Matthias Wessendorf <[email protected]>

* fixing import

Signed-off-by: Matthias Wessendorf <[email protected]>

* First Sink

Signed-off-by: Matthias Wessendorf <[email protected]>

* Updates on sink

Signed-off-by: Matthias Wessendorf <[email protected]>

* starting common types for source

Signed-off-by: Matthias Wessendorf <[email protected]>

* starting common types for sink

Signed-off-by: Matthias Wessendorf <[email protected]>

* some code share and refactorings

* use other image coordinates

Signed-off-by: Matthias Wessendorf <[email protected]>

* Add missing api and tests for sink and run codegen

Signed-off-by: Matthias Wessendorf <[email protected]>

* Fixes

Signed-off-by: Matthias Wessendorf <[email protected]>

* Adding cert

Signed-off-by: Matthias Wessendorf <[email protected]>

* Adding test

Signed-off-by: Matthias Wessendorf <[email protected]>

* Rekt test for sink

Signed-off-by: Matthias Wessendorf <[email protected]>

* Rekt test for sink

Signed-off-by: Matthias Wessendorf <[email protected]>

* correct status code

Signed-off-by: Matthias Wessendorf <[email protected]>

* More...

Signed-off-by: Matthias Wessendorf <[email protected]>

* re-add service

Signed-off-by: Matthias Wessendorf <[email protected]>

* delete unversioned common api

Signed-off-by: Matthias Wessendorf <[email protected]>

* Updates, latest conflicts and formatting
Clean ups
Adding tests for reconciler

Signed-off-by: Matthias Wessendorf <[email protected]>

---------

Signed-off-by: Matthias Wessendorf <[email protected]>

conflict

Signed-off-by: Matthias Wessendorf <[email protected]>
openshift-merge-bot bot pushed a commit to openshift-knative/eventing that referenced this pull request Feb 14, 2025
* IntegrationSource CRD (knative#8238)

* Add initial raw API types and controller/reconciler. Levering internally the container source for the handling of the kamelet workloads. the logic for applying the env-vars is a bit verbose, but can be improved.
Tests/lifeccyle and some other improvements are still missing. but this works.

Signed-off-by: Matthias Wessendorf <[email protected]>

* adding reflection to make the mapping less verbose, and more flexible to extend...

Signed-off-by: Matthias Wessendorf <[email protected]>

* Add DDB Streams

Signed-off-by: Matthias Wessendorf <[email protected]>

* polish and add minimal tests

Signed-off-by: Matthias Wessendorf <[email protected]>

* Update pkg/apis/sources/v1alpha1/integration_lifecycle.go

Co-authored-by: Pierangelo Di Pilato <[email protected]>

* Update pkg/apis/sources/v1alpha1/integration_lifecycle.go

Co-authored-by: Pierangelo Di Pilato <[email protected]>

* Fix compiler warnings

Signed-off-by: Matthias Wessendorf <[email protected]>

* use other image coordinates

Signed-off-by: Matthias Wessendorf <[email protected]>

* first test for source

Signed-off-by: Matthias Wessendorf <[email protected]>

* Adding initial rekt test

Signed-off-by: Matthias Wessendorf <[email protected]>

* Adding SSL support and test for IntegrationSource

Signed-off-by: Matthias Wessendorf <[email protected]>

* Formatting fixes

Signed-off-by: Matthias Wessendorf <[email protected]>

* Disable OIDC tests for now

Signed-off-by: Matthias Wessendorf <[email protected]>

* use knative nightly images

Signed-off-by: Matthias Wessendorf <[email protected]>

* Fixing linters

Signed-off-by: Matthias Wessendorf <[email protected]>

* addressing comments, Part1

Signed-off-by: Matthias Wessendorf <[email protected]>

---------

Signed-off-by: Matthias Wessendorf <[email protected]>
Co-authored-by: Pierangelo Di Pilato <[email protected]>

* Add missing copyright boilerplate (knative#8305)

:lipstick: add missing copyright boilerplate

Signed-off-by: Matthias Wessendorf <[email protected]>

* [chore] Use a generic access/secret key in the secret for AWS connections (knative#8312)

:key: Use a generic access/secret key in the secret for AWS connections

Signed-off-by: Matthias Wessendorf <[email protected]>

* [chore] Do not generate SSL env vars for each struct entry, just once per containersource/image (knative#8315)

:lipstick: Do not generate SSL env vars for each struct entry, just once per containersource/image

Signed-off-by: Matthias Wessendorf <[email protected]>

* Update Integration API and use custom tags on structs for better readability (knative#8321)

Update API and use custom tags on structs

Signed-off-by: Matthias Wessendorf <[email protected]>

* Integration Api moved to versioned common (knative#8325)

Moving API to a versioned common/integration folder

Signed-off-by: Matthias Wessendorf <[email protected]>

* Introducing common integration helper and change package name (knative#8327)

* create new integration/source folder and factor common code into helper.go

Signed-off-by: Matthias Wessendorf <[email protected]>

* 💄 improve formatting...

Signed-off-by: Matthias Wessendorf <[email protected]>

* fixing import

Signed-off-by: Matthias Wessendorf <[email protected]>

---------

Signed-off-by: Matthias Wessendorf <[email protected]>

* chore: correct comments (knative#8336)

:lipstick: correct comments ...

Signed-off-by: Matthias Wessendorf <[email protected]>

* Add IntegrationSink CRD  (knative#8304)

* create new integration/source folder and factor common code into helper.go

Signed-off-by: Matthias Wessendorf <[email protected]>

* 💄 improve formatting...

Signed-off-by: Matthias Wessendorf <[email protected]>

* fixing import

Signed-off-by: Matthias Wessendorf <[email protected]>

* First Sink

Signed-off-by: Matthias Wessendorf <[email protected]>

* Updates on sink

Signed-off-by: Matthias Wessendorf <[email protected]>

* starting common types for source

Signed-off-by: Matthias Wessendorf <[email protected]>

* starting common types for sink

Signed-off-by: Matthias Wessendorf <[email protected]>

* some code share and refactorings

* use other image coordinates

Signed-off-by: Matthias Wessendorf <[email protected]>

* Add missing api and tests for sink and run codegen

Signed-off-by: Matthias Wessendorf <[email protected]>

* Fixes

Signed-off-by: Matthias Wessendorf <[email protected]>

* Adding cert

Signed-off-by: Matthias Wessendorf <[email protected]>

* Adding test

Signed-off-by: Matthias Wessendorf <[email protected]>

* Rekt test for sink

Signed-off-by: Matthias Wessendorf <[email protected]>

* Rekt test for sink

Signed-off-by: Matthias Wessendorf <[email protected]>

* correct status code

Signed-off-by: Matthias Wessendorf <[email protected]>

* More...

Signed-off-by: Matthias Wessendorf <[email protected]>

* re-add service

Signed-off-by: Matthias Wessendorf <[email protected]>

* delete unversioned common api

Signed-off-by: Matthias Wessendorf <[email protected]>

* Updates, latest conflicts and formatting
Clean ups
Adding tests for reconciler

Signed-off-by: Matthias Wessendorf <[email protected]>

---------

Signed-off-by: Matthias Wessendorf <[email protected]>

conflict

Signed-off-by: Matthias Wessendorf <[email protected]>

* IntegrationSink: rek-test templating support for bool annotations (knative#8342)

bool annotations for represented as string, to avoid parsing errors

Signed-off-by: Matthias Wessendorf <[email protected]>

* Update integrationsink crd  (knative#8347)

* Align IntegrationSink CRD w/ IntegrationSource CRD

Signed-off-by: Matthias Wessendorf <[email protected]>

* 💄 Remove old job-sink references

Signed-off-by: Matthias Wessendorf <[email protected]>

* Docs change

Signed-off-by: Matthias Wessendorf <[email protected]>

---------

Signed-off-by: Matthias Wessendorf <[email protected]>

* Avoid using twice hardcoded strings for the names of the Kube Secret keys, for AWS access/secret key vals (knative#8345)

:lipstick: Avoid using twice hardcoded strings for the names of the Kubescret keys, for AWS access/secret key vals

Signed-off-by: Matthias Wessendorf <[email protected]>

* Add integrationsinks-addressable-resolver cluster role (knative#8356)

:passport_control: Add integrationsinks-addressable-resolver cluster role

Signed-off-by: Matthias Wessendorf <[email protected]>

* Align label generation and apply on container-source as well (knative#8355)

Signed-off-by: Matthias Wessendorf <[email protected]>

* We changed the event type on the images (knative#8364)

:fire: We changed the event type on the images

Signed-off-by: Matthias Wessendorf <[email protected]>

* Move image to map (knative#8362)

Signed-off-by: Matthias Wessendorf <[email protected]>

* Adding sns sink to Integration Sink (knative#8365)

adding sns sink

Signed-off-by: Matthias Wessendorf <[email protected]>

* Clean up remove session token (knative#8369)

Remove session token and profile credential name

Signed-off-by: Matthias Wessendorf <[email protected]>

* Chore: field semantic correction  (knative#8370)

:lipstick: We renamed those fields to , and we reflect that now in validation

Signed-off-by: Matthias Wessendorf <[email protected]>

* Remove unused filter function (knative#8375)

:broom: Remove unused filter function

Signed-off-by: Matthias Wessendorf <[email protected]>

* add new integration CRDs to storage version migrator tool (knative#8386)

Signed-off-by: Matthias Wessendorf <[email protected]>

* Going OIDC for Integration Source (knative#8383)

Going OIDC for Integration Source:
- Generating OIDC specific evn_vars for Knative client of camel
- Adding rekt-test for OIDC feature of the source

Signed-off-by: Matthias Wessendorf <[email protected]>

* Adding reconciler test for OIDC of the Integration Source (knative#8404)

Signed-off-by: Matthias Wessendorf <[email protected]>

* Not just render oidc env-vars when cfg is on, we need also a present audience to ensure the camel containers work correct (knative#8448)

Signed-off-by: Matthias Wessendorf <[email protected]>

* run `make generate-release`

---------

Signed-off-by: Matthias Wessendorf <[email protected]>
Co-authored-by: Pierangelo Di Pilato <[email protected]>
Co-authored-by: Christoph Stäbler <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/test-and-release Test infrastructure, tests or release lgtm Indicates that a PR is ready to be merged. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants