Skip to content

feat: funnel metrics to ClickHouse and enable exporters for observability data #407

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 47 commits into
base: main
Choose a base branch
from
Draft

feat: funnel metrics to ClickHouse and enable exporters for observability data #407

wants to merge 47 commits into from

Conversation

@rohanarora
Copy link
Collaborator

@rohanarora rohanarora commented Nov 17, 2025

This PR implements a comprehensive observability data pipeline to ClickHouse with support for on-demand exports.

Changes:

  • Metrics ingestion: Prometheus metrics are now pushed to the Prometheus database in ClickHouse
  • Bug fix: Resolved issues with the Kubernetes events exporter
  • Export recorder: Added capability to export the following observability data on-call:
  1. Logs
  2. Metrics
  3. Traces
  4. Kubernetes events
  5. Kubernetes resource objects (polled every 5 minutes)

This enables centralized storage of observability data in ClickHouse and an exporter to serve as an example.

@rohanarora
fix: Kubernetes event exporter
80c83b2 
Signed-off-by: Rohan R. Arora <[email protected]>
@rohanarora
bump: enabling ClickHouse to receive Prometheus metrics
ff63de5 
Signed-off-by: Rohan R. Arora <[email protected]>
@rohanarora
feat: adding the ability to create snapshot of K8s objects and pushin…
9587f6b 
…g it to ClickHouse

Signed-off-by: Rohan R. Arora <[email protected]>
@rohanarora
bump: initial pass
fa7808e 
Signed-off-by: Rohan R. Arora <[email protected]>
@rohanarora
feat: leveraging the existing recorder roles as opposed to creating a…
6053bfe 
… new one

Signed-off-by: Rohan R. Arora <[email protected]>
@rohanarora
fix: argument validation failure
682ab2f 
Signed-off-by: Rohan R. Arora <[email protected]>
@rohanarora
bump: setting it up as a K8s job
5f8c2a2 
Signed-off-by: Rohan R. Arora <[email protected]>
@rohanarora
fix: missing ClickHouse credentials
3c7754d 
Signed-off-by: Rohan R. Arora <[email protected]>
@rohanarora
fix: missing ClickHouse credentials - 2
be4216b 
Signed-off-by: Rohan R. Arora <[email protected]>
@rohanarora
fix: environment variables for credentials
f950e36 
Signed-off-by: Rohan R. Arora <[email protected]>
@rohanarora
bump: addressing Could not find a version that satisfies the requirem…
a8538ba 
…ent pyarrow==22.0.1

Signed-off-by: Rohan R. Arora <[email protected]>
@rohanarora
bump:setting ClickHouse endpoint leveraging the environment variable
f3a5231 
Signed-off-by: Rohan R. Arora <[email protected]>
@rohanarora
fix: username instead of user
9602031 
Signed-off-by: Rohan R. Arora <[email protected]>
@rohanarora
bump: can skip the port number for ClickHouse
d1d86fd 
Signed-off-by: Rohan R. Arora <[email protected]>
@rohanarora
fix: remove protocol from clickhouse hostname
55108b6 
Signed-off-by: Rohan R. Arora <[email protected]>
@rohanarora
bump: correcting labels
59abe3c 
Signed-off-by: Rohan R. Arora <[email protected]>
@rohanarora
bump: resource bump for job
84f361d 
Signed-off-by: Rohan R. Arora <[email protected]>
@rohanarora
bump: consolidating install-uninstall for ClickHouse exporter
4a08c28 
Signed-off-by: Rohan R. Arora <[email protected]>
@rohanarora
bump: dedicated ClickHouse-based collection node
9de101a 
Signed-off-by: Rohan R. Arora <[email protected]>
@rohanarora
bump: resource bump for job - 2
de25bbc 
Signed-off-by: Rohan R. Arora <[email protected]>
@rohanarora
fix: label
c62f222 
Signed-off-by: Rohan R. Arora <[email protected]>
@rohanarora
bump: further resource bump for job - 2
72f0558 
Signed-off-by: Rohan R. Arora <[email protected]>
@rohanarora
bump: allowing for more time for the job to complete
1a1af29 
Signed-off-by: Rohan R. Arora <[email protected]>
@rohanarora
bump: further resource bump for job - 3
7bea403 
Signed-off-by: Rohan R. Arora <[email protected]>
@rohanarora
fix: directory path / location for data collected from ClickHouse
c433b55 
Signed-off-by: Rohan R. Arora <[email protected]>
@rohanarora
bump: adding missing recursive argument
cf6a994 
Signed-off-by: Rohan R. Arora <[email protected]>
@rohanarora
bump: ClickHouse resource limits
b888349 
Signed-off-by: Rohan R. Arora <[email protected]>
@rohanarora
bump: Prometheus namespace deletion taking longer than expected
57804fb 
Signed-off-by: Rohan R. Arora <[email protected]>
@rohanarora
bump: enabling logs and traces
22211ee 
Signed-off-by: Rohan R. Arora <[email protected]>
@rohanarora
fix: correction from recursive to recurse
53670e0
@rohanarora rohanarora requested a review from Red-GV as a code owner November 17, 2025 15:16
@rohanarora rohanarora marked this pull request as draft November 17, 2025 15:17
@rohanarora
fix: correcting Makefile command
8f2fb56 
Signed-off-by: Rohan R. Arora <[email protected]>
@rohanarora
fix: correcting phase value
6cdb947 
Signed-off-by: Rohan R. Arora <[email protected]>
@rohanarora
bump: elastic IP correction
64e5748 
Signed-off-by: Rohan R. Arora <[email protected]>
@rohanarora
bump: wrapping up exporter from the SRE lens
5768194 
Signed-off-by: Rohan R. Arora <[email protected]>
@rohanarora
bump: bumping up resource limits-requests for flagd and recommendatio…
aac337c 
…n service

Signed-off-by: Rohan R. Arora <[email protected]>
@rohanarora
bump: adding filters for logs and traces; defaulting to error and war…
61cc1bc 
…ning

Signed-off-by: Rohan R. Arora <[email protected]>
@rohanarora
bump: chart version bump consistent with main
43b0be0 
Signed-off-by: Rohan R. Arora <[email protected]>
@rohanarora
bump: adding pause to allow for the application to stabilize post ins…
897a73e 
…tallation

Signed-off-by: Rohan R. Arora <[email protected]>
@rohanarora
bump: adding default set of namespaces for k8s objects
7b2c7cd 
Signed-off-by: Rohan R. Arora <[email protected]>
@rohanarora
bump: accounting service resource bump
48ebd27 
Signed-off-by: Rohan R. Arora <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Red-GV Red-GV Awaiting requested review from Red-GV Red-GV is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@rohanarora