Skip to content

Conversation

@Dugong42
Copy link

@Dugong42 Dugong42 commented Mar 28, 2025

Problem

Connectors produce too many (100+/s) repetitive connection logs when querying a large number of rows.

A message is logged by the CachedConnectionProvider for each fetching of rows regardless of whether a new connection is created or the existing connection is used:

Database connection established

See #1493

Solution

The repetitive log has been moved to the CachedConnectionProvider.newConnection() method to only record new connections.

Access to the CachedConnectionProvider is logged as DEBUG when an existing Connection is used.

@Dugong42 Dugong42 requested a review from a team as a code owner March 28, 2025 15:39
@confluent-cla-assistant
Copy link

confluent-cla-assistant bot commented Mar 28, 2025

🎉 All Contributor License Agreements have been signed. Ready to merge.
✅ Dugong42
Please push an empty commit if you would like to re-run the checks to verify CLA status for all contributors.

@Dugong42 Dugong42 changed the title Reduce Repetitive Logs in Bulk Mode by Adjusting Connection Logging MINOR: Reduce Repetitive Logs in Bulk Mode by Adjusting Connection Logging Mar 28, 2025
@Dugong42 Dugong42 changed the base branch from master to 10.3.x March 28, 2025 15:53
@Dugong42 Dugong42 changed the base branch from 10.3.x to 10.8.x March 28, 2025 15:55
@Dugong42 Dugong42 changed the base branch from 10.8.x to master March 28, 2025 15:55
@Dugong42 Dugong42 changed the title MINOR: Reduce Repetitive Logs in Bulk Mode by Adjusting Connection Logging MINOR: Reduce repetitive logs on large queries by adjusting connection logging Mar 28, 2025
@Dugong42 Dugong42 force-pushed the feat/reduce-cached-connection-logging branch from ff52855 to 8f5f5d9 Compare March 28, 2025 16:09
@Dugong42 Dugong42 changed the base branch from master to 10.6.x March 28, 2025 16:09
@Dugong42 Dugong42 force-pushed the feat/reduce-cached-connection-logging branch from 8f5f5d9 to 23ecb52 Compare March 28, 2025 16:42
@Dugong42 Dugong42 force-pushed the feat/reduce-cached-connection-logging branch from 23ecb52 to b13021c Compare March 28, 2025 17:03
@Dugong42
Copy link
Author

Dugong42 commented Mar 31, 2025

As it is recommended to back-port fixes to earlier branches, I'm submitting this change to the 10.6 branch.
However there'll be a small conflict when merging to 10.7, as the log message that I modified was changed from DEBUG to INFO on that branch.

@Dugong42
Copy link
Author

Dugong42 commented May 7, 2025

Hi, any chance for a review ? :)
@rajdangwal
Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant