Skip to content

Update collector#73

Merged
stefanDeveloper merged 7 commits intomainfrom
update-collector
Apr 22, 2025
Merged

Update collector#73
stefanDeveloper merged 7 commits intomainfrom
update-collector

Conversation

@lamr02n
Copy link
Collaborator

@lamr02n lamr02n commented Apr 16, 2025

The Log Collector used to have a high latency. This PR changed its implementation from an internal queue to the same approach used in the Log Storage module. First tests show a drastic improvement in latency.

Resolves #71.

@lamr02n lamr02n added bug Something isn't working enhancement New feature or request labels Apr 16, 2025
@lamr02n lamr02n self-assigned this Apr 16, 2025
@lamr02n lamr02n linked an issue Apr 16, 2025 that may be closed by this pull request
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

@lamr02n lamr02n marked this pull request as draft April 16, 2025 14:13
@lamr02n lamr02n requested a review from Copilot April 16, 2025 15:21
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR updates the Log Collector implementation to reduce latency by replacing the internal queue approach with a synchronous send method similar to what is used in the Log Storage module. Key changes include:

  • Converting the send() method from asynchronous to a synchronous function that accepts timestamp and message arguments.
  • Adapting test cases to patch and call send() instead of store(), with adjustments to the test classes accordingly.

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
tests/logcollector/test_collector.py Updated test cases to reflect the new synchronous send() interface and updated mocking accordingly.
src/logcollector/collector.py Modified start(), fetch(), and send() methods to implement the new, low-latency Log Collector design.

@github-actions
Copy link

github-actions bot commented Apr 16, 2025

☂️ Python Coverage

current status: ✅

Overall Coverage

Lines Covered Coverage Threshold Status
1437 1428 99% 0% 🟢

New Files

No new covered files...

Modified Files

File Coverage Status
src/logcollector/collector.py 100% 🟢
src/monitoring/clickhouse_batch_sender.py 99% 🟢
TOTAL 99% 🟢

updated for commit: 6dbfc2a by action🐍

@codecov
Copy link

codecov bot commented Apr 16, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.88%. Comparing base (09e5f1f) to head (6dbfc2a).
Report is 8 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main      #73      +/-   ##
==========================================
- Coverage   98.90%   98.88%   -0.02%     
==========================================
  Files          15       15              
  Lines        1455     1437      -18     
  Branches      151      149       -2     
==========================================
- Hits         1439     1421      -18     
  Misses          9        9              
  Partials        7        7              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@lamr02n lamr02n marked this pull request as ready for review April 22, 2025 06:43
Copy link
Owner

@stefanDeveloper stefanDeveloper left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good!

@stefanDeveloper stefanDeveloper merged commit e0b0f9c into main Apr 22, 2025
18 checks passed
@stefanDeveloper stefanDeveloper deleted the update-collector branch April 22, 2025 10:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improve Log Collector's performance

3 participants