Skip to content

line protocol support #569

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 7 commits into
base: master
Choose a base branch
from
Draft

line protocol support #569

wants to merge 7 commits into from

Conversation

lmangani
Copy link
Member

@lmangani lmangani commented Apr 24, 2025

Add GigAPI / InfluxDB Line Protocol Export Support

This PR adds support for exporting HEP data to GigAPI using the Line Protocol format.

Features

  • New Line Protocol exporter that converts HEP packets to InfluxDB Line Protocol format
  • Measurement names based on HEP type (e.g., hep_1 for SIP, hep_5 for RTCP)
  • Comprehensive tagging system including:
    • Source/destination IP and ports
    • Protocol type (TCP/UDP)
    • Hostname and node information
  • Rich field data including:
    • Timestamps in milliseconds
    • Payload size
    • SIP-specific fields (method, response, call ID)
    • Full payload content

Configuration

New configuration options have been added:

# InfluxDB connection URL
LineProtoURL = "http://localhost:8086"

# Batch settings
LineProtoBulk = 400      # Number of lines to batch before sending
LineProtoTimer = 4       # Time in seconds to wait before sending a batch
LineProtoBuffer = 100000 # Size of the channel buffer

# HEP type filter (default: 1,5,100)
LineProtoHEPFilter = [1,5,100]

Example Output

The exporter generates Line Protocol entries in this format:

hep_1,src_ip=192.168.1.1,dst_ip=192.168.1.2,src_port=5060,dst_port=5060 create_date=1618426800000i,sip_method="INVITE",call_id="[email protected]",payload_size=245i 1618426800000000000

Web Interface

The web configuration interface has been updated to include the new Line Protocol settings, allowing users to configure the exporter through the UI.

Dependencies

No new external dependencies were added. The implementation uses the standard Go HTTP client and string manipulation libraries.

Documentation

The feature is documented in the code and configuration examples. Users can refer to the InfluxDB Line Protocol specification for details about the output format.

… with excessive size value

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
@lmangani lmangani requested a review from Copilot April 24, 2025 10:17
Copilot

This comment was marked as resolved.

@sipcapture sipcapture deleted a comment from CLAassistant Apr 24, 2025
@CLAassistant
Copy link

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.


Lorenzo Mangani seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

@lmangani lmangani requested a review from Copilot April 24, 2025 10:38
Copilot

This comment was marked as resolved.

@lmangani lmangani requested a review from Copilot April 24, 2025 10:42
Copilot

This comment was marked as resolved.

@lmangani lmangani requested a review from Copilot April 24, 2025 10:59
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds support for exporting HEP data to GigAPI using InfluxDB’s Line Protocol format. Key changes include the introduction of a new channel and processing logic for Line Protocol packets in the server, registration and implementation of a new "lineproto" remote logger, and updates to web configuration and application settings to support the exporter.

Reviewed Changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
server/server.go Added new channel (lineCh) and processing logic for Line Protocol export
remotelog/remotelog.go Registered the new "lineproto" remote logger in the handler mapping
remotelog/lineproto.go Implemented Line Protocol exporter with batching and HTTP POST support
config/webconfig.go Integrated new UI settings for Line Protocol exporter configuration
config/config.go Added configuration fields for Line Protocol parameters

@@ -78,6 +80,10 @@ func NewHEPInput() *HEPInput {
h.useLK = true
h.lokiCh = make(chan *decoder.HEP, config.Setting.LokiBuffer)
}
if len(config.Setting.LineProtoURL) > 2 {
Copy link
Preview

Copilot AI Apr 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider using a more explicit check for LineProtoURL (e.g., checking if the string is non-empty) to improve code clarity and reduce potential misconfigurations.

Suggested change
if len(config.Setting.LineProtoURL) > 2 {
if trimmedURL := strings.TrimSpace(config.Setting.LineProtoURL); trimmedURL != "" {

Copilot uses AI. Check for mistakes.

}
lastWarn = time.Now()
}
break
Copy link
Preview

Copilot AI Apr 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ensure that breaking out of the filter loop after the first matching value is intended behavior, particularly when a HEP packet might satisfy multiple filter criteria.

Suggested change
break
// Continue checking other filters

Copilot uses AI. Check for mistakes.

Comment on lines +45 to +49
_, err = http.Get(l.URL)
if err != nil {
return err
}
return nil
Copy link
Preview

Copilot AI Apr 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider checking the HTTP response status after the GET request to validate a successful connection rather than just relying on the absence of an error.

Suggested change
_, err = http.Get(l.URL)
if err != nil {
return err
}
return nil
resp, err := http.Get(l.URL)
if err != nil {
return err
}
defer resp.Body.Close()
if resp.StatusCode/100 != 2 {
return fmt.Errorf("server returned HTTP status %s (%d)", resp.Status, resp.StatusCode)
}

Copilot uses AI. Check for mistakes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants