Skip to content

Commit 48f9206

Browse files
feat: OpenTelemetry syslog exporter (#2132)
* Initial addition of syslog exporter * Add syslog exporter converter, docs, and changelog * update docs * Update docs * Fix syslogexporter converter * Fix promtail converter for syslog * Update docs/sources/reference/components/otelcol/otelcol.exporter.syslog.md Co-authored-by: Clayton Cornell <[email protected]> * Update docs/sources/reference/components/otelcol/otelcol.exporter.syslog.md Co-authored-by: Clayton Cornell <[email protected]> * Update docs for syslog exporter * Fix marshaling of string alias * Make the syslog exporter test consistent * Add more reliability in to syslog test * Update docs/sources/reference/components/otelcol/otelcol.exporter.syslog.md Co-authored-by: Clayton Cornell <[email protected]> * Apply suggestions from code review Co-authored-by: Clayton Cornell <[email protected]> --------- Co-authored-by: Clayton Cornell <[email protected]>
1 parent 6aec342 commit 48f9206

File tree

12 files changed

+601
-25
lines changed

12 files changed

+601
-25
lines changed

CHANGELOG.md

+2
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,8 @@ Main (unreleased)
2020

2121
- Add `otelcol.receiver.solace` component to receive traces from a Solace broker. (@wildum)
2222

23+
- Add `otelcol.exporter.syslog` component to export logs in syslog format (@dehaansa)
24+
2325
### Enhancements
2426

2527
- Add second metrics sample to the support bundle to provide delta information (@dehaansa)

docs/sources/reference/compatibility/_index.md

+1
Original file line numberDiff line numberDiff line change
@@ -299,6 +299,7 @@ The following components, grouped by namespace, _export_ OpenTelemetry `otelcol.
299299
- [otelcol.exporter.otlphttp](../components/otelcol/otelcol.exporter.otlphttp)
300300
- [otelcol.exporter.prometheus](../components/otelcol/otelcol.exporter.prometheus)
301301
- [otelcol.exporter.splunkhec](../components/otelcol/otelcol.exporter.splunkhec)
302+
- [otelcol.exporter.syslog](../components/otelcol/otelcol.exporter.syslog)
302303
- [otelcol.processor.attributes](../components/otelcol/otelcol.processor.attributes)
303304
- [otelcol.processor.batch](../components/otelcol/otelcol.processor.batch)
304305
- [otelcol.processor.deltatocumulative](../components/otelcol/otelcol.processor.deltatocumulative)
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,244 @@
1+
---
2+
canonical: https://grafana.com/docs/alloy/latest/reference/components/otelcol/otelcol.exporter.syslog/
3+
description: Learn about otelcol.exporter.syslog
4+
title: otelcol.exporter.syslog
5+
---
6+
7+
# otelcol.exporter.syslog
8+
9+
{{< docs/shared lookup="stability/public_preview.md" source="alloy" version="<ALLOY_VERSION>" >}}
10+
11+
`otelcol.exporter.syslog` accepts logs from other `otelcol` components and writes them over the network using the syslog protocol.
12+
It supports syslog protocols [RFC5424][] and [RFC3164][] and can send data over `TCP` or `UDP`.
13+
14+
{{< admonition type="note" >}}
15+
`otelcol.exporter.syslog` is a wrapper over the upstream OpenTelemetry Collector `syslog` exporter.
16+
Bug reports or feature requests will be redirected to the upstream repository, if necessary.
17+
{{< /admonition >}}
18+
19+
You can specify multiple `otelcol.exporter.syslog` components by giving them different labels.
20+
21+
[RFC5424]: https://www.rfc-editor.org/rfc/rfc5424
22+
[RFC3164]: https://www.rfc-editor.org/rfc/rfc3164
23+
24+
## Usage
25+
26+
```alloy
27+
otelcol.exporter.syslog "LABEL" {
28+
endpoint = "HOST"
29+
}
30+
```
31+
32+
### Supported Attributes
33+
34+
The exporter creates one syslog message for each log record based on the following attributes of the log record.
35+
If an attribute is missing, the default value is used. The log's timestamp field is used for the syslog message's time.
36+
RFC3164 only supports a subset of the attributes supported by RFC5424, and the default values are not the same between
37+
the two protocols. Refer to the [OpenTelemetry documentation][upstream_readme] for the exporter for more details.
38+
39+
| Attribute name | Type | RFC5424 Default value | RFC3164 supported | RFC3164 Default value
40+
| ----------------- | ------ | ---------------------- |------------------ | ----------------------
41+
| `appname` | string | `-` | yes | empty string
42+
| `hostname` | string | `-` | yes | `-`
43+
| `message` | string | empty string | yes | empty string
44+
| `msg_id` | string | `-` | no |
45+
| `priority` | int | `165` | yes | `165`
46+
| `proc_id` | string | `-` | no |
47+
| `structured_data` | map | `-` | no |
48+
| `version` | int | `1` | no |
49+
50+
[upstream_readme]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/<OTEL_VERSION>/exporter/syslogexporter
51+
52+
## Arguments
53+
54+
`otelcol.exporter.syslog` supports the following arguments:
55+
56+
| Name | Type | Description | Default | Required |
57+
|------------------------|-----------|---------------------------------------------------------------------------|-----------------------------------|----------|
58+
| `endpoint` | `string` | The endpoint to send syslog formatted logs to. | | yes |
59+
| `network` | `string` | The type of network connection to use to send logs. | tcp | no |
60+
| `port` | `int` | The port where the syslog server accepts connections. | 514 | no |
61+
| `protocol` | `string` | The syslog protocol that the syslog server supports. | rfc5424 | no |
62+
| `enable_octet_counting`| `bool` | Whether to enable rfc6587 octet counting. | false | no |
63+
| `timeout` | `duration`| Time to wait before marking a request as failed. | 5s | no |
64+
65+
The `network` argument specifies if the syslog endpoint is using the TCP or UDP protocol.
66+
`network` must be one of `tcp`, `udp`
67+
68+
The `protocol` argument specifies the syslog format supported by the endpoint.
69+
`protocol` must be one of `rfc5424`, `rfc3164`
70+
71+
## Blocks
72+
73+
The following blocks are supported inside the definition of `otelcol.exporter.syslog`:
74+
75+
| Hierarchy | Block | Description | Required |
76+
|------------------|----------------------|----------------------------------------------------------------------------|----------|
77+
| tls | [tls][] | Configures TLS for a TCP connection. | no |
78+
| sending_queue | [sending_queue][] | Configures batching of data before sending. | no |
79+
| retry_on_failure | [retry_on_failure][] | Configures retry mechanism for failed requests. | no |
80+
| debug_metrics | [debug_metrics][] | Configures the metrics that this component generates to monitor its state. | no |
81+
82+
[tls]: #tls-block
83+
[sending_queue]: #sending_queue-block
84+
[retry_on_failure]: #retry_on_failure-block
85+
[debug_metrics]: #debug_metrics-block
86+
87+
### tls block
88+
89+
The `tls` block configures TLS settings used for a connection to a TCP syslog server.
90+
91+
{{< docs/shared lookup="reference/components/otelcol-tls-client-block.md" source="alloy" version="<ALLOY_VERSION>" >}}
92+
93+
### sending_queue block
94+
95+
The `sending_queue` block configures an in-memory buffer of batches before data is sent to the syslog server.
96+
97+
{{< docs/shared lookup="reference/components/otelcol-queue-block.md" source="alloy" version="<ALLOY_VERSION>" >}}
98+
99+
### retry_on_failure block
100+
101+
The `retry_on_failure` block configures how failed requests to the syslog server are retried.
102+
103+
{{< docs/shared lookup="reference/components/otelcol-retry-block.md" source="alloy" version="<ALLOY_VERSION>" >}}
104+
105+
### debug_metrics block
106+
107+
{{< docs/shared lookup="reference/components/otelcol-debug-metrics-block.md" source="alloy" version="<ALLOY_VERSION>" >}}
108+
109+
## Exported fields
110+
111+
The following fields are exported and can be referenced by other components:
112+
113+
| Name | Type | Description
114+
|--------|--------------------|-----------------------------------------------------------------
115+
|`input` | `otelcol.Consumer` | A value that other components can use to send telemetry data to.
116+
117+
`input` accepts `otelcol.Consumer` data for logs. Other telemetry signals are ignored.
118+
119+
## Component health
120+
121+
`otelcol.exporter.syslog` is only reported as unhealthy if given an invalid configuration.
122+
123+
## Debug information
124+
125+
`otelcol.exporter.syslog` doesn't expose any component-specific debug information.
126+
127+
## Examples
128+
129+
### TCP endpoint without TLS
130+
131+
This example creates an exporter to send data to a syslog server expecting RFC5424-compliant messages over TCP without TLS:
132+
133+
```alloy
134+
otelcol.exporter.syslog "default" {
135+
endpoint = "localhost"
136+
tls {
137+
insecure = true
138+
insecure_skip_verify = true
139+
}
140+
}
141+
```
142+
143+
### Use the `otelcol.processor.transform` component to format logs from `loki.source.syslog`
144+
145+
This example shows one of the methods for annotating your loki messages into the format expected
146+
by the exporter using a `otelcol.receiver.loki` component in addition to the `otelcol.processor.transform`
147+
component. This example assumes that the log messages being parsed have come from a `loki.source.syslog`
148+
component. This is just an example of some of the techniques that can be applied, and not a fully functioning
149+
example for a specific incoming log.
150+
151+
```alloy
152+
otelcol.receiver.loki "default" {
153+
output {
154+
logs = [otelcol.processor.transform.syslog.input]
155+
}
156+
}
157+
158+
otelcol.processor.transform "syslog" {
159+
error_mode = "ignore"
160+
161+
log_statements {
162+
context = "log"
163+
164+
statements = [
165+
`set(attributes["message"], attributes["__syslog_message"])`,
166+
`set(attributes["appname"], attributes["__syslog_appname"])`,
167+
`set(attributes["hostname"], attributes["__syslog_hostname"])`,
168+
169+
// To set structured data you can chain index ([]) operations.
170+
`set(attributes["structured_data"]["auth@32473"]["user"], attributes["__syslog_message_sd_auth_32473_user"])`,
171+
`set(attributes["structured_data"]["auth@32473"]["user_host"], attributes["__syslog_message_sd_auth_32473_user_host"])`,
172+
`set(attributes["structured_data"]["auth@32473"]["valid"], attributes["__syslog_message_sd_auth_32473_authenticated"])`,
173+
]
174+
}
175+
176+
output {
177+
metrics = []
178+
logs = [otelcol.exporter.syslog.default.input]
179+
traces = []
180+
}
181+
}
182+
```
183+
184+
### Use the `otelcol.processor.transform` component to format OpenTelemetry logs
185+
186+
This example shows one of the methods for annotating your messages in the OpenTelemetry log format into the format expected
187+
by the exporter using an `otelcol.processor.transform` component. This example assumes that the log messages being
188+
parsed have come from another OpenTelemetry receiver in JSON format (or have been transformed to OpenTelemetry logs using
189+
an `otelcol.receiver.loki` component). This is just an example of some of the techniques that can be applied, and not a
190+
fully functioning example for a specific incoming log format.
191+
192+
```alloy
193+
otelcol.processor.transform "syslog" {
194+
error_mode = "ignore"
195+
196+
log_statements {
197+
context = "log"
198+
199+
statements = [
200+
// Parse body as JSON and merge the resulting map with the cache map, ignoring non-json bodies.
201+
// cache is a field exposed by OTTL that is a temporary storage place for complex operations.
202+
`merge_maps(cache, ParseJSON(body), "upsert") where IsMatch(body, "^\\{")`,
203+
204+
// Set some example syslog attributes using the values from a JSON message body
205+
// If the attribute doesn't exist in cache then nothing happens.
206+
`set(attributes["message"], cache["log"])`,
207+
`set(attributes["appname"], cache["application"])`,
208+
`set(attributes["hostname"], cache["source"])`,
209+
210+
// To set structured data you can chain index ([]) operations.
211+
`set(attributes["structured_data"]["auth@32473"]["user"], attributes["user"])`,
212+
`set(attributes["structured_data"]["auth@32473"]["user_host"], cache["source"])`,
213+
`set(attributes["structured_data"]["auth@32473"]["valid"], cache["authenticated"])`,
214+
215+
// Example priority setting, using facility 1 (user messages) and default to Info
216+
`set(attributes["priority"], 14)`,
217+
`set(attributes["priority"], 12) where severity_number == SEVERITY_NUMBER_WARN`,
218+
`set(attributes["priority"], 11) where severity_number == SEVERITY_NUMBER_ERROR`,
219+
`set(attributes["priority"], 10) where severity_number == SEVERITY_NUMBER_FATAL`,
220+
]
221+
}
222+
223+
output {
224+
metrics = []
225+
logs = [otelcol.exporter.syslog.default.input]
226+
traces = []
227+
}
228+
}
229+
```
230+
231+
<!-- START GENERATED COMPATIBLE COMPONENTS -->
232+
233+
## Compatible components
234+
235+
`otelcol.exporter.syslog` has exports that can be consumed by the following components:
236+
237+
- Components that consume [OpenTelemetry `otelcol.Consumer`](../../../compatibility/#opentelemetry-otelcolconsumer-consumers)
238+
239+
{{< admonition type="note" >}}
240+
Connecting some components may not be sensible or components may require further configuration to make the connection work correctly.
241+
Refer to the linked documentation for more details.
242+
{{< /admonition >}}
243+
244+
<!-- END GENERATED COMPATIBLE COMPONENTS -->

go.mod

+1
Original file line numberDiff line numberDiff line change
@@ -830,6 +830,7 @@ require (
830830
github.com/ebitengine/purego v0.8.0 // indirect
831831
github.com/elastic/lunes v0.1.0 // indirect
832832
github.com/moby/sys/userns v0.1.0 // indirect
833+
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/syslogexporter v0.112.0 // indirect
833834
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/kafka/topic v0.112.0 // indirect
834835
github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10 // indirect
835836
go.opentelemetry.io/collector/connector/connectorprofiles v0.112.0 // indirect

go.sum

+2
Original file line numberDiff line numberDiff line change
@@ -1940,6 +1940,8 @@ github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexp
19401940
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter v0.112.0/go.mod h1:QwYTlmQDuLeaxS0HkIG9K9x45vQhHzL0SvI8inxzMeU=
19411941
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/splunkhecexporter v0.112.0 h1:bIoCW8VYBEGnvpNYlamlvkPyeoQHCtfGgEuuELJYWYE=
19421942
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/splunkhecexporter v0.112.0/go.mod h1:7usJQKG52/DDvzJ7Vm5+QEBE1eAYrVhEYbzYFzfkn2Q=
1943+
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/syslogexporter v0.112.0 h1:p48hoUvtg9lWOlTFbaG9DfxKg15KK3V6cMXpyfCfoT4=
1944+
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/syslogexporter v0.112.0/go.mod h1:CIFj32FwT/eauhbQgxUs53LObCOzoRhjLzZgDjOJB4Y=
19431945
github.com/open-telemetry/opentelemetry-collector-contrib/extension/basicauthextension v0.112.0 h1:RY0/7LTffj76403QxSlEjb0gnF788Qyfpxc+y32Rd6c=
19441946
github.com/open-telemetry/opentelemetry-collector-contrib/extension/basicauthextension v0.112.0/go.mod h1:1Z84oB3hwUH1B3IsL46csEtu7WA1qQJ/p6USTulGJf4=
19451947
github.com/open-telemetry/opentelemetry-collector-contrib/extension/bearertokenauthextension v0.112.0 h1:GLh1rnXcY4P2hkMwuMLYCZMjZxze1KnciJXJTOFXOJ8=

internal/component/all/all.go

+1
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,7 @@ import (
7878
_ "github.com/grafana/alloy/internal/component/otelcol/exporter/otlphttp" // Import otelcol.exporter.otlphttp
7979
_ "github.com/grafana/alloy/internal/component/otelcol/exporter/prometheus" // Import otelcol.exporter.prometheus
8080
_ "github.com/grafana/alloy/internal/component/otelcol/exporter/splunkhec" // Import otelcol.exporter.splunkhec
81+
_ "github.com/grafana/alloy/internal/component/otelcol/exporter/syslog" // Import otelcol.exporter.syslog
8182
_ "github.com/grafana/alloy/internal/component/otelcol/extension/jaeger_remote_sampling" // Import otelcol.extension.jaeger_remote_sampling
8283
_ "github.com/grafana/alloy/internal/component/otelcol/processor/attributes" // Import otelcol.processor.attributes
8384
_ "github.com/grafana/alloy/internal/component/otelcol/processor/batch" // Import otelcol.processor.batch

internal/component/common/config/types.go

+29
Original file line numberDiff line numberDiff line change
@@ -408,3 +408,32 @@ func (o *OAuth2Config) Validate() error {
408408

409409
return o.ProxyConfig.Validate()
410410
}
411+
412+
type SysLogFormat string
413+
414+
const (
415+
// A modern Syslog RFC
416+
SyslogFormatRFC5424 SysLogFormat = "rfc5424"
417+
// A legacy Syslog RFC also known as BSD-syslog
418+
SyslogFormatRFC3164 SysLogFormat = "rfc3164"
419+
)
420+
421+
// MarshalText implements encoding.TextMarshaler
422+
func (s SysLogFormat) MarshalText() (text []byte, err error) {
423+
return []byte(s), nil
424+
}
425+
426+
// UnmarshalText implements encoding.TextUnmarshaler
427+
func (s *SysLogFormat) UnmarshalText(text []byte) error {
428+
str := string(text)
429+
switch str {
430+
case "rfc5424":
431+
*s = SyslogFormatRFC5424
432+
case "rfc3164":
433+
*s = SyslogFormatRFC3164
434+
default:
435+
return fmt.Errorf("unknown syslog format: %s", str)
436+
}
437+
438+
return nil
439+
}

internal/component/loki/source/syslog/types.go

+14-21
Original file line numberDiff line numberDiff line change
@@ -11,33 +11,26 @@ import (
1111
st "github.com/grafana/alloy/internal/component/loki/source/syslog/internal/syslogtarget"
1212
)
1313

14-
const (
15-
// A modern Syslog RFC
16-
SyslogFormatRFC5424 = "rfc5424"
17-
// A legacy Syslog RFC also known as BSD-syslog
18-
SyslogFormatRFC3164 = "rfc3164"
19-
)
20-
2114
// ListenerConfig defines a syslog listener.
2215
type ListenerConfig struct {
23-
ListenAddress string `alloy:"address,attr"`
24-
ListenProtocol string `alloy:"protocol,attr,optional"`
25-
IdleTimeout time.Duration `alloy:"idle_timeout,attr,optional"`
26-
LabelStructuredData bool `alloy:"label_structured_data,attr,optional"`
27-
Labels map[string]string `alloy:"labels,attr,optional"`
28-
UseIncomingTimestamp bool `alloy:"use_incoming_timestamp,attr,optional"`
29-
UseRFC5424Message bool `alloy:"use_rfc5424_message,attr,optional"`
30-
MaxMessageLength int `alloy:"max_message_length,attr,optional"`
31-
TLSConfig config.TLSConfig `alloy:"tls_config,block,optional"`
32-
SyslogFormat string `alloy:"syslog_format,attr,optional"`
16+
ListenAddress string `alloy:"address,attr"`
17+
ListenProtocol string `alloy:"protocol,attr,optional"`
18+
IdleTimeout time.Duration `alloy:"idle_timeout,attr,optional"`
19+
LabelStructuredData bool `alloy:"label_structured_data,attr,optional"`
20+
Labels map[string]string `alloy:"labels,attr,optional"`
21+
UseIncomingTimestamp bool `alloy:"use_incoming_timestamp,attr,optional"`
22+
UseRFC5424Message bool `alloy:"use_rfc5424_message,attr,optional"`
23+
MaxMessageLength int `alloy:"max_message_length,attr,optional"`
24+
TLSConfig config.TLSConfig `alloy:"tls_config,block,optional"`
25+
SyslogFormat config.SysLogFormat `alloy:"syslog_format,attr,optional"`
3326
}
3427

3528
// DefaultListenerConfig provides the default arguments for a syslog listener.
3629
var DefaultListenerConfig = ListenerConfig{
3730
ListenProtocol: st.DefaultProtocol,
3831
IdleTimeout: st.DefaultIdleTimeout,
3932
MaxMessageLength: st.DefaultMaxMessageLength,
40-
SyslogFormat: SyslogFormatRFC5424,
33+
SyslogFormat: config.SyslogFormatRFC5424,
4134
}
4235

4336
// SetToDefault implements syntax.Defaulter.
@@ -85,11 +78,11 @@ func (sc ListenerConfig) Convert() (*scrapeconfig.SyslogTargetConfig, error) {
8578
}, nil
8679
}
8780

88-
func convertSyslogFormat(format string) (scrapeconfig.SyslogFormat, error) {
81+
func convertSyslogFormat(format config.SysLogFormat) (scrapeconfig.SyslogFormat, error) {
8982
switch format {
90-
case SyslogFormatRFC3164:
83+
case config.SyslogFormatRFC3164:
9184
return scrapeconfig.SyslogFormatRFC3164, nil
92-
case SyslogFormatRFC5424:
85+
case config.SyslogFormatRFC5424:
9386
return scrapeconfig.SyslogFormatRFC5424, nil
9487
default:
9588
return "", fmt.Errorf("unknown syslog format %q", format)

0 commit comments

Comments
 (0)