File source warning: Found line that exceeds max_line_bytes; discarding. internal_log_rate_limit=true

[bmallya]

Hello,
I have an instance of vector where in I am using "File" as a source.

[sources.fileSource]
type = "file"
include = ["/var/log/*","/tmp/logs/**init*"]
read_from = "beginning"

With this configuration I notice the following lines keep coming.

2023-08-18T09:29:34.670059Z  WARN source{component_kind="source" component_id=file_source component_type=file component_name=file_source}:file_server: file_source::buffer: Found line that exceeds max_line_bytes; discarding. internal_log_rate_limit=true

Is there a way to find out which file is generating this error? I tried enabling internal_logs, but that also did not give me more information.

[jszwedko]

Unfortunately not at the moment, that is being tracked by #11657 . It should be a relatively straightforward change; we just haven't been able to get to it yet.

[trahim]

Any plans on doing this @jszwedko anytime soon? We are also trying to track down a long line, don't know if possible but would be great if it could give a line number as well

[jszwedko]

Unfortunately nothing yet, but we'd be happy to see a PR if someone feels motivated.

[bmallya]

Thanks @jszwedko