Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for file plugin to write to file path based on a property #10200

Open
BHoffV opened this issue Apr 10, 2025 · 0 comments
Open

Add support for file plugin to write to file path based on a property #10200

BHoffV opened this issue Apr 10, 2025 · 0 comments

Comments

@BHoffV
Copy link

BHoffV commented Apr 10, 2025

Is your feature request related to a problem? Please describe.
We use the file output plugin to write to files based on their original file path. However due to limitations of the file output plugin we have to modify the tag to be the file path we want to write to. This is causes a few issues, one of which is not being able to use the tag for determining which filters to apply and another due to buffer concerns with the rewrite tag plugin.

Describe the solution you'd like
We would like the file output plugin to have a parameter that specifies the log entry property to use to determine the output path. If that variable is not specified to fallback to using the tag.

Example:
I have a log entry that has been parsed and the path has been parsed as to be like this:
TheTag: [1488543156, {"path"=>"/path/to/file.log", "log"=> "This is a log message"}]

We would like us to specify that the variable path be used as the destination file.

Describe alternatives you've considered
The issues described in additional context were not resolvable so we ended up creating a custom golang file output plugin that does the functionality that we require. This is acceptable in the short term but we'd greatly prefer to stick to baseline plugins.

Additional context
So our system is reading logs and sending it via forward to another destination for writing to a file:
Tail Input -> Filters -> Forward Output ------> Forward Input -> File Output

However due to the requirements of the file plugin we had to insert the Rewrite Tag filter into the pipeline to set the tag equal to the property of the path that we had already parsed. The Rewrite Tag plugin was the biggest issue. We haven't tested this in awhile so its possible this is something that has been looked at but the way the system worked was that if you capped the memory of the Rewrite Tag emitter then the Rewrite Tag paused, the log entry skipped over the rewrite tag filter and hit the output plugin and sent to the forward without being modified and not having a valid path tag. But if we disabled the buffer cap then in times of network issues the buffer grew massively and caused our file buffer to fill up and ultimately led to crashes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant