out_kinesis_streams: integrate with shared compression lib

[reegnz]

Support for gzip compression. Basically copied the approach used in out_kinesis_firehose.

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• Example configuration file for the change
• Debug log output from testing the change

• Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• Run local packaging test showing all targets (including any new ones) build.
• Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

• Documentation required for this feature

Backporting

• Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[reegnz]

I basically copied what got implemented in out_kinesis_firehose, but I'm sort of a C noob, so I'd appreciate some help in testing (can probably do it myself given enough time but other people are more savvy with this stuff than me).

[reegnz]

I want to migrate away from fluentd where this feature was present, and it's a pretty big blocker for me right now.

Original PR was #4371 where I'm backporting the compression implementation from.

@PettitWesley maybe I can get you to help out, given that you were the original implementor of the out_kinesis_stream feature?

[reegnz]

Opened a ticket that covers this issue: #9451

[github-actions]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[reegnz]

Still live. Need feedback from someone competent with the topic.

[reegnz]

@cosmo0920 I see you've been involved in a kinesis_output PR, maybe you could have a look?

[github-actions]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[reegnz]

Although stale, this is still in demand.

[cosmo0920]

It could be useful for reducing bandwidth of requesting to AWS.
Do you have an ETA for finishing up to mark as ready for review?

[reegnz]

Let me quote myself:

but I'm sort of a C noob, so I'd appreciate some help in testing

[github-actions]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.