max_send_limit_bytes setting is not working as expected

[pratikshavyas]

Describe the bug

As per readme file it is mentioned that: max_send_limit_bytes - default: nil - Max byte size to send message to avoid MessageSizeTooLarge. For example, if you set 1000000(message.max.bytes in kafka), Message more than 1000000 byes will be dropped.
However, I see that even if I have set max_send_limit_bytes as 5 MB, there are requests being sent to kafka for 20MB and 16 MB.
Why is fluentd not able to limit the message size? Is there anything else needs to be configured here?
Please suggest asap.

To Reproduce

configuration for the plugin:

<match *>
  @type copy
  <store ignore_error>
    <buffer>
      @type file
      @log_level error
      path /logskafka
      timekey 1d
      flush_thread_count 4
      chunk_limit_size 2MB
      overflow_action drop_oldest_chunk
      flush_mode interval
      flush_interval 5s
      total_limit_size 2GB
      max_send_limit_bytes 5000000
    </buffer>
    @type kafka2
    @log_level error
    brokers service:8097
    topic_key topic
    default_topic messages
  </store>
</match>

Expected behavior

Fluentd should limit the message being sent to kafka to 5 MB as max_send_limit_bytes is set to 5 MB

Your Environment

- Fluentd version: 1.17.0
- fluent-plugin-kafka version: 0.19.2

Your Configuration

configuration for the plugin:

<match *>
  @type copy
  <store ignore_error>
    <buffer>
      @type file
      @log_level error
      path /logskafka
      timekey 1d
      flush_thread_count 4
      chunk_limit_size 2MB
      overflow_action drop_oldest_chunk
      flush_mode interval
      flush_interval 5s
      total_limit_size 2GB
      max_send_limit_bytes 5000000
    </buffer>
    @type kafka2
    @log_level error
    brokers service:8097
    topic_key topic
    default_topic messages
  </store>
</match>

Your Error Log

No logs

Additional context

No response

[pratikshavyas]

#323 (comment) As per this comment, it looks like that documentation does not have right information and it needs an update. This comment is also a bit ambiguous, Kindly check on this and provide an explanation on what all configurations have an impact and can be configured to avoid MessageSizeTooLarge exception.

[Watson1978]

I think you should set a value to out plugin instead of buffer.

<match *>
  @type copy
  <store ignore_error>
    <buffer>
      @type file
      @log_level error
      path /logskafka
      timekey 1d
      flush_thread_count 4
      chunk_limit_size 2MB
      overflow_action drop_oldest_chunk
      flush_mode interval
      flush_interval 5s
      total_limit_size 2GB
      # max_send_limit_bytes 5000000
    </buffer>

    @type kafka2
    @log_level error
    brokers service:8097
    topic_key topic
    default_topic messages

    max_send_limit_bytes 5000000
  </store>
</match>

[github-actions]

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

[github-actions]

This issue was automatically closed because of stale in 30 days