[fs2-kafka; zio-kafka] Using `KafkaConsumerTracer.traceConsumerStream` causes some commits to fail

[soujiro32167]

Repro:

kafkaConsumer
      .subscribeTo("topic") *>
      kafkaConsumer.partitionedStream
        .map(s =>
          KafkaConsumerTracer
            .traceConsumerStream(ztracer, s)
            .evalMapChunkWithTracer(ztracer, "handler") {
              case CommittableConsumerRecord(_, offset) =>
                ZIO.succeed(offset)
            }
            .endTracingEachElement
        )
        .parJoinUnbounded
        .through(commitBatchWithin(500, 5.seconds))
        .compile
        .drain

Commits don't get recorded

[calvinlfer]

Hey @soujiro32167
Is it that commits fail to get committed by Kafka or is it that commits don’t show up in the traces/spans?

[soujiro32167]

Under some conditions, the commit itself fails. What you'd see is events being consumed, and then at some point there is a lag that won't go away, even though all offsets have been processed

[calvinlfer]

This was a problem with FS2 Kafka where this problem would appear (without tracing) and moving to the higher level consumeChunk API fixed it

[soujiro32167]

It may be, yeah. consumeChunk resolved the issue