v3.0.0-M9

This milestone release replaces v3.0.0-M8, which was released from the wrong branch and should not be used.

What's Changed

• Update testcontainers-scala-kafka, ... to 0.40.6 in series/3.x by @scala-steward in #957
• Update kafka-avro-serializer to 7.0.3 in series/3.x by @scala-steward in #959
• Update sbt-header to 5.7.0 in series/3.x by @scala-steward in #958
• Update scala3-library to 3.1.2 in series/3.x by @scala-steward in #961
• Update scala3-library to 3.1.2 by @scala-steward in #949
• Update vulcan to 1.8.3 by @scala-steward in #945
• Update cats-effect-laws, ... to 3.3.11 by @scala-steward in #950
• Update sbt-mima-plugin to 1.1.0 by @scala-steward in #946
• Update sbt-header to 5.7.0 by @scala-steward in #964
• Update kafka-avro-serializer to 7.1.1 in series/3.x by @scala-steward in #974
• Update testcontainers-scala-kafka, ... to 0.40.7 by @scala-steward in #975
• Revert #902 by @bplommer in #976
• Update kafka-avro-serializer to 7.1.2 in series/3.x by @scala-steward in #1006
• Update scala3-library to 3.1.3 by @scala-steward in #999
• Adding sbt-typelevel (second try) by @bplommer in #1012
• Pin kafka-clients to 2.x in series/2.x by @bplommer in #1014
• Update vulcan to 1.8.3 in series/3.x by @scala-steward in #939
• Update testcontainers-scala-kafka, ... to 0.40.8 by @scala-steward in #995
• Update cats-effect-laws, ... to 3.3.13 in series/3.x by @scala-steward in #1008
• Update scala3-library to 3.1.3 in series/3.x by @scala-steward in #1007
• Update cats-effect-laws, ... to 3.3.13 by @scala-steward in #1000
• Ignore cats-effect/fs2 updates in series/3.x, pin kafka-client to 3 by @bplommer in #1015
• Ignore testcontainers updates in 3.x by @bplommer in #1016
• Update kafka-clients to 3.1.1 in series/3.x by @scala-steward in #984
• Update kafka-avro-serializer to 6.2.5 by @scala-steward in #996
• Update sbt-mima-plugin to 1.1.0 in series/3.x by @scala-steward in #940
• Remove ConsumerSettings.unit and ProducerSettings.nothing by @bplommer in #977
• Optimise {Committable,}ProducerRecords apply methods by @bastewart in #978
• Update scala-library to 2.12.16 by @scala-steward in #998
• Fix for #992 A NPE is thrown from AvroDeserializer if the value is empty by @channingwalton in #993
• Add chunk constructors to {Committable,}ProducerRecords by @bastewart in #979
• Add implicit Deserializer attempt by @geirolz in #983
• Fix cancellation behaviour in KafkaAdminClient by @bplommer in #1017
• Remove sbt-ci-release plugin, use sbt-typelevel for latest version by @bplommer in #1018
• Fix release command by @bplommer in #1019
• Update github workflow by @bplommer in #1021
• Removing references to commitBatch and commitBatchOption in Documentation by @ASRagab in #1022
• Refactor future cancellation test to avoid dependence on KakfaFuture internals by @bplommer in #1023
• Merge 2.x into 3.x by @bplommer in #1020

New Contributors

• @bastewart made their first contribution in #978
• @channingwalton made their first contribution in #993
• @geirolz made their first contribution in #983
• @ASRagab made their first contribution in #1022

Full Changelog: v3.0.0-M7...v3.0.0-M9

fs2-kakfa v3.0.0-M8

This is a milestone for the forthcoming v3.0.0, based on the v3.x series of the Java kafka-clients library. It contains the same changes as v2.5.0; for differences from the v2.x series, see release notes for previous milestones.

What's Changed

• Update scala3-library to 3.1.2 by @scala-steward in #949
• Update vulcan to 1.8.3 by @scala-steward in #945
• Update cats-effect-laws, ... to 3.3.11 by @scala-steward in #950
• Update sbt-mima-plugin to 1.1.0 by @scala-steward in #946
• Update sbt-header to 5.7.0 by @scala-steward in #964
• Update testcontainers-scala-kafka, ... to 0.40.7 by @scala-steward in #975
• Revert #902 by @bplommer in #976
• Update scala3-library to 3.1.3 by @scala-steward in #999
• Adding sbt-typelevel (second try) by @bplommer in #1012
• Pin kafka-clients to 2.x in series/2.x by @bplommer in #1014
• Update testcontainers-scala-kafka, ... to 0.40.8 by @scala-steward in #995
• Update cats-effect-laws, ... to 3.3.13 by @scala-steward in #1000
• Update kafka-avro-serializer to 6.2.5 by @scala-steward in #996
• Remove ConsumerSettings.unit and ProducerSettings.nothing by @bplommer in #977
• Optimise {Committable,}ProducerRecords apply methods by @bastewart in #978
• Update scala-library to 2.12.16 by @scala-steward in #998
• Fix for #992 A NPE is thrown from AvroDeserializer if the value is empty by @channingwalton in #993
• Add chunk constructors to {Committable,}ProducerRecords by @bastewart in #979
• Add implicit Deserializer attempt by @geirolz in #983
• Fix cancellation behaviour in KafkaAdminClient by @bplommer in #1017
• Remove sbt-ci-release plugin, use sbt-typelevel for latest version by @bplommer in #1018
• Fix release command by @bplommer in #1019
• Update github workflow by @bplommer in #1021
• Removing references to commitBatch and commitBatchOption in Documentation by @ASRagab in #1022
• Refactor future cancellation test to avoid dependence on KakfaFuture internals by @bplommer in #1023

New Contributors

• @bastewart made their first contribution in #978
• @channingwalton made their first contribution in #993
• @geirolz made their first contribution in #983
• @ASRagab made their first contribution in #1022

Full Changelog: v3.0.0-M7...v3.0.0-M8

v2.5.0

This release is identical to v2.5.0-RC1. It is backwards binary compatible with previous stable releases in the 2.x series, but not with milestones for v2.5.0. It includes several enhancements and bug fixes - for full details see notes for v2.5.0-M2, v2.5.0-M3 and v2.5.0-RC1.

What's Changed

• Move deserialization from KafkaConsumerActor to KafkaConsumer by @bplommer in #902
• withParallelism default 100 -> 10000 in docs by @jbwheatley in #907
• Use IOApp.Simple in docs by @bplommer in #908
• Add produce method to KafkaProducerConnection by @bplommer in #900
• Update cats-effect-laws, ... to 3.3.7 by @scala-steward in #910
• Update logback-classic to 1.2.11 by @scala-steward in #909
• Use semaphore-like permits for some actor operations by @bplommer in #906
• Allow creating settings with placeholder serdes, changing serdes in settings by @bplommer in #911
• Move derived consumer methods to traits by @bplommer in #916
• Update testcontainers version, set image name and version explicitly by @bplommer in #923
• Use permit for fetch, rework permits by @bplommer in #918
• Use a shared test container for each suite by @bplommer in #925
• Update testcontainers-scala-kafka, ... to 0.40.4 by @scala-steward in #928
• Update sbt-mdoc to 2.3.2 by @scala-steward in #929
• Update cats-effect-laws, ... to 3.3.9 by @scala-steward in #930
• Additional consumer functionality exposed by @dmedser in #919
• Fix auto-registration of Avro union (etc) types by @bplommer in #932
• Update sbt-gha, add JDK 17 to pipeline by @bplommer in #927
• Update scala3-library to 3.1.2 by @scala-steward in #949
• Update vulcan to 1.8.3 by @scala-steward in #945
• Update cats-effect-laws, ... to 3.3.11 by @scala-steward in #950
• Update sbt-mima-plugin to 1.1.0 by @scala-steward in #946
• Update sbt-header to 5.7.0 by @scala-steward in #964
• Update testcontainers-scala-kafka, ... to 0.40.7 by @scala-steward in #975
• Revert #902 by @bplommer in #976
• Update scala3-library to 3.1.3 by @scala-steward in #999
• Adding sbt-typelevel (second try) by @bplommer in #1012
• Pin kafka-clients to 2.x in series/2.x by @bplommer in #1014
• Update testcontainers-scala-kafka, ... to 0.40.8 by @scala-steward in #995
• Update cats-effect-laws, ... to 3.3.13 by @scala-steward in #1000
• Update kafka-avro-serializer to 6.2.5 by @scala-steward in #996
• Remove ConsumerSettings.unit and ProducerSettings.nothing by @bplommer in #977
• Optimise {Committable,}ProducerRecords apply methods by @bastewart in #978
• Update scala-library to 2.12.16 by @scala-steward in #998
• Fix for #992 A NPE is thrown from AvroDeserializer if the value is empty by @channingwalton in #993
• Add chunk constructors to {Committable,}ProducerRecords by @bastewart in #979
• Add implicit Deserializer attempt by @geirolz in #983
• Fix cancellation behaviour in KafkaAdminClient by @bplommer in #1017
• Remove sbt-ci-release plugin, use sbt-typelevel for latest version by @bplommer in #1018
• Fix release command by @bplommer in #1019
• Update github workflow by @bplommer in #1021
• Removing references to commitBatch and commitBatchOption in Documentation by @ASRagab in #1022

New Contributors

• @jbwheatley made their first contribution in #907
• @dmedser made their first contribution in #919
• @bastewart made their first contribution in #978
• @channingwalton made their first contribution in #993
• @geirolz made their first contribution in #983
• @ASRagab made their first contribution in #1022

Full Changelog: v2.4.0...v2.5.0

fs2-kafka v2.5.0-RC1

This release contains several enhancements and bug fixes. It also reverts two changes from v2.5.0-M2: see below.

What's Changed

Enhancements

• Optimise {Committable,}ProducerRecords apply methods by @bastewart in #978
• Add chunk constructors to {Committable,}ProducerRecords by @bastewart in #979
• Add implicit Deserializer attempt by @geirolz in #983

Bug fixes

• Fix for #992 A NPE is thrown from AvroDeserializer if the value is empty by @channingwalton in #993
• Fix cancellation behaviour in KafkaAdminClient by @bplommer in #1017

Reverted changes

• Remove ConsumerSettings.unit and ProducerSettings.nothing by @bplommer in #977
  • These were added in #911 to serve as placeholders to use with the newly added withSerializers/withDeserializers method, but the same purpose can be served by using Array[Byte] for key and value types.
• Revert #902 by @bplommer in #976
  • Restores previous behaviour whereby deserialization happens inside KafkaConsumerActor - see #962 for rationale.

Updates

• Update scala3-library to 3.1.2 by @scala-steward in #949
• Update vulcan to 1.8.3 by @scala-steward in #945
• Update cats-effect-laws, ... to 3.3.11 by @scala-steward in #950
• Update sbt-mima-plugin to 1.1.0 by @scala-steward in #946
• Update sbt-header to 5.7.0 by @scala-steward in #964
• Update testcontainers-scala-kafka, ... to 0.40.7 by @scala-steward in #975
• Update scala3-library to 3.1.3 by @scala-steward in #999
• Update testcontainers-scala-kafka, ... to 0.40.8 by @scala-steward in #995
• Update cats-effect-laws, ... to 3.3.13 by @scala-steward in #1000
• Update kafka-avro-serializer to 6.2.5 by @scala-steward in #996
• Update scala-library to 2.12.16 by @scala-steward in #998

CI

• Adding sbt-typelevel (second try) by @bplommer in #1012
• Pin kafka-clients to 2.x in series/2.x by @bplommer in #1014
• Remove sbt-ci-release plugin, use sbt-typelevel for latest version by @bplommer in #1018
• Fix release command by @bplommer in #1019
• Update github workflow by @bplommer in #1021

New Contributors

• @bastewart made their first contribution in #978
• @channingwalton made their first contribution in #993
• @geirolz made their first contribution in #983

Full Changelog: v2.5.0-M3...v2.5.0-RC1

fs2-kafka v3.0.0-M7

This milestone release for fs2-kafka 3.x contains the same improvements as v2.5.0-M3.

fs2-kafka v2.5.0-M3

This milestone release exposes some additional consumer functionality and fixes a long-standing bug/shortcoming in the Vulcan integration module, inherited from the Confluent Kafka Avro serializer, whereby auto-registration of Union and some other schemas would not work as expected.

What's Changed

Enhancements

• Additional consumer functionality exposed by @dmedser in #919
• Fix auto-registration of Avro union (etc) types by @bplommer in #932

Internals and updates

• Move derived consumer methods to traits by @bplommer in #916
• Update testcontainers version, set image name and version explicitly by @bplommer in #923
• Use permit for fetch, rework permits by @bplommer in #918
• Use a shared test container for each suite by @bplommer in #925
• Update testcontainers-scala-kafka, ... to 0.40.4 by @scala-steward in #928
• Update sbt-mdoc to 2.3.2 by @scala-steward in #929
• Update cats-effect-laws, ... to 3.3.9 by @scala-steward in #930
• Update sbt-gha, add JDK 17 to pipeline by @bplommer in #927

New Contributors

• @dmedser made their first contribution in #919

Full Changelog: 2.5.0-M1...v2.5.0-M3

fs2-kafka v3.0.0-M6

This milestone makes the following improvements to the API, in addition to the changes in v2.5.0-M1. As always, feedback is welcome!

What's Changed

• Introduce type-level distinction between key and value serializers/deserializers by @bplommer in #898
• Simplify ProducerRecords types by removing passthrough by @bplommer in #901
  • The passthrough functionality was orthogonal to the producer functionality - if needed it should instead be replicated in user code.

Full Changelog: v3.0.0-M5...v3.0.0-M6

fs2-kafka v2.5.0-M2

This release is identical to v2.5.0-M1, which failed to publish.

This release adds some new features and significantly simplifies the internals of the consumer. Although we don't expect these changes to cause any problems, we're releasing this as a milestone to allow for some extra battle testing before the next stable release.

Note: there is a small change to the behaviour of consumers when using partitionedStream or partitionsMapStream. Previously, failure to deserialize a record would cause the entire consumer to fail; now, it will only cause a single partition stream to fail.

What's Changed

New features

• Add produce method to KafkaProducerConnection by @bplommer in #900
  • KafkaProducerConnection allocates a producer without being tied to particular serializers, allowing creation of multiple KafkaProducer instances that share a single set of connections to the broker. With this change, it is possible to produce records directly from KafkaProducerConnection by providing serializers when calling produce.
• Allow creating settings with placeholder serdes, changing serdes in settings by @bplommer in #911
  • This allows creation of ProducerSettings and ConsumerSettings to be decoupled from provision of serializers and deserializers, which is particularly useful when creating multiple producers or consumers with different types.

Docs improvements

• withParallelism default 100 -> 10000 in docs by @jbwheatley in #907
• Use IOApp.Simple in docs by @bplommer in #908

Simplification of internals

• Move deserialization from KafkaConsumerActor to KafkaConsumer by @bplommer in #902
• Use semaphore-like permits for some actor operations by @bplommer in #906

Dependency updates

• Update cats-effect-laws, ... to 3.3.7 by @scala-steward in #910
• Update logback-classic to 1.2.11 by @scala-steward in #909

New Contributors

• @jbwheatley made their first contribution in #907

Full Changelog: v2.4.0...2.5.0-M1

fs2-kafka v2.5.0-M1

This release adds some new features and significantly simplifies the internals of the consumer. Although we don't expect these changes to cause any problems, we're releasing this as a milestone to allow for some extra battle testing before the next stable release.

Note: there is a small change to the behaviour of consumers when using partitionedStream or partitionsMapStream. Previously, failure to deserialize a record would cause the entire consumer to fail; now, it will only cause a single partition stream to fail.

What's Changed

New features

• Add produce method to KafkaProducerConnection by @bplommer in #900
  • KafkaProducerConnection allocates a producer without being tied to particular serializers, allowing creation of multiple KafkaProducer instances that share a single set of connections to the broker. With this change, it is possible to produce records directly from KafkaProducerConnection by providing serializers when calling produce.
• Allow creating settings with placeholder serdes, changing serdes in settings by @bplommer in #911
  • This allows creation of ProducerSettings and ConsumerSettings to be decoupled from provision of serializers and deserializers, which is particularly useful when creating multiple producers or consumers with different types.

Docs improvements

• withParallelism default 100 -> 10000 in docs by @jbwheatley in #907
• Use IOApp.Simple in docs by @bplommer in #908

Simplification of internals

• Move deserialization from KafkaConsumerActor to KafkaConsumer by @bplommer in #902
• Use semaphore-like permits for some actor operations by @bplommer in #906

Dependency updates

• Update cats-effect-laws, ... to 3.3.7 by @scala-steward in #910
• Update logback-classic to 1.2.11 by @scala-steward in #909

New Contributors

• @jbwheatley made their first contribution in #907

Full Changelog: v2.4.0...2.5.0-M1

fs2-kafka v3.0.0-M5

This is the latest milestone release for the fs2-kafka v3.x series, which uses Kafka 3. It updates the kafka-clients library to 3.1 and includes the same changes as v2.4.0.