README.md

Awesome Kafka in Clojure

Kafka and the Confluent Platform are awesome in Clojure.

• Awesome Kafka in Clojure
  • Architecture
  • Kafka Producers & Consumers
  • Kafka Connect
  • Kafka Streams
  • KSQL
  • Avro & Confluent Schema Registry
  • Streaming Platforms
  • Management Tools
  • Testing Tools
  • Presentations

Architecture

• The Log: What every software engineer should know about real-time data's unifying abstraction
• Confluent Blog Series on Data and Services
• The Commander Pattern - Presentation
• Loving a Log Oriented Architecture
• Designing Event Driven Systems - Free Book from Confluent

Kafka Producers & Consumers

• FundingCircle/jackdaw
• ymilky/franzy
• ccann/gregor
• pyr/kinsky
• cognitect-labs/pedestal.kafka
• dvlopt/milena

Kafka Connect

• yanatan16/kafka-connect-slack-sink
  Example connector written in Clojure.

Kafka Streams

Projects:

• FundingCircle/jackdaw
• bobby/kafka-streams-clojure
• capitalone/cqrs-manager-for-distributed-reactive-services
• dvlopt/milena
• cddr/ksml

Info:

• Confluent Kafka Streams Docs
• Proof of Concept using Kafka Streams and KTables

KSQL

• Nothing yet!

Avro & Confluent Schema Registry

Avro and Confluent Schema Registry can be difficult to navigate, especially for beginners, so this section includes more description.

Listed Avro libraries must support complex and nested Avro types.

• FundingCircle/jackdaw
• damballa/abracad
  A Clojure library for de/serializing Clojure data structures with Avro. The incumbent Avro library for Clojure. No support for GenericRecords or Confluent Schema Registry.
• ymilky/franzy-avro
  Add-on to Franzy. Uses Abracad under the hood. (De)Serializes to Avro bytes. No support for GenericRecords or Schema Registry.
• ovotech/kafka-avro-confluent
  Serializes to byte-array but supports Confluent AvroSerdes via custom byte assembly. Offers a Schema Registry client.
• konukhov/kfk-avro-bridge
  Provides conversion of Clojure data structures to/from Avro GenericRecords that are compatible with the built-in Confluent serializers. Use with another library, like Abracad, to generate the Avro schema used for conversion.
• farbetter/roe
  An Apache Avro library for Clojure and ClojureScript.
• neilwashere
  Avro schema validation errors can be hard to debug. This lib provides some helpful tools.
• pfeodrippe/spec-to-avro
  A work in progress to convert Clojure Specs to Avro. An incomplete work in progress but an interesting idea.
• piotr-yuxuan/dove Infer specs from any Avro type. Think about avro->spec. Useful for generative testing or pre-serialization validation.
• deercreeklabs/lancaster
  An Apache Avro DSL and helper library

Streaming Platforms

• onyx-platform/onyx

Management Tools

• ymilky/franzy-admin
• otto-de/clj-kafka-utils
• dvlopt/milena

Testing Tools

• confluent-inc
• ymilky/franzy-embedded
• ymilky/franzy-mocks
• ymilky/travel-zoo

Presentations

These presentations mustn't be exclusively on Kafka but should relate to Clojure's synergy with the distributed log.

• Building a Data Pipeline with Kafka and Clojure - David Pick
• One Million Clicks per Minute with Kafka and Clojure - Devon Peticolas
• Tame Kafka with Clojure - Andrea Crotti
• The Database as a Value - Rich Hickey
• The Language of the System - Rich Hickey

License