Pulsar support and MQTT relaying

• Added support for Apache Pulsar
• You can now relay MQTT messages to a Batch.sh collection

Support for multiple kafka brokers

• Kafka read/write: multiple kafka brokers can now be specified by repeating the --address flag for each broker
• Kafka relay: multiple kafka brokers can now be specified by separating them by a comma. Example: PLUMBER_RELAY_KAFKA_ADDRESS="broker1.domain.com:9092,broker2.domain.com:9092"

Adding --include-full-document option for cdc-mongo

Merge pull request #107 from batchcorp/blinktag/include_full_doc

Adding --include-full-document option for cdc-mongo

Fixes for Postgres CDC flags

Merge pull request #103 from batchcorp/blinktag/postgres_cdc_flag_fixes

Flag fixes for CDC Postgres

Improvements to Postgres and Mongo CDC

Improvements:

• A new record is shipped for every change for cdc-postgres, instead of grouping by commit
• Reduced cdc-postgres code deduplication

Fixes:

• Adding missing stats.Incr() calls for cdc-postgres and cdc-mongo
• You may now listen for changes to your entire mongo deployment by omitting the database and collection flags

Minor fix for cdc-postgres flags

v0.22.1

Requiring posgres hostname, username, and database name

PostgreSQL and MongoDB Change Data Capture

This release adds two new relay commands: cdc-postgres and cdc-mongo to allow relaying of CDC data from Postgres' logical replication streams and MongoDB's change stream events.

See https://docs.batch.sh/event-ingestion/change-data-capture/mongodb and https://docs.batch.sh/event-ingestion/change-data-capture/postgresql for documentation on how to stream these events into a Batch.sh collection

v0.21.2

Fixing APK command

Additional flags for nats-streaming

--all - Delivers all available messages
--since - Delivers messages from an interval ago. Ex: 1m, 24h
--last - Delivers last received message
--seq - Delivers message from a given sequence ID

Azure Event Hub,Nats streaming support and SQS FIFO fix

• plumber read azure-eventhub and plumber write azure-eventhub to publish and consume from an azure event hub
• plumber read nats-streaming and plumber write nats-streaming to publish and consume from Nats jetstream
• Added --message-group-id' and --message-deduplication-idflags toplumber write aws-sqs` to support FIFO queus