Ballerina CDC Connector

The Ballerina Change Data Capture (CDC) module provides APIs to capture and process database change events in real-time. This module enables developers to define services that handle change capture events such as inserts, updates, and deletes. It is built on top of the Debezium framework and supports popular databases like MySQL and Microsoft SQL Server.

With the CDC module, you can:

• Capture real-time changes from databases.
• Process and react to database events programmatically.
• Build event-driven applications with ease.

Quickstart

Step 1: Import Required Modules

Add the following imports to your Ballerina program:

• ballerinax/cdc: Core module that provides APIs to capture and process database change events.
• ballerinax/mysql: Provides MySQL-specific listener and types for CDC. Replace with the corresponding module for your database if needed.
• ballerinax/mysql.cdc.driver as _: Debezium-based driver for MySQL CDC. Use the appropriate driver for your database (e.g., mssql.cdc.driver, postgresql.cdc.driver, or oracledb.cdc.driver).

import ballerinax/cdc;
import ballerinax/mysql;
import ballerinax/mysql.cdc.driver as _;

Step 2: Configure the CDC Listener

Create a CDC listener for your MySQL database by specifying the connection details:

listener mysql:CdcListener mysqlListener = new ({
    hostname: "localhost",
    port: 3306,
    username: "username",
    password: "password",
    databaseInclude: ["inventory"]
});

Step 3: Define the CDC Service

Implement a cdc:Service to handle database change events:

service on mysqlListener {

    remote function onRead(record {} after) returns error? {
        // Handle the read event
        log:printInfo(`Record read: ${after}`);
    }

    remote function onCreate(record {} after) returns error? {
        // Handle the create event
        log:printInfo(`Record created: ${after}`);
    }

    remote function onUpdate(record {} before, record {} after) returns error? {
        // Handle the update event
        log:printInfo(`Record updated from: ${before}, to ${after}`);
    }

    remote function onDelete(record {} before) returns error? {
        // Handle the delete event
        log:printInfo(`Record deleted: ${before}`);
    }
}

Step 4: Run the Application

Run your Ballerina application:

bal run

Examples

The cdc module provides practical examples illustrating its usage in various real-world scenarios. Explore these examples to understand how to capture and process database change events effectively.

1. Fraud Detection - Detect suspicious transactions in a financial database and send fraud alerts via email. This example showcases how to integrate the CDC module with the Gmail connector to notify stakeholders of potential fraud.

2. Cache Management - Synchronize a Redis cache with changes in a MySQL database. It listens to changes in the products, vendors, and product_reviews tables and updates the Redis cache accordingly.

Issues and projects

The Issues and Projects tabs are disabled for this repository as this is part of the Ballerina library. To report bugs, request new features, start new discussions, view project boards, etc., visit the Ballerina library parent repository.

This repository only contains the source code for the package.

Build from the source

Prerequisites

1. Download and install Java SE Development Kit (JDK) version 21. You can download it from either of the following sources:

   • Oracle JDK
   • OpenJDK

   Note: After installation, remember to set the JAVA_HOME environment variable to the directory where JDK was installed.

2. Download and install Docker.

   Note: Ensure that the Docker daemon is running before executing any tests.

3. Export your GitHub personal access token with read package permissions as follows.

    export packageUser=<Username>
    export packagePAT=<Personal access token>
   

Build options

Execute the commands below to build from the source.

1. To build the package:

   ./gradlew clean build

2. To run the tests:

   ./gradlew clean test

3. To build the without the tests:

   ./gradlew clean build -x test

4. To debug package with a remote debugger:

   ./gradlew clean build -Pdebug=<port>

5. To debug with the Ballerina language:

   ./gradlew clean build -PbalJavaDebug=<port>

6. Publish the generated artifacts to the local Ballerina Central repository:

   ./gradlew clean build -PpublishToLocalCentral=true

7. Publish the generated artifacts to the Ballerina Central repository:

   ./gradlew clean build -PpublishToCentral=true

Contribute to Ballerina

As an open-source project, Ballerina welcomes contributions from the community.

For more information, go to the contribution guidelines.

Code of conduct

All the contributors are encouraged to read the Ballerina Code of Conduct.

Useful links

• For more information go to the cdc package.
• For example demonstrations of the usage, go to Ballerina By Examples.
• Chat live with us via our Discord server.
• Post all technical questions on Stack Overflow with the #ballerina tag.