Skip to content
/ distributed-messaging-system Public

a Kafka-like distributed message broker with replication, partitioning, and fault-tolerant messaging for scalable communication

0 stars 0 forks Branches Tags Activity
Star
Notifications You must be signed in to change notification settings

jeevan10017/distributed-messaging-system

BranchesTags

Repository files navigation

Kafka like distributed-messaging-system

A lightweight, from-scratch implementation of a Kafka broker in C++. This project handles core Kafka networking and API requests, supporting multiple sequential and concurrent client connections through a multi-threaded architecture.

About The Project

This project is a deep dive into network programming and protocol implementation. Built as an educational exercise, it demystifies the inner workings of a distributed messaging system like Apache Kafka by implementing a portion of its binary protocol from the ground up, without relying on any external Kafka libraries.

The primary goal is not to create a production-ready broker, but to serve as a hands-on learning tool for understanding low-level TCP server design, concurrency, and byte-level data manipulation.

Features

  • Multi-Threaded TCP Server: Built using POSIX sockets, the server can handle multiple client connections concurrently, assigning each client to its own dedicated thread.

  • Sequential Request Handling: A single client can maintain a connection and send multiple requests in sequence.

  • Kafka Protocol Implementation: Manual serialization and deserialization of Kafka's binary protocol, including:

    • Handling Big-Endian byte order (htons, htonl)
    • Parsing complex request headers with variable-length fields
    • Building byte-perfect responses according to the protocol specification
    • Differentiating between standard and "compact" string/array formats

  • API Implementation:

    • ApiVersions (Key 18): Responds with the broker's supported API versions. Includes logic to handle requests for both supported and unsupported versions.
    • DescribeTopicPartitions (Key 75): Responds to requests for topic information. For this simulation, it currently treats all topics as unknown and returns the appropriate UNKNOWN_TOPIC_OR_PARTITION error code and response structure.

Getting Started

To get a local copy up and running, follow these simple steps.

Prerequisites

You will need a C++ compiler, CMake, and Make installed on your system.

  • g++ / Clang
  • CMake >= 3.10
  • Make

Building the Project

  1. Clone the repository:

    git clone https://github.com/jeevan10017/distributed-messaging-system

  2. Navigate to the project directory:

    cd distributed-messaging-system

  3. Create a build directory and run CMake:

    mkdir build
cd build
cmake ..

  4. Compile the source code:

    make

Usage

After a successful build, the executable will be located in the build directory.

To start the server, run:

./build/kafka

The server will start and listen for client connections on port 9092.

Implementation Highlights

Concurrency Model: Thread-Per-Client

The server uses a classic and robust "thread-per-client" concurrency model:

  1. The main function initializes the server socket and enters an infinite loop, blocking on the accept() call
  2. When a new client connects, accept() returns a unique socket descriptor for that client
  3. A new std::thread is immediately created, and the client-handling logic is passed to this thread as its main function
  4. The main thread detach()es the new client thread, allowing it to run independently in the background
  5. The main thread immediately loops back to accept(), ready to handle the next incoming connection without delay

This design provides excellent isolation between clients and simplifies the logic within the client handler, as it doesn't need to worry about other concurrent connections.

Low-Level Protocol Handling

All Kafka requests and responses are handled at the byte level. The server manually constructs response buffers by:

  • Calculating the exact message size based on the contents
  • Writing multi-byte integers in Big-Endian (network) byte order
  • Correctly serializing different data structures as defined by the Kafka protocol, such as standard strings, compact arrays, and tagged fields
  • Parsing incoming requests by carefully navigating byte offsets and interpreting variable-length fields to extract necessary information like API keys and topic names

Test Results

The implementation has been thoroughly tested and passes all required stages:

remote: [compile] -- Running vcpkg install
remote: [compile] All requested packages are currently installed.
remote: [compile] All requested installations completed successfully in: 72.2 us
remote: [compile] -- Running vcpkg install - done
remote: [compile] -- The C compiler identification is GNU 14.2.0
remote: [compile] -- The CXX compiler identification is GNU 14.2.0
remote: [compile] -- Detecting C compiler ABI info
remote: [compile] -- Detecting C compiler ABI info - done
remote: [compile] -- Check for working C compiler: /usr/bin/cc - skipped
remote: [compile] -- Detecting C compile features
remote: [compile] -- Detecting C compile features - done
remote: [compile] -- Detecting CXX compiler ABI info
remote: [compile] -- Detecting CXX compiler ABI info - done
remote: [compile] -- Check for working CXX compiler: /usr/local/bin/c++ - skipped
remote: [compile] -- Detecting CXX compile features
remote: [compile] -- Detecting CXX compile features - done
remote: [compile] -- Configuring done (5.0s)
remote: [compile] -- Generating done (0.0s)
remote: [compile] -- Build files have been written to: /app/build
remote: [compile] [ 50%] Building CXX object CMakeFiles/kafka.dir/src/main.cpp.o
remote: [compile] [100%] Linking CXX executable kafka
remote: [compile] [100%] Built target kafka
remote: [compile] Moved ./.codecrafters/run.sh → ./your_program.sh
remote: [compile] Compilation successful.
remote:
remote: [tester::#FD8] Running tests for Stage #FD8 (Consuming Messages - Fetch multiple messages from disk)
remote: [tester::#FD8] [Serializer] Finished writing log files to: /tmp/kraft-combined-logs
remote: [tester::#FD8] $ ./your_program.sh /tmp/server.properties
remote: [your_program] Logs from your program will appear here!
remote: [your_program] Waiting for a client to connect...
remote: [tester::#FD8] Sending "Fetch" (version: 16) request (Correlation id: 1428513231)
remote: [tester::#FD8] ✓ Correlation ID: 1428513231
remote: [tester::#FD8] ✓ Throttle Time: 0
remote: [tester::#FD8] ✓ Error Code: 0 (NO_ERROR)
remote: [tester::#FD8]   ✓ TopicResponse[0] Topic UUID: 00000000-0000-4000-8000-000000000060
remote: [tester::#FD8]     ✓ PartitionResponse[0] Error code: 0 (NO_ERROR)
remote: [tester::#FD8]     ✓ PartitionResponse[0] Partition Index: 0
remote: [tester::#FD8]       ✓ RecordBatch[0] BaseOffset: 0
remote: [tester::#FD8]         ✓ Record[0] Value: Hello CodeCrafters!
remote: [tester::#FD8]       ✓ RecordBatch[1] BaseOffset: 1
remote: [tester::#FD8]         ✓ Record[0] Value: Hello Universe!
remote: [tester::#FD8] ✓ RecordBatch bytes match with the contents on disk
remote: [tester::#FD8] Test passed.
remote: [your_program] Client connected
remote: [your_program] Idx       | Hex                                             | ASCII
remote: [your_program] ----------+-------------------------------------------------+-----------------
remote: [your_program] 00000000  00 00 00 00 00 00 00 00  00 00 00 4b 00 00 00 00  |...........K....|
remote: [your_program] 00000010  02 55 60 53 93 00 00 00  00 00 00 00 00 01 91 e0  |.U`S............|
remote: [your_program] 00000020  5b 6d 8b 00 00 01 91 e0  5b 6d 8b 00 00 00 00 00  |[m......[m......|
remote: [your_program] 00000030  00 00 00 00 00 00 00 00  00 00 00 00 01 32 00 00  |.............2..|
remote: [your_program] 00000040  00 01 26 48 65 6c 6c 6f  20 43 6f 64 65 43 72 61  |..&Hello CodeCra|
remote: [your_program] 00000050  66 74 65 72 73 21 00 00  00 00 00 00 00 00 01 00  |fters!..........|
remote: [your_program] 00000060  00 00 47 00 00 00 00 02  b8 7c 9c ff 00 00 00 00  |..G......|......|
remote: [your_program] 00000070  00 00 00 00 01 91 e0 5b  6d 8b 00 00 01 91 e0 5b  |.......[m......[|
remote: [your_program] 00000080  6d 8b 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |m...............|
remote: [your_program] 00000090  00 00 00 01 2a 00 00 00  01 1e 48 65 6c 6c 6f 20  |....*.....Hello |
remote: [your_program] 000000a0  55 6e 69 76 65 72 73 65  21 00                    |Universe!.|
remote:
remote: [tester::#EG2] Running tests for Stage #EG2 (Consuming Messages - Fetch single message from disk)
remote: [tester::#EG2] [Serializer] Finished writing log files to: /tmp/kraft-combined-logs
remote: [tester::#EG2] $ ./your_program.sh /tmp/server.properties
remote: [your_program] Logs from your program will appear here!
remote: [your_program] Waiting for a client to connect...
remote: [tester::#EG2] Sending "Fetch" (version: 16) request (Correlation id: 686742283)
remote: [tester::#EG2] ✓ Correlation ID: 686742283
remote: [tester::#EG2] ✓ Throttle Time: 0
remote: [tester::#EG2] ✓ Error Code: 0 (NO_ERROR)
remote: [tester::#EG2]   ✓ TopicResponse[0] Topic UUID: 00000000-0000-4000-8000-000000000091
remote: [tester::#EG2]     ✓ PartitionResponse[0] Error code: 0 (NO_ERROR)
remote: [tester::#EG2]     ✓ PartitionResponse[0] Partition Index: 0
remote: [tester::#EG2]       ✓ RecordBatch[0] BaseOffset: 0
remote: [tester::#EG2]         ✓ Record[0] Value: Hello Earth!
remote: [tester::#EG2] ✓ RecordBatch bytes match with the contents on disk
remote: [tester::#EG2] Test passed.
remote:
remote: [tester::#CM4] Running tests for Stage #CM4 (Consuming Messages - Fetch with an empty topic)
remote: [tester::#CM4] [Serializer] Finished writing log files to: /tmp/kraft-combined-logs
remote: [tester::#CM4] $ ./your_program.sh /tmp/server.properties
remote: [your_program] Waiting for a client to connect...
remote: [your_program] Logs from your program will appear here!
remote: [tester::#CM4] Sending "Fetch" (version: 16) request (Correlation id: 1944274147)
remote: [tester::#CM4] ✓ Correlation ID: 1944274147
remote: [tester::#CM4] ✓ Throttle Time: 0
remote: [tester::#CM4] ✓ Error Code: 0 (NO_ERROR)
remote: [tester::#CM4]   ✓ TopicResponse[0] Topic UUID: 00000000-0000-4000-8000-000000000028
remote: [tester::#CM4]     ✓ PartitionResponse[0] Error code: 0 (NO_ERROR)
remote: [tester::#CM4]     ✓ PartitionResponse[0] Partition Index: 0
remote: [tester::#CM4]     ✓ RecordBatches: []
remote: [tester::#CM4] Test passed.
remote: [your_program] Client connected
remote: [your_program] Idx       | Hex                                             | ASCII
remote: [your_program] ----------+-------------------------------------------------+-----------------
remote:
remote: [tester::#HN6] Running tests for Stage #HN6 (Consuming Messages - Fetch with an unknown topic)
remote: [tester::#HN6] [Serializer] Finished writing log files to: /tmp/kraft-combined-logs
remote: [tester::#HN6] $ ./your_program.sh /tmp/server.properties
remote: [your_program] Waiting for a client to connect...
remote: [your_program] Logs from your program will appear here!
remote: [tester::#HN6] Sending "Fetch" (version: 16) request (Correlation id: 679618191)
remote: [tester::#HN6] ✓ Correlation ID: 679618191
remote: [tester::#HN6] ✓ Throttle Time: 0
remote: [tester::#HN6] ✓ Error Code: 0 (NO_ERROR)
remote: [tester::#HN6]   ✓ TopicResponse[0] Topic UUID: 00000000-0000-0000-0000-000000008640
remote: [tester::#HN6]     ✓ PartitionResponse[0] Error code: 100 (UNKNOWN_TOPIC_ID)
remote: [tester::#HN6]     ✓ PartitionResponse[0] Partition Index: 0
remote: [tester::#HN6]     ✓ RecordBatches: []
remote: [tester::#HN6] Test passed.
remote:
remote: [tester::#DH6] Running tests for Stage #DH6 (Consuming Messages - Fetch with no topics)
remote: [tester::#DH6] [Serializer] Finished writing log files to: /tmp/kraft-combined-logs
remote: [tester::#DH6] $ ./your_program.sh /tmp/server.properties
remote: [your_program] Waiting for a client to connect...
remote: [your_program] Logs from your program will appear here!
remote: [tester::#DH6] Sending "Fetch" (version: 16) request (Correlation id: 866864631)
remote: [tester::#DH6] ✓ Correlation ID: 866864631
remote: [tester::#DH6] ✓ Throttle Time: 0
remote: [tester::#DH6] ✓ Error Code: 0 (NO_ERROR)
remote: [tester::#DH6]   ✓ TopicResponses: []
remote: [tester::#DH6] Test passed.
remote:
remote: [tester::#GS0] Running tests for Stage #GS0 (Consuming Messages - Include Fetch in APIVersions)
remote: [tester::#GS0] $ ./your_program.sh /tmp/server.properties
remote: [tester::#GS0] Waiting for a client to connect...
remote: [tester::#GS0] Logs from your program will appear here!
remote: [tester::#GS0] Sending "ApiVersions" (version: 4) request (Correlation id: 704311803)
remote: [tester::#GS0] ✓ Correlation ID: 704311803
remote: [tester::#GS0] ✓ Error code: 0 (NO_ERROR)
remote: [tester::#GS0] ✓ API keys array length: 3
remote: [tester::#GS0] ✓ MinVersion for FETCH is <= 16 & >= 0
remote: [tester::#GS0] ✓ MaxVersion for FETCH is >= 16
remote: [tester::#GS0] ✓ MinVersion for API_VERSIONS is <= 4 & >= 0
remote: [tester::#GS0] ✓ MaxVersion for API_VERSIONS is >= 4
remote: [tester::#GS0] Test passed.
remote:
remote: [tester::#WQ2] Running tests for Stage #WQ2 (Listing Partitions - List for multiple topics)
remote: [tester::#WQ2] [Serializer] Finished writing log files to: /tmp/kraft-combined-logs
remote: [tester::#WQ2] $ ./your_program.sh /tmp/server.properties
remote: [your_program] Logs from your program will appear here!
remote: [your_program] Waiting for a client to connect...
remote: [tester::#WQ2] Sending "DescribeTopicPartitions" (version: 0) request (Correlation id: 736216611)
remote: [tester::#WQ2] ✓ Correlation ID: 736216611
remote: [tester::#WQ2] ✓ Throttle Time: 0
remote: [tester::#WQ2]   ✓ TopicResponse[0] Error code: 0
remote: [tester::#WQ2]   ✓ TopicResponse[0] Topic Name: foo
remote: [tester::#WQ2]   ✓ TopicResponse[0] Topic UUID: 00000000-0000-4000-8000-000000000091
remote: [tester::#WQ2]     ✓ PartitionResponse[0] Error code: 0
remote: [tester::#WQ2]     ✓ PartitionResponse[0] Partition Index: 0
remote: [tester::#WQ2]   ✓ TopicResponse[1] Error code: 0
remote: [tester::#WQ2]   ✓ TopicResponse[1] Topic Name: pax
remote: [tester::#WQ2]   ✓ TopicResponse[1] Topic UUID: 00000000-0000-4000-8000-000000000028
remote: [tester::#WQ2]     ✓ PartitionResponse[0] Error code: 0
remote: [tester::#WQ2]     ✓ PartitionResponse[0] Partition Index: 0
remote: [tester::#WQ2]   ✓ TopicResponse[2] Error code: 0
remote: [tester::#WQ2]   ✓ TopicResponse[2] Topic Name: paz
remote: [tester::#WQ2]   ✓ TopicResponse[2] Topic UUID: 00000000-0000-4000-8000-000000000060
remote: [tester::#WQ2]     ✓ PartitionResponse[0] Error code: 0
remote: [tester::#WQ2]     ✓ PartitionResponse[0] Partition Index: 0
remote: [tester::#WQ2]     ✓ PartitionResponse[1] Error code: 0
remote: [tester::#WQ2]     ✓ PartitionResponse[1] Partition Index: 1
remote: [tester::#WQ2] Test passed.
remote: [your_program] Client connected
remote:
remote: [tester::#KU4] Running tests for Stage #KU4 (Listing Partitions - List for multiple partitions)
remote: [tester::#KU4] [Serializer] Finished writing log files to: /tmp/kraft-combined-logs
remote: [tester::#KU4] $ ./your_program.sh /tmp/server.properties
remote: [your_program] Waiting for a client to connect...
remote: [your_program] Logs from your program will appear here!
remote: [tester::#KU4] Sending "DescribeTopicPartitions" (version: 0) request (Correlation id: 314149278)
remote: [tester::#KU4] ✓ Correlation ID: 314149278
remote: [tester::#KU4] ✓ Throttle Time: 0
remote: [tester::#KU4]   ✓ TopicResponse[0] Error code: 0
remote: [tester::#KU4]   ✓ TopicResponse[0] Topic Name: paz
remote: [tester::#KU4]   ✓ TopicResponse[0] Topic UUID: 00000000-0000-4000-8000-000000000060
remote: [tester::#KU4]     ✓ PartitionResponse[0] Error code: 0
remote: [tester::#KU4]     ✓ PartitionResponse[0] Partition Index: 0
remote: [tester::#KU4]     ✓ PartitionResponse[1] Error code: 0
remote: [tester::#KU4]     ✓ PartitionResponse[1] Partition Index: 1
remote: [tester::#KU4] Test passed.
remote:
remote: [tester::#EA7] Running tests for Stage #EA7 (Listing Partitions - List for a single partition)
remote: [tester::#EA7] [Serializer] Finished writing log files to: /tmp/kraft-combined-logs
remote: [tester::#EA7] $ ./your_program.sh /tmp/server.properties
remote: [tester::#EA7] Waiting for a client to connect...
remote: [tester::#EA7] Logs from your program will appear here!
remote: [tester::#EA7] Sending "DescribeTopicPartitions" (version: 0) request (Correlation id: 595624181)
remote: [tester::#EA7] ✓ Correlation ID: 595624181
remote: [tester::#EA7] ✓ Throttle Time: 0
remote: [tester::#EA7]   ✓ TopicResponse[0] Error code: 0
remote: [tester::#EA7]   ✓ TopicResponse[0] Topic Name: foo
remote: [tester::#EA7]   ✓ TopicResponse[0] Topic UUID: 00000000-0000-4000-8000-000000000091
remote: [tester::#EA7] Test passed.
remote:
remote: [tester::#VT6] Running tests for Stage #VT6 (Listing Partitions - List for an unknown topic)
remote: [tester::#VT6] $ ./your_program.sh /tmp/server.properties
remote: [your_program] Logs from your program will appear here!
remote: [your_program] Waiting for a client to connect...
remote: [tester::#VT6] Sending "DescribeTopicPartitions" (version: 0) request (Correlation id: 1942830623)
remote: [tester::#VT6] ✓ Correlation ID: 1942830623
remote: [tester::#VT6] ✓ Throttle Time: 0
remote: [tester::#VT6]   ✓ TopicResponse[0] Error code: 3
remote: [tester::#VT6]   ✓ TopicResponse[0] Topic Name: unknown-topic-qux
remote: [tester::#VT6]   ✓ TopicResponse[0] Topic UUID: 00000000-0000-0000-0000-000000000000
remote: [tester::#VT6] Test passed.
remote: [your_program] Client connected
remote:
remote: [tester::#YK1] Running tests for Stage #YK1 (Listing Partitions - Include DescribeTopicPartitions in APIVersions)
remote: [tester::#YK1] $ ./your_program.sh /tmp/server.properties
remote: [tester::#YK1] Waiting for a client to connect...
remote: [your_program] Logs from your program will appear here!
remote: [tester::#YK1] Sending "ApiVersions" (version: 4) request (Correlation id: 424359743)
remote: [tester::#YK1] ✓ Correlation ID: 424359743
remote: [tester::#YK1] ✓ Error code: 0 (NO_ERROR)
remote: [tester::#YK1] ✓ API keys array length: 3
remote: [tester::#YK1] ✓ MinVersion for API_VERSIONS is <= 4 & >= 0
remote: [tester::#YK1] ✓ MaxVersion for API_VERSIONS is >= 4
remote: [tester::#YK1] ✓ MinVersion for DESCRIBE_TOPIC_PARTITIONS is <= 0 & >= 0
remote: [tester::#YK1] ✓ MaxVersion for DESCRIBE_TOPIC_PARTITIONS is >= 0
remote: [tester::#YK1] Test passed.
remote:
remote: [tester::#SK0] Running tests for Stage #SK0 (Concurrent Clients - Concurrent requests)
remote: [tester::#SK0] $ ./your_program.sh /tmp/server.properties
remote: [your_program] Waiting for a client to connect...
remote: [your_program] Logs from your program will appear here!
remote: [tester::#SK0] Sending request 1 of 3: "ApiVersions" (version: 4) request (Correlation id: -64857363)
remote: [tester::#SK0] Sending request 2 of 3: "ApiVersions" (version: 4) request (Correlation id: -1184034673)
remote: [tester::#SK0] Sending request 3 of 3: "ApiVersions" (version: 4) request (Correlation id: -1197805985)
remote: [tester::#SK0] ✓ Correlation ID: -1197805985
remote: [tester::#SK0] ✓ Error code: 0 (NO_ERROR)
remote: [tester::#SK0] ✓ API keys array is non-empty
remote: [tester::#SK0] ✓ API version 4 is supported for API_VERSIONS
remote: [tester::#SK0] ✓ Test 3 of 3: Passed
remote: [tester::#SK0] ✓ Correlation ID: -1184034673
remote: [tester::#SK0] ✓ Error code: 0 (NO_ERROR)
remote: [tester::#SK0] ✓ API keys array is non-empty
remote: [tester::#SK0] ✓ API version 4 is supported for API_VERSIONS
remote: [tester::#SK0] ✓ Test 2 of 3: Passed
remote: [tester::#SK0] ✓ Correlation ID: -64857363
remote: [tester::#SK0] ✓ Error code: 0 (NO_ERROR)
remote: [tester::#SK0] ✓ API keys array is non-empty
remote: [tester::#SK0] ✓ API version 4 is supported for API_VERSIONS
remote: [tester::#SK0] ✓ Test 1 of 3: Passed
remote: [tester::#SK0] Test passed.
remote: [your_program] Client connected
remote: [your_program] Client connected
remote:
remote: [tester::#NH4] Running tests for Stage #NH4 (Concurrent Clients - Serial requests)
remote: [tester::#NH4] $ ./your_program.sh /tmp/server.properties
remote: [your_program] Waiting for a client to connect...
remote: [your_program] Logs from your program will appear here!
remote: [tester::#NH4] Sending request 1 of 2: "ApiVersions" (version: 4) request (Correlation id: 876772789)
remote: [tester::#NH4] ✓ Correlation ID: 876772789
remote: [tester::#NH4] ✓ Error code: 0 (NO_ERROR)
remote: [tester::#NH4] ✓ API keys array is non-empty
remote: [tester::#NH4] ✓ API version 4 is supported for API_VERSIONS
remote: [tester::#NH4] ✓ Test 1 of 2: Passed
remote: [tester::#NH4] Sending request 2 of 2: "ApiVersions" (version: 4) request (Correlation id: 669245750)
remote: [tester::#NH4] ✓ Correlation ID: 669245750
remote: [tester::#NH4] ✓ Error code: 0 (NO_ERROR)
remote: [tester::#NH4] ✓ API keys array is non-empty
remote: [tester::#NH4] ✓ API version 4 is supported for API_VERSIONS
remote: [tester::#NH4] ✓ Test 2 of 2: Passed
remote: [tester::#NH4] Test passed.
remote: [your_program] Client connected
remote:
remote: [tester::#PV1] Running tests for Stage #PV1 (Handle APIVersions requests)
remote: [tester::#PV1] $ ./your_program.sh /tmp/server.properties
remote: [your_program] Logs from your program will appear here!
remote: [your_program] Waiting for a client to connect...
remote: [tester::#PV1] Sending "ApiVersions" (version: 4) request (Correlation id: 1282148698)
remote: [tester::#PV1] ✓ Correlation ID: 1282148698
remote: [tester::#PV1] ✓ Error code: 0 (NO_ERROR)
remote: [tester::#PV1] ✓ API keys array is non-empty
remote: [tester::#PV1] ✓ API version 4 is supported for API_VERSIONS
remote: [tester::#PV1] Test passed.
remote:
remote: [tester::#NC5] Running tests for Stage #NC5 (Parse API Version)
remote: [tester::#NC5] $ ./your_program.sh /tmp/server.properties
remote: [tester::#NC5] Waiting for a client to connect...
remote: [your_program] Logs from your program will appear here!
remote: [tester::#NC5] Sending "ApiVersions" (version: 19189) request (Correlation id: 570642089)
remote: [tester::#NC5] ✓ Correlation ID: 570642089
remote: [tester::#NC5] ✓ Error code: 35 (UNSUPPORTED_VERSION)
remote: [tester::#NC5] Test passed.
remote:
remote: [tester::#WA6] Running tests for Stage #WA6 (Parse Correlation ID)
remote: [tester::#WA6] $ ./your_program.sh /tmp/server.properties
remote: [tester::#WA6] Waiting for a client to connect...
remote: [your_program] Logs from your program will appear here!
remote: [tester::#WA6] Sending "ApiVersions" (version: 4) request (Correlation id: 332188075)
remote: [tester::#WA6] ✓ Correlation ID: 332188075
remote: [tester::#WA6] Test passed.
remote: [your_program] Client connected
remote:
remote: [tester::#NV3] Running tests for Stage #NV3 (Send Correlation ID)
remote: [tester::#NV3] $ ./your_program.sh /tmp/server.properties
remote: [tester::#NV3] Waiting for a client to connect...
remote: [your_program] Logs from your program will appear here!
remote: [tester::#NV3] Sending "ApiVersions" (version: 4) request (Correlation id: 7)
remote: [tester::#NV3] ✓ Correlation ID: 7
remote: [tester::#NV3] Test passed.
remote: 
remote: [tester::#VI6] Running tests for Stage #VI6 (Bind to a port)
remote: [tester::#VI6] $ ./your_program.sh /tmp/server.properties
remote: [tester::#VI6] Connecting to port 9092...
remote: [your_program] Waiting for a client to connect...
remote: [your_program] Logs from your program will appear here!
remote: [tester::#VI6] Test passed.

Test Summary:

  • ✅ Consuming Messages: Fetch multiple/single messages from disk, empty topics, unknown topics
  • ✅ Listing Partitions: Multiple topics, multiple partitions, single partition, unknown topics
  • ✅ Concurrent Clients: Concurrent and serial request handling
  • ✅ API Implementation: ApiVersions and DescribeTopicPartitions support
  • ✅ Protocol Handling: Correlation ID parsing, API version validation
  • ✅ Network: Port binding and client connection handling

progress-banner

Acknowledgments

This project was inspired by and built as part of a coding challenge, providing an excellent framework for learning these concepts.

About

a Kafka-like distributed message broker with replication, partitioning, and fault-tolerant messaging for scalable communication

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

No packages published

Languages