Implement batch-based Kafka producer logic

[Ahmed-Ghanam]

Description

Publishing and consuming messages individually has proven inefficient for high-volume workloads. To improve throughput and reduce latency, a batch-oriented producer will be implemented in the Altinn Notifications API.

This enhancement will enable the API to publish batches of newly registered notification orders, email notifications, and SMS notifications for processing.

This issue focuses on the producer-side implementation and adds a new batching method alongside the existing single-message API to maintain backward compatibility.

Acceptance Criteria

• Add a new batching method to IKafkaProducer.
• Implement batching in KafkaProducer.
• Improve producer configuration.
• Enhance topic-creation logic.
• Add structured batch logging.

Non-Goals

• Updating consumers to handle batch consumption (covered in a separate issue).
• Changing message schema or Kafka topic structure.

Tasks

• Extend IKafkaProducer with a new batch API.
• Implement batch processing in KafkaProducer.
• Add metrics instrumentation.
• Enhance logging and error handling.
• Add/update unit tests.

Additional Notes

Batching functionality will be introduced alongside existing single-message publishing to ensure backward compatibility. A future cleanup may remove the single-message method once consumers and upstream services have migrated.

[Ahmed-Ghanam]

Batch Processing Performance Report

After applying the batch-producing logic in YT01, the following observations were recorded:

Timestamp (UTC)	Total Processed Notification Orders	Interval (sec)	Orders/sec
2025-11-27 11:02:16.525307	4,344	—	—
2025-11-27 11:05:05.192977	17,714	168.67	79.5
2025-11-27 11:06:14.915746	23,138	69.72	77.7
2025-11-27 11:07:22.459436	28,632	67.54	81.3
2025-11-27 11:09:12.763690	37,256	110.30	78.2

Summary

• Start Time: 2025-11-27 11:00:00+00
• End Time: 2025-11-27 11:09:12+00
• Total Orders Processed: 37,256
• Average Processing Rate: Avg Orders/sec = 37,256 ÷ 552.76 ≈ 67.4

Key Insights

• The system maintained a consistent throughput of approximately 78 orders per second during peak intervals.