Kafka

Kafka Destination Plugin

Apache Kafka is a distributed streaming platform that is used for building real-time data pipelines and streaming applications. It provides a high-throughput, low-latency platform for processing and transmitting large amounts of data, including logs, metrics, events, and other types of telemetry data.
This plugin provides a flexible and configurable way to transmit your data to Kafka. Whether you're working with logs, metrics, traces, or other types of telemetry data, the Kafka Destination Plugin in Calyptia Core provides a reliable and efficient way to integrate your data with Kafka for real-time processing and analysis.
Configuration Parameters
The following are configuration Parameters for Kafka Destination Plugin.
Key
Description
Brokers
Single of multiple list of Kafka Brokers, e.g: 192.168.1.3:9092, 192.168.1.4:9092.
Topics
Single entry or list of topics separated by comma (,) that Fluent Bit will use to send messages to Kafka. If only one topic is set, that one will be used for all records. Instead if multiple topics exists, the one set in the record by Topic_Key will be used.
Format
Specify data format, options available: json, msgpack.
The following are Advanced configuration Parameters for Kafka Destination Plugin.
Key
Description
Format
Specify data format, options available: json, msgpack.
Message Key
Optional Key to store the message
Timestamp Key
Set the key to store the record timestamp
Timestamp Format
Set the format to iso8601 or double
Body Key
Specify the key which contains the body
Queue Full Retries
Fluent Bit queues data into rdkafka library, if for some reason the underlying library cannot flush the records the queue might fills up blocking new addition of records. The queue_full_retries option set the number of local retries to enqueue the data. The default value is 10 times, the interval between each retry is 1 second. Setting the queue_full_retries value to 0 sets an unlimited number of retries
Message Key Field
If set, the value of Message_Key_Field in the record will indicate the message key. If not set nor found in the record, Message_Key will be used (if set).

Key	Description
Brokers	Single of multiple list of Kafka Brokers, e.g: 192.168.1.3:9092, 192.168.1.4:9092.
Topics	Single entry or list of topics separated by comma (,) that Fluent Bit will use to send messages to Kafka. If only one topic is set, that one will be used for all records. Instead if multiple topics exists, the one set in the record by Topic_Key will be used.
Format	Specify data format, options available: json, msgpack.

Key	Description
Format	Specify data format, options available: json, msgpack.
Message Key	Optional Key to store the message
Timestamp Key	Set the key to store the record timestamp
Timestamp Format	Set the format to iso8601 or double
Body Key	Specify the key which contains the body
Queue Full Retries	Fluent Bit queues data into rdkafka library, if for some reason the underlying library cannot flush the records the queue might fills up blocking new addition of records. The queue_full_retries option set the number of local retries to enqueue the data. The default value is 10 times, the interval between each retry is 1 second. Setting the queue_full_retries value to 0 sets an unlimited number of retries
Message Key Field	If set, the value of Message_Key_Field in the record will indicate the message key. If not set nor found in the record, Message_Key will be used (if set).

OpenSearch

SumoLogic

Last modified 2mo ago