Kafka

Kafka source plugin

Apache Kafka is an open-source distributed streaming platform that is used to collect, store, and process large volumes of real-time data streams. It is particularly well-suited for collecting telemetry data, which is the data produced by sensors and devices that monitor the performance and health of systems, machines, and applications.

Kafka is designed to handle high volumes of data in real time, making it a popular choice for applications that require low latency and high throughput. It uses a publish-subscribe model to collect and distribute data among multiple data producers and consumers. Data producers, such as sensors or applications, publish data to Kafka topics, while data consumers subscribe to these topics to receive the data.

You can use the Kafka Source Plugin to configure your calyptia core pipeline to collect data from your OpenTelemetry instances.

Configuration Parameters

The following are configuration Parameters for Kafka Source Plugin.

Key	Description
Brokers	Single of multiple lists of Kafka Brokers, e.g: 192.168.1.3:9092, 192.168.1.4:9092.
Topics	Single entry or list of topics separated by comma (,) that Calyptia Core will use to receive messages to Kafka.
Minimum Queued Messages	Minimum number of messages per topic+partition Calyptia Core tries to maintain in the local consumer queue.