Kafka
The Kafka input plugin allows subscribing to one or more Kafka topics to collect messages from an Apache Kafka service. This plugin uses the official librdkafka C library (built-in dependency).
Configuration Parameters
Getting Started
In order to subscribe/collect messages from Apache Kafka, you can run the plugin from the command line or through the configuration file:
Command Line
The kafka plugin can read parameters through the -p argument (property), e.g:
Configuration File
In your main configuration file append the following Input & Output sections:
Example of using kafka input/output plugins
The calyptia-fluent-bit source repository contains a full example of using calyptia-fluent-bit to process kafka records:
The above will connect to the broker listening on kafka-broker:9092
and subscribe to the fb-source
topic, polling for new messages every 100 milliseconds.
Every message received is then processed with kafka.lua
and sent back to the fb-sink
topic of the same broker.
The example can be executed locally with make start
in the examples/kafka_filter
directory (docker/compose is used).
Last updated