Golang Output Plugins

Calyptia Fluent Bit currently supports integration of Golang plugins built as shared objects for output plugins only. The interface for the Golang plugins is currently under development but is functional.
Build a Go Plugin
The fluent-bit-go package is available to assist developers in creating Go plugins.
​https://github.com/fluent/fluent-bit-go​
At a minimum, a Go plugin looks like this:
package main
​
import "github.com/fluent/fluent-bit-go/output"
​
//export FLBPluginRegister
func FLBPluginRegister(def unsafe.Pointer) int {
    // Gets called only once when the plugin.so is loaded
    return output.FLBPluginRegister(ctx, "gstdout", "Stdout GO!")
}
​
//export FLBPluginInit
func FLBPluginInit(plugin unsafe.Pointer) int {
    // Gets called only once for each instance you have configured.
    return output.FLB_OK
}
​
//export FLBPluginFlushCtx
func FLBPluginFlushCtx(ctx, data unsafe.Pointer, length C.int, tag *C.char) int {
    // Gets called with a batch of records to be written to an instance.
    return output.FLB_OK
}
​
//export FLBPluginExit
func FLBPluginExit() int {
    return output.FLB_OK
}
​
func main() {
}
the code above is a template to write an output plugin, it's really important to keep the package name as main and add an explicit main() function. This is a requirement as the code will be build as a shared library.
To build the code above, use the following line:
$ go build -buildmode=c-shared -o out_gstdout.so out_gstdout.go
Once built, a shared library called out\_gstdout.so will be available. It's really important to double check the final .so file is what we expect. Doing a ldd over the library we should see something similar to this:
$ ldd out_gstdout.so
    linux-vdso.so.1 =>  (0x00007fff561dd000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fc4aeef0000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc4aeb27000)
    /lib64/ld-linux-x86-64.so.2 (0x000055751a4fd000)
Run Calyptia Fluent Bit with the new plugin
$ bin/calyptia-fluent-bit -e /path/to/out_gstdout.so -i cpu -o gstdout
Configuration file
Calyptia Fluent Bit can load / run Golang plugin using two configuration file.
Plugins configuration file
​Main configuration file​
Plugins configuration file
Key
Description
Default Value
Path
A path for a Golang plugin.
​
Example
[PLUGINS]
    Path /path/to/out_gstdout.so
Main configuration file
The keys for Golang plugin available as of this version are described in the following table:
Key
Description
Default Value
Plugins_file
Path for a plugins configuration file. A plugins configuration file allows to define paths for external plugins, for an example see here.
​
Example
The following is an example of a main configuration file.
[SERVICE]
    plugins_file /path/to/plugins.conf
​
[INPUT]
    Name dummy
​
[OUTPUT]
    Name gstdout
Config key constraint
Some config keys are reserved by Calyptia Fluent Bit and must not be used by a custom plugin, they are: alias,host,ipv6,listen,log_level,log_suppress_interval,match,match_regex,mem_buf_limit,port,retry_limit,routable,storage.pause_on_chunks_overlimit, storage.total_limit_size, storage.type, tag,threaded,tls,tls.ca_file, tls.ca_path, tls.crt_file, tls.debug, tls.key_file, tls.key_passwd, tls.verify, tls.vhost, workers
Run using a configuration file
We can load a main configuration file using -c option. Note: No need to specify a plugins configuration file from command line.
calyptia-fluent-bit -c calyptia-fluent-bit.conf

Key	Description	Default Value
Path	A path for a Golang plugin.

Key	Description	Default Value
Plugins_file	Path for a plugins configuration file. A plugins configuration file allows to define paths for external plugins, for an example see here.

Last modified 3mo ago