Calyptia Core Agent
23.4
23.4
  • Calyptia Fluent Bit v23.04 Documentation
  • Differences with Open Source
  • Performance and Benchmarking
  • Concepts
    • Key Concepts
    • Buffering
    • Data Pipeline
      • Input
      • Parser
      • Filter
      • Buffer
      • Router
      • Output
  • Installation
    • Getting Started with Calyptia Fluent Bit
    • Supported Platforms
    • Linux Packages
      • Amazon Linux
      • Redhat / CentOS
      • Debian
      • Ubuntu
    • Docker
    • Kubernetes
    • macOS
    • Windows
    • EKS Add On
  • Administration
    • Configuring Calyptia Fluent Bit
      • Classic mode
        • Format and Schema
        • Configuration File
        • Variables
        • Commands
        • Upstream Servers
        • Record Accessor
      • YAML Configuration File
      • Unit Sizes
      • Multiline Parsing
    • Transport Security
    • Buffering & Storage
    • Backpressure
    • Scheduling and Retries
    • Networking
    • Memory Management
    • Monitoring
    • HTTP Proxy
    • Hot Reload
    • Troubleshooting
  • Local Testing
    • Validating your Data and Structure
    • Running a Logging Pipeline Locally
  • Data Pipeline
    • Inputs
      • Collectd
      • CPU Log Based Metrics
      • Disk I/O Log Based Metrics
      • Docker Log Based Metrics
      • Docker Events
      • Dummy
      • Elasticsearch
      • Exec
      • Exec Wasi
      • Fluent Bit Metrics
      • Forward
      • Head
      • HTTP
      • Kafka
      • Health
      • Kernel Logs
      • Memory Metrics
      • MQTT
      • Network I/O Log Based Metrics
      • NGINX Exporter Metrics
      • Node Exporter Metrics
      • Podman Metrics
      • Process Log Based Metrics
      • Prometheus Scrape Metrics
      • Random
      • Serial Interface
      • Standard Input
      • StatsD
      • Syslog
      • Systemd
      • Tail
      • TCP
      • Thermal
      • OpenTelemetry
      • Windows Event Log
      • Windows Event Log (winevtlog)
      • Windows Exporter Metrics
    • Parsers
      • Configuring Parser
      • JSON
      • Regular Expression
      • LTSV
      • Logfmt
      • Decoders
    • Filters
      • AWS Metadata
      • CheckList
      • ECS Metadata
      • Expect
      • GeoIP2 Filter
      • Grep
      • Kubernetes
      • Log to Metrics
      • Lua
      • Parser
      • Record Modifier
      • Modify
      • Multiline
      • Nest
      • Nightfall
      • Rewrite Tag
      • Standard Output
      • Throttle
      • Tensorflow
      • Wasm
    • Outputs
      • Amazon CloudWatch
      • Amazon Kinesis Data Firehose
      • Amazon Kinesis Data Streams
      • Amazon S3
      • Azure Blob
      • Azure Data Explorer
      • Azure Log Analytics
      • Counter
      • Datadog
      • Elasticsearch
      • File
      • FlowCounter
      • Forward
      • GELF
      • Google Cloud BigQuery
      • HTTP
      • InfluxDB
      • Kafka
      • Kafka REST Proxy
      • LogDNA
      • Loki
      • NATS
      • New Relic
      • NULL
      • Observe
      • OpenSearch
      • OpenTelemetry
      • PostgreSQL
      • Prometheus Exporter
      • Prometheus Remote Write
      • SkyWalking
      • Slack
      • Splunk
      • Stackdriver
      • Standard Output
      • Syslog
      • TCP & TLS
      • Treasure Data
      • Vivo Exporter
      • WebSocket
  • Calyptia Fluent Bit for Developers
    • Golang Output Plugins
    • WASM Filter Plugins
    • WASM Input Plugins
Powered by GitBook
On this page
  • Build a Go Plugin
  • Run Calyptia Fluent Bit with the new plugin
  • Configuration file
  • Plugins configuration file
  • Main configuration file
  • Run using a configuration file
  1. Calyptia Fluent Bit for Developers

Golang Output Plugins

PreviousWebSocketNextWASM Filter Plugins

Last updated 1 year ago

Calyptia Fluent Bit currently supports integration of Golang plugins built as shared objects for output plugins only. The interface for the Golang plugins is currently under development but is functional.

Build a Go Plugin

The fluent-bit-go package is available to assist developers in creating Go plugins.

At a minimum, a Go plugin looks like this:

package main

import "github.com/fluent/fluent-bit-go/output"

//export FLBPluginRegister
func FLBPluginRegister(def unsafe.Pointer) int {
    // Gets called only once when the plugin.so is loaded
    return output.FLBPluginRegister(ctx, "gstdout", "Stdout GO!")
}

//export FLBPluginInit
func FLBPluginInit(plugin unsafe.Pointer) int {
    // Gets called only once for each instance you have configured.
    return output.FLB_OK
}

//export FLBPluginFlushCtx
func FLBPluginFlushCtx(ctx, data unsafe.Pointer, length C.int, tag *C.char) int {
    // Gets called with a batch of records to be written to an instance.
    return output.FLB_OK
}

//export FLBPluginExit
func FLBPluginExit() int {
    return output.FLB_OK
}

func main() {
}

the code above is a template to write an output plugin, it's really important to keep the package name as main and add an explicit main() function. This is a requirement as the code will be build as a shared library.

To build the code above, use the following line:

$ go build -buildmode=c-shared -o out_gstdout.so out_gstdout.go

Once built, a shared library called out\_gstdout.so will be available. It's really important to double check the final .so file is what we expect. Doing a ldd over the library we should see something similar to this:

$ ldd out_gstdout.so
    linux-vdso.so.1 =>  (0x00007fff561dd000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fc4aeef0000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc4aeb27000)
    /lib64/ld-linux-x86-64.so.2 (0x000055751a4fd000)

Run Calyptia Fluent Bit with the new plugin

$ bin/calyptia-fluent-bit -e /path/to/out_gstdout.so -i cpu -o gstdout

Configuration file

Calyptia Fluent Bit can load / run Golang plugin using two configuration file.

  • Plugins configuration file

Plugins configuration file

Key
Description
Default Value

Path

A path for a Golang plugin.

Example

[PLUGINS]
    Path /path/to/out_gstdout.so

Main configuration file

The keys for Golang plugin available as of this version are described in the following table:

Key
Description
Default Value

Plugins_file

Example

The following is an example of a main configuration file.

[SERVICE]
    plugins_file /path/to/plugins.conf

[INPUT]
    Name dummy

[OUTPUT]
    Name gstdout

Config key constraint

Some config keys are reserved by Calyptia Fluent Bit and must not be used by a custom plugin, they are: alias,host,ipv6,listen,log_level,log_suppress_interval,match,match_regex,mem_buf_limit,port,retry_limit,routable,storage.pause_on_chunks_overlimit, storage.total_limit_size, storage.type, tag,threaded,tls,tls.ca_file, tls.ca_path, tls.crt_file, tls.debug, tls.key_file, tls.key_passwd, tls.verify, tls.vhost, workers

Run using a configuration file

We can load a main configuration file using -c option. Note: No need to specify a plugins configuration file from command line.

calyptia-fluent-bit -c calyptia-fluent-bit.conf

Path for a plugins configuration file. A plugins configuration file allows to define paths for external plugins, for an example .

https://github.com/fluent/fluent-bit-go
Main configuration file
see here