Lua
The Lua filter allows you to modify the incoming records (even split one record into multiple records) using custom Lua scripts.
Due to the necessity to have a flexible filtering mechanism, it is now possible to extend Calyptia Fluent Bit capabilities by writing custom filters using Lua programming language. A Lua-based filter takes two steps:
Configure the Filter in the main configuration
Prepare a Lua script that will be used by the Filter
Configuration Parameters
The plugin supports the following configuration parameters:
Getting Started
In order to test the filter, you can run the plugin from the command line or through the configuration file. The following examples use the dummy input plugin for data ingestion, invoke Lua filter using the test.lua script and call the cb_print() function which only prints the same information to the standard output:
Command Line
From the command line you can use the following options:
Configuration File
In your main configuration file append the following Input, Filter & Output sections:
Lua Script Filter API
The life cycle of a filter have the following steps:
Upon Tag matching by this filter, it may process or bypass the record.
If tag matched, it will accept the record and invoke the function defined in the
call
property which basically is the name of a function defined in the Luascript
.Invoke Lua function and pass each record in JSON format.
Upon return, validate return value and continue the pipeline.
Callback Prototype
The Lua script can have one or multiple callbacks that can be used by this filter. The function prototype is as follows:
Function Arguments
Return Values
Each callback must return three values:
Code Examples
For functional examples of this interface, please refer to the code samples provided in the source code of the project located here:
https://github.com/fluent/fluent-bit/tree/master/scripts
Inline configuration
The Calyptia Fluent Bit smoke tests include examples to verify during CI.
In classic mode:
Environment variable processing
As an example that combines a bit of LUA processing with the Kubernetes filter that demonstrates using environment variables with LUA regex and substitutions.
Kubernetes pods generally have various environment variables set by the infrastructure automatically which may contain useful information.
In this example, we want to extract part of the Kubernetes cluster API name.
The environment variable is set like so: KUBERNETES_SERVICE_HOST: api.sandboxbsh-a.project.domain.com
We want to extract the sandboxbsh
name and add it to our record as a special key.
Number Type
+Lua treats number as double. It means an integer field (e.g. IDs, log levels) will be converted double. To avoid type conversion, The type_int_key
property is available.
Protected Mode
Calyptia Fluent Bit supports protected mode to prevent crash when executes invalid Lua script. See also Error Handling in Application Code.
Record Split
The Lua callback function can return an array of tables (i.e., array of records) in its third record return value. With this feature, the Lua filter can split one input record into multiple records according to custom logic.
For example:
Lua script
Configuration
Input
Output
See also Calyptia Fluent Bit: PR 811.
Response code filtering
In this example, we want to filter istio logs to exclude lines with response codes between 1 and 399. Istio is configured to write the logs in json format.
Lua script
Script response_code_filter.lua
Configuration
Configuration to get istio logs and apply response code filter to them.
Input
Output
In the output only the messages with response code 0 or greater than 399 are shown.
Last updated