WASM Filter Plugins
Calyptia Fluent Bit currently supports integration of wasm plugins built as wasm/wasi objects for input and filter plugins only. The interface for the WASM filter plugins is currently under development but is functional.
flb-wamrc is just flb- prefixed AOT (Ahead Of Time) compiler that is provided from wasm-micro-runtime.For
flb-wamrc support, users have to install llvm infrastructure, e.g:# apt install -y llvm
Currently, Calyptia Fluent Bit supports the following WASM toolchains:
Currently, Calyptia Fluent Bit's WASM filter assumes C ABI that is also known as
wasm32-unknown-unknown on Rust target and wasm32-wasi on TinyGo target.TinyGo and WASI SDK support wasm target by default. When using Rust's
wasm32-unknown-unknown target, users must install wasm32-unknown-unknown via rustup. Then, installing that target components as:$ rustup target add wasm32-unknown-unknown
WASM filter plugins execute the function that has the following signagure.
For C:
// We can use an arbitrary function name for filter operations w/ WASM.
char* c_filter(char*, int, uint32_t, uint32_t, char*, int);
For Go(TinyGo):
//export go_filter
// And this function should be placed in the main package.
func go_filter(tag *uint8, tag_len uint, time_sec uint, time_nsec uint, record *uint8, record_len uint) *uint8
For Rust:
// #[no_mangle] attribute is needed for preventing mangles and align C ABI.
// Also we can use an arbitrary function name for filter operations w/ WASM.
#[no_mangle]
pub extern “C” fn rust_filter(tag: *const c_char, tag_len: u32, time_sec: u32, time_nsec: u32, record: *const c_char, record_len: u32)
Note that
//export XXX on TinyGo and #[no_mangle] attributes on Rust are required. This is because TinyGo and Rust will mangle their function names if they are not specified.Once built, a WASM program will be available. Then, that built program can be executed with the following Calyptia Fluent Bit configuration:
[INPUT]
Name dummy
Tag dummy.local
[FILTER]
Name wasm
Tag dummy.*
WASM_Path /path/to/built_filter.wasm
Function_Name super_awesome_filter
accessible_paths .,/path/to/calyptia-fluent-bit
[OUTPUT]
Name stdout
Match *
[0] dummy.local: [1666270588.271704000, {"lang"=>"Rust", "message"=>"dummy", "original"=>"{"message":"dummy"}", "tag"=>"dummy.local", "time"=>"2022-10-20T12:56:28.271704000 +0000"}]
[0] dummy.local: [1666270589.270348000, {"lang"=>"Rust", "message"=>"dummy", "original"=>"{"message":"dummy"}", "tag"=>"dummy.local", "time"=>"2022-10-20T12:56:29.270348000 +0000"}]
[0] dummy.local: [1666270590.271107000, {"lang"=>"Rust", "message"=>"dummy", "original"=>"{"message":"dummy"}", "tag"=>"dummy.local", "time"=>"2022-10-20T12:56:30.271107000 +0000"}]
To optimize WASM program execution, there is the option of using
flb-wamrc. flb-wamrc will reduce runtime footprint and to be best perforemance for filtering operations. This tool will be built when -DFLB_WAMRC=On cmake option is specififed and llvm infrastructure is installed on the building box.$ flb-wamrc -o /path/to/built_wasm.aot /path/to/built_wasm.wasm
For further optimizations to the specific CPU such as Cortex-A57 series, e.g:
$ flb-wamrc --size-level=3 --target=aarch64v8 --cpu=cortex-a57 -o /path/to/built_wasm.aot /path/to/built_wasm.wasm
Then, when AOT (Ahead Of Time) compiling is succeeded:
Create AoT compiler with:
target: aarch64v8
target cpu: cortex-a57
cpu features:
opt level: 3
size level: 3
output format: AoT file
Compile success, file /path/to/built_wasm.aot was generated.
Note that AOT compiling should generate CPU architecture-dependent objects. If users want to use AOT compiled object on the different archtecture, it must align the target and target cpu for actual environments.
- C Filter
- https://github.com/fluent/fluent-bit/tree/master/examples/filter_wasm_c
- Rust Filter
- https://github.com/fluent/fluent-bit/tree/master/examples/filter_rust
- TinyGo Filter
- https://github.com/fluent/fluent-bit/tree/master/examples/filter_wasm_go