WASM Filter Plugins

WebAssembly is binary instruction format for stack based virtual machine.

Calyptia Core Agent supports integration of wasm plugins built as wasm/wasi objects for input and filter plugins only. The interface for the WASM filter plugins is currently under development but is functional.

Prerequisites

Building flb-wamrc (optional)

flb-wamrc is just flb- prefixed AOT (Ahead Of Time) compiler that is provided from wasm-micro-runtime.

For flb-wamrc support, users have to install llvm infrastructure, e.g:

# apt install -y llvm

For Build WASM programs

Calyptia Core Agent supports the following WASM toolchains:

• Rust on wasm32-unknown-unknown.

  • rustc 1.62.1 (e092d0b6b 2022-07-16) or later

• TinyGo on wasm32-wasi

  • v0.24.0 or later

• WASI SDK 13 or later.

Build a WASM Filter for Filter Plugin

The Calyptia Core Agent WASM filter assumes C ABI that is also known as wasm32-unknown-unknown on Rust target and wasm32-wasi on TinyGo target.

To Install Additional Components

TinyGo and WASI SDK support wasm target by default. When using Rust's wasm32-unknown-unknown target, users must install wasm32-unknown-unknown by using rustup. Then, installing that target components as:

$ rustup target add wasm32-unknown-unknown

Requirements of WASM programs

WASM filter plugins execute the function that has the following signagure.

For C:

// We can use an arbitrary function name for filter operations w/ WASM.
char* c_filter(char*, int, uint32_t, uint32_t, char*, int);

For Go(TinyGo):

//export go_filter
// And this function should be placed in the main package.
func go_filter(tag *uint8, tag_len uint, time_sec uint, time_nsec uint, record *uint8, record_len uint) *uint8

For Rust:

// #[no_mangle] attribute is needed for preventing mangles and align C ABI.
// Also we can use an arbitrary function name for filter operations w/ WASM.
#[no_mangle]
pub extern “C” fn rust_filter(tag: *const c_char, tag_len: u32, time_sec: u32, time_nsec: u32, record: *const c_char, record_len: u32)

Note that //export XXX on TinyGo and #[no_mangle] attributes on Rust are required. This is because TinyGo and Rust will mangle their function names if they are not specified.

Once built, a WASM program will be available. Then, that built program can be executed with the following Calyptia Core Agent configuration:

[INPUT]
    Name dummy
    Tag dummy.local

[FILTER]
    Name wasm
    Tag  dummy.*
    WASM_Path /path/to/built_filter.wasm
    Function_Name super_awesome_filter
    accessible_paths .,/path/to/calyptia-fluent-bit

[OUTPUT]
    Name  stdout
    Match *

For example, one of the examples of Rust WASM filter should generate its filtered logs as follows:

[0] dummy.local: [1666270588.271704000, {"lang"=>"Rust", "message"=>"dummy", "original"=>"{"message":"dummy"}", "tag"=>"dummy.local", "time"=>"2022-10-20T12:56:28.271704000 +0000"}]
[0] dummy.local: [1666270589.270348000, {"lang"=>"Rust", "message"=>"dummy", "original"=>"{"message":"dummy"}", "tag"=>"dummy.local", "time"=>"2022-10-20T12:56:29.270348000 +0000"}]
[0] dummy.local: [1666270590.271107000, {"lang"=>"Rust", "message"=>"dummy", "original"=>"{"message":"dummy"}", "tag"=>"dummy.local", "time"=>"2022-10-20T12:56:30.271107000 +0000"}]

Optimize execution of WASM programs

To optimize WASM program execution, there is the option of using flb-wamrc. flb-wamrc will reduce runtime footprint and to be best perforemance for filtering operations. This tool will be built when -DFLB_WAMRC=On cmake option is specififed and llvm infrastructure is installed on the building box.

$ flb-wamrc -o /path/to/built_wasm.aot /path/to/built_wasm.wasm

For further optimizations to the specific CPU such as Cortex-A57 series:

$ flb-wamrc --size-level=3 --target=aarch64v8 --cpu=cortex-a57 -o /path/to/built_wasm.aot /path/to/built_wasm.wasm

Then, when AOT (Ahead Of Time) compiling is succeeded:

Create AoT compiler with:
  target:        aarch64v8
  target cpu:    cortex-a57
  cpu features:
  opt level:     3
  size level:    3
  output format: AoT file
Compile success, file /path/to/built_wasm.aot was generated.

Note that AOT compiling should generate CPU architecture-dependent objects. If users want to use AOT compiled object on the different archtecture, it must align the target and target cpu for actual environments.

Further concrete examples

• C Filter

  • https://github.com/fluent/fluent-bit/tree/master/examples/filter_wasm_c

• Rust Filter

  • https://github.com/fluent/fluent-bit/tree/master/examples/filter_rust

• TinyGo Filter

  • https://github.com/fluent/fluent-bit/tree/master/examples/filter_wasm_go