Calyptia Core Agent
23.4
23.4
  • Calyptia Fluent Bit v23.04 Documentation
  • Differences with Open Source
  • Performance and Benchmarking
  • Concepts
    • Key Concepts
    • Buffering
    • Data Pipeline
      • Input
      • Parser
      • Filter
      • Buffer
      • Router
      • Output
  • Installation
    • Getting Started with Calyptia Fluent Bit
    • Supported Platforms
    • Linux Packages
      • Amazon Linux
      • Redhat / CentOS
      • Debian
      • Ubuntu
    • Docker
    • Kubernetes
    • macOS
    • Windows
    • EKS Add On
  • Administration
    • Configuring Calyptia Fluent Bit
      • Classic mode
        • Format and Schema
        • Configuration File
        • Variables
        • Commands
        • Upstream Servers
        • Record Accessor
      • YAML Configuration File
      • Unit Sizes
      • Multiline Parsing
    • Transport Security
    • Buffering & Storage
    • Backpressure
    • Scheduling and Retries
    • Networking
    • Memory Management
    • Monitoring
    • HTTP Proxy
    • Hot Reload
    • Troubleshooting
  • Local Testing
    • Validating your Data and Structure
    • Running a Logging Pipeline Locally
  • Data Pipeline
    • Inputs
      • Collectd
      • CPU Log Based Metrics
      • Disk I/O Log Based Metrics
      • Docker Log Based Metrics
      • Docker Events
      • Dummy
      • Elasticsearch
      • Exec
      • Exec Wasi
      • Fluent Bit Metrics
      • Forward
      • Head
      • HTTP
      • Kafka
      • Health
      • Kernel Logs
      • Memory Metrics
      • MQTT
      • Network I/O Log Based Metrics
      • NGINX Exporter Metrics
      • Node Exporter Metrics
      • Podman Metrics
      • Process Log Based Metrics
      • Prometheus Scrape Metrics
      • Random
      • Serial Interface
      • Standard Input
      • StatsD
      • Syslog
      • Systemd
      • Tail
      • TCP
      • Thermal
      • OpenTelemetry
      • Windows Event Log
      • Windows Event Log (winevtlog)
      • Windows Exporter Metrics
    • Parsers
      • Configuring Parser
      • JSON
      • Regular Expression
      • LTSV
      • Logfmt
      • Decoders
    • Filters
      • AWS Metadata
      • CheckList
      • ECS Metadata
      • Expect
      • GeoIP2 Filter
      • Grep
      • Kubernetes
      • Log to Metrics
      • Lua
      • Parser
      • Record Modifier
      • Modify
      • Multiline
      • Nest
      • Nightfall
      • Rewrite Tag
      • Standard Output
      • Throttle
      • Tensorflow
      • Wasm
    • Outputs
      • Amazon CloudWatch
      • Amazon Kinesis Data Firehose
      • Amazon Kinesis Data Streams
      • Amazon S3
      • Azure Blob
      • Azure Data Explorer
      • Azure Log Analytics
      • Counter
      • Datadog
      • Elasticsearch
      • File
      • FlowCounter
      • Forward
      • GELF
      • Google Cloud BigQuery
      • HTTP
      • InfluxDB
      • Kafka
      • Kafka REST Proxy
      • LogDNA
      • Loki
      • NATS
      • New Relic
      • NULL
      • Observe
      • OpenSearch
      • OpenTelemetry
      • PostgreSQL
      • Prometheus Exporter
      • Prometheus Remote Write
      • SkyWalking
      • Slack
      • Splunk
      • Stackdriver
      • Standard Output
      • Syslog
      • TCP & TLS
      • Treasure Data
      • Vivo Exporter
      • WebSocket
  • Calyptia Fluent Bit for Developers
    • Golang Output Plugins
    • WASM Filter Plugins
    • WASM Input Plugins
Powered by GitBook
On this page
  • Prerequisites
  • For Build WASM programs
  • Build a WASM Filter for Filter Plugin
  • To Install Additional Components
  • Requirements of WASM programs
  • Optimize execution of WASM programs
  • Further Concrete Examples
  1. Calyptia Fluent Bit for Developers

WASM Filter Plugins

PreviousGolang Output PluginsNextWASM Input Plugins

Last updated 1 year ago

is binary instruction format for stack based virtual machine.

Calyptia Fluent Bit currently supports integration of wasm plugins built as wasm/wasi objects for input and filter plugins only. The interface for the WASM filter plugins is currently under development but is functional.

Prerequisites

Building flb-wamrc (optional)

flb-wamrc is just flb- prefixed AOT (Ahead Of Time) compiler that is provided from .

For flb-wamrc support, users have to install llvm infrastructure, e.g:

# apt install -y llvm

For Build WASM programs

Currently, Calyptia Fluent Bit supports the following WASM toolchains:

  • Rust on wasm32-unknown-unknown.

    • rustc 1.62.1 (e092d0b6b 2022-07-16) or later

  • on wasm32-wasi

    • v0.24.0 or later

  • 13 or later.

Build a WASM Filter for Filter Plugin

Currently, Calyptia Fluent Bit's WASM filter assumes C ABI that is also known as wasm32-unknown-unknown on Rust target and wasm32-wasi on TinyGo target.

To Install Additional Components

$ rustup target add wasm32-unknown-unknown

Requirements of WASM programs

WASM filter plugins execute the function that has the following signagure.

For C:

// We can use an arbitrary function name for filter operations w/ WASM.
char* c_filter(char*, int, uint32_t, uint32_t, char*, int);

For Go(TinyGo):

//export go_filter
// And this function should be placed in the main package.
func go_filter(tag *uint8, tag_len uint, time_sec uint, time_nsec uint, record *uint8, record_len uint) *uint8

For Rust:

// #[no_mangle] attribute is needed for preventing mangles and align C ABI.
// Also we can use an arbitrary function name for filter operations w/ WASM.
#[no_mangle]
pub extern “C” fn rust_filter(tag: *const c_char, tag_len: u32, time_sec: u32, time_nsec: u32, record: *const c_char, record_len: u32)

Note that //export XXX on TinyGo and #[no_mangle] attributes on Rust are required. This is because TinyGo and Rust will mangle their function names if they are not specified.

Once built, a WASM program will be available. Then, that built program can be executed with the following Calyptia Fluent Bit configuration:

[INPUT]
    Name dummy
    Tag dummy.local

[FILTER]
    Name wasm
    Tag  dummy.*
    WASM_Path /path/to/built_filter.wasm
    Function_Name super_awesome_filter
    accessible_paths .,/path/to/calyptia-fluent-bit

[OUTPUT]
    Name  stdout
    Match *
[0] dummy.local: [1666270588.271704000, {"lang"=>"Rust", "message"=>"dummy", "original"=>"{"message":"dummy"}", "tag"=>"dummy.local", "time"=>"2022-10-20T12:56:28.271704000 +0000"}]
[0] dummy.local: [1666270589.270348000, {"lang"=>"Rust", "message"=>"dummy", "original"=>"{"message":"dummy"}", "tag"=>"dummy.local", "time"=>"2022-10-20T12:56:29.270348000 +0000"}]
[0] dummy.local: [1666270590.271107000, {"lang"=>"Rust", "message"=>"dummy", "original"=>"{"message":"dummy"}", "tag"=>"dummy.local", "time"=>"2022-10-20T12:56:30.271107000 +0000"}]

Optimize execution of WASM programs

To optimize WASM program execution, there is the option of using flb-wamrc. flb-wamrc will reduce runtime footprint and to be best perforemance for filtering operations. This tool will be built when -DFLB_WAMRC=On cmake option is specififed and llvm infrastructure is installed on the building box.

$ flb-wamrc -o /path/to/built_wasm.aot /path/to/built_wasm.wasm

For further optimizations to the specific CPU such as Cortex-A57 series, e.g:

$ flb-wamrc --size-level=3 --target=aarch64v8 --cpu=cortex-a57 -o /path/to/built_wasm.aot /path/to/built_wasm.wasm

Then, when AOT (Ahead Of Time) compiling is succeeded:

Create AoT compiler with:
  target:        aarch64v8
  target cpu:    cortex-a57
  cpu features:
  opt level:     3
  size level:    3
  output format: AoT file
Compile success, file /path/to/built_wasm.aot was generated.

Note that AOT compiling should generate CPU architecture-dependent objects. If users want to use AOT compiled object on the different archtecture, it must align the target and target cpu for actual environments.

Further Concrete Examples

  • C Filter

    • https://github.com/fluent/fluent-bit/tree/master/examples/filter_wasm_c

  • Rust Filter

    • https://github.com/fluent/fluent-bit/tree/master/examples/filter_rust

  • TinyGo Filter

    • https://github.com/fluent/fluent-bit/tree/master/examples/filter_wasm_go

TinyGo and WASI SDK support wasm target by default. When using Rust's wasm32-unknown-unknown target, users must install wasm32-unknown-unknown via . Then, installing that target components as:

For example, one of the examples of should generate its filtered logs as follows:

WebAssembly
wasm-micro-runtime
TinyGo
WASI SDK
rustup
Rust WASM filter