Kubernetes

Kubernetes Production Grade Log Processor

Calyptia Fluent Bit is a lightweight and extensible Log and Metrics Processor that comes with full support for Kubernetes:

• Process Kubernetes containers logs from the file system or Systemd/Journald.

• Enrich logs with Kubernetes Metadata.

• Centralize your logs in third party storage services like Elasticsearch, InfluxDB, HTTP, etc.

We highly recommend our Calyptia Core solution for automated Kubernetes collection

Concepts

Before getting started it is important to understand how Calyptia Fluent Bit is deployed. Kubernetes manages a cluster of nodes, so our log agent tool will need to run on every node to collect logs from every POD, hence Calyptia Fluent Bit is deployed as a DaemonSet (a POD that runs on every node of the cluster).

When Calyptia Fluent Bit runs, it will read, parse and filter the logs of every POD and will enrich each entry with the following information (metadata):

• Pod Name

• Pod ID

• Container Name

• Container ID

• Labels

• Annotations

To obtain this information, a built-in filter plugin called kubernetes talks to the Kubernetes API Server to retrieve relevant information such as the pod_id, labels and annotations, other fields such as pod_name, container_id and container_name are retrieved locally from the log file names. All of this is handled automatically, no intervention is required from a configuration aspect.

Installation

Calyptia Fluent Bit is typically deployed as a DaemonSet. This ensures it is present on every node wihtin your Kubernetes cluster.

Details

The default configuration of Calyptia Fluent Bit makes sure of the following:

• Consume all containers logs from the running Node.

• The Tail input plugin will not append more than 5MB into the engine until they are flushed to the Elasticsearch backend. This limit aims to provide a workaround for backpressure scenarios.

• The Kubernetes filter will enrich the logs with Kubernetes metadata, specifically labels and annotations. The filter only goes to the API Server when it cannot find the cached info, otherwise it uses the cache.

• The default backend in the configuration is Elasticsearch set by the Elasticsearch Output Plugin. It uses the Logstash format to ingest the logs. If you need a different Index and Type, please refer to the plugin option and do your own adjustments.

• There is an option called Retry_Limit set to False, that means if Calyptia Fluent Bit cannot flush the records to Elasticsearch it will re-try indefinitely until it succeeds.

Container Runtime Interface (CRI) parser

Calyptia Fluent Bit by default assumes that logs are formatted by the Docker interface standard. However, when using CRI you can run into issues with malformed JSON if you do not modify the parser used. Calyptia Fluent Bit includes a CRI log parser that can be used instead. An example of the parser is seen below:

# CRI Parser
[PARSER]
    # http://rubular.com/r/tjUt3Awgg4
    Name cri
    Format regex
    Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
    Time_Key    time
    Time_Format %Y-%m-%dT%H:%M:%S.%L%z

To use this parser change the Input section for your configuration from docker to cri

[INPUT]
    Name tail
    Path /var/log/containers/*.log
    Parser cri
    Tag kube.*
    Mem_Buf_Limit 5MB
    Skip_Long_Lines On