Key Concepts

There are a few key concepts that are really important to understand how Calyptia Core Agent operates

Before diving into Calyptia Core Agent, it’s good to get acquainted with some of the key concepts of the service. This document provides a gentle introduction to those concepts and common Calyptia Core Agent terminology. We’ve provided a list below of all the terms we’ll cover, but we recommend reading this document from start to finish to gain a more general understanding of our log and stream processor.

  • Event or Record

  • Filtering

  • Tag

  • Timestamp

  • Match

  • Structured Message

Event or Record

Every incoming piece of data that belongs to a log or a metric that is retrieved by Calyptia Core Agent is considered an Event or a Record.

As an example consider the following content of a Syslog file:

Jan 18 12:52:16 flb systemd[2222]: Starting GNOME Terminal Server
Jan 18 12:52:16 flb dbus-daemon[2243]: [session uid=1000 pid=2243] Successfully activated service 'org.gnome.Terminal'
Jan 18 12:52:16 flb systemd[2222]: Started GNOME Terminal Server.
Jan 18 12:52:16 flb gsd-media-keys[2640]: # watch_fast: "/org/gnome/terminal/legacy/" (establishing: 0, active: 0)

It contains four lines and all of them represents four independent Events.

Internally an Event is comprised of:

  • timestamp

  • key/value metadata (since v2.1.0)

  • payload

Event format

The Calyptia Core Agent wire protocol represents an Event as a 2-element array with a nested array as the first element:

[[TIMESTAMP, METADATA], MESSAGE]

where

  • TIMESTAMP is a timestamp in seconds as an integer or floating point value (not a string);

  • METADATA is a possibly-empty object containing event metadata; and

  • MESSAGE is an object containing the event body.

Previous Calyptia Core Agent versions prior to v23.04 instead used:

[TIMESTAMP, MESSAGE]

to represent events. This format is still supported for reading input event streams.

Filtering

In some cases it is required to perform modifications on the Events content, the process to alter, enrich or drop Events is called Filtering.

There are many use cases when Filtering is required like:

  • Append specific information to the Event like an IP address or metadata.

  • Select a specific piece of the Event content.

  • Drop Events that matches certain pattern.

Tag

Every Event that gets into Calyptia Core Agent gets assigned a Tag. This tag is an internal string that is used in a later stage by the Router to decide which Filter or Output phase it must go through.

Most of the tags are assigned manually in the configuration. If a tag is not specified, Calyptia Core Agent will assign the name of the Input plugin instance from where that Event was generated from.

The only input plugin that does NOT assign tags is Forward input. This plugin speaks the Fluentd wire protocol called Forward where every Event already comes with a Tag associated. Calyptia Core Agent will always use the incoming Tag set by the client.

A Tagged record must always have a Matching rule. To learn more about Tags and Matches, see the Routing section.

Timestamp

The Timestamp represents the time when an Event was created. Every Event contains a Timestamp associated. The Timestamp is a numeric fractional integer in the format:

SECONDS.NANOSECONDS

Seconds

It is the number of seconds that have elapsed since the Unix epoch.

Nanoseconds

Fractional second or one thousand-millionth of a second.

A timestamp always exists, either set by the Input plugin or discovered through a data parsing process.

Match

Calyptia Core Agent allows to deliver your collected and processed Events to one or multiple destinations, this is done through a routing phase. A Match represent a simple rule to select Events where it Tags matches a defined rule.

To learn more about Tags and Matches, see the Routing section.

Structured Messages

Source events can have or not have a structure. A structure defines a set of keys and values inside the Event message. As an example consider the following two messages:

No structured message

"Project Calyptia Fluent Bit created on 1398289291"

Structured Message

{"project": "Calyptia Fluent Bit", "created": 1398289291}

At a low level both are just an array of bytes, but the Structured message defines keys and values, having a structure helps to implement faster operations on data modifications.

Calyptia Core Agent always handles every Event message as a structured message. For performance reasons, we use a binary serialization data format called MessagePack.

Consider MessagePack as a binary version of JSON on steroids.

Last updated