Tap can be used to generate events or records detailing what messages pass through Calyptia Core Agent, at what time and what filters affect them.
Simple example
First, we will make sure that the container image we are going to use actually supports Calyptia Core Agent Tap (available in Calyptia Core Agent 22.10.+):
If the --enable-chunk-trace option is present it means Calyptia Core Agent has support for Calyptia Core Agent Tap but it is disabled by default, so remember to enable it with this option.
Tap support is enabled and disabled through the embedded web server, so enable it like so (or the equivalent option in the configuration file):
All the records that now appear are those emitted by the activities of the dummy plugin.
Complex example
This example takes the same steps but demonstrates the same mechanism works with more complicated configurations. In this example we will follow a single input of many which passes through several filters.
When activating Tap, any plugin parameter can be given. These can be used to modify, for example, the output format, the name of the time key, the format of the date, etc.
In the next example we will use the parameter "format": "json" to demonstrate how in Tap, stdout can be shown in Json format.
This parameter shows stdout in Json format, however, as mentioned before, parameters can be passed to any plugin.
Visit the following link for more information on other output plugins: Outputs Plugins
Analysis of a single Tap record
Here we analyze a single record from a filter event to explain the meaning of each field in detail. We chose a filter record since it includes the most details of all the record types.
The type defines at what stage the event is generated:
type=1: input record
this is the unadulterated input record
type=2: filtered record
this is a record once it has been filtered. One record is generated per filter.
type=3: pre-output record
this is the record right before it is sent for output.
Since this is a record generated by the manipulation of a record by a filter is has the type 2.
start_time and end_time
This records the start and end of an event, it is a bit different for each event type:
type 1: when the input is received, both the start and end time.
type 2: the time when filtering is matched until it has finished processing.
type 3: the time when the input is received and when it is finally slated for output.
trace_id
This is a string composed of a prefix and a number which is incremented with each record received by the input during the Tap session.
plugin_instance
This is the plugin instance name as it is generated by Calyptia Core Agent at runtime.
plugin_alias
If an alias is set this field will contain the alias set for a plugin.
records
This is an array of all the records being sent. Since Calyptia Core Agent handles records in chunks of multiple records and chunks are indivisible the same is done in the Tap output. Each record consists of its timestamp followed by the actual data which is a composite type of keys and values.
Dump internals signal
When the service is running we can export metrics to see the overall status of the data flow of the service. But there are other use cases where we would like to know the current status of the internals of the service, specifically to answer questions like what's the current status of the internal buffers ? , the Dump Internals feature is the answer.
Calyptia Core Agent v1.4 introduces the Dump Internals feature that can be triggered easily from the command line triggering the CONT Unix signal.
This feature is only available on Linux and BSD family operating systems.
Usage
Run the following kill command to signal Calyptia Core Agent:
kill -CONT `pidof fluent-bit`
The command pidof aims to lookup the Process ID of Calyptia Core Agent.
Calyptia Core Agent will dump the following information to the standard output interface (stdout):
[engine] caught signal (SIGCONT)
[2020/03/23 17:39:02] Calyptia Fluent Bit Dump
===== Input =====
syslog_debug (syslog)
│
├─ status
│ └─ overlimit : no
│ ├─ mem size : 60.8M (63752145 bytes)
│ └─ mem limit : 61.0M (64000000 bytes)
│
├─ tasks
│ ├─ total tasks : 92
│ ├─ new : 0
│ ├─ running : 92
│ └─ size : 171.1M (179391504 bytes)
│
└─ chunks
└─ total chunks : 92
├─ up chunks : 35
├─ down chunks: 57
└─ busy chunks: 92
├─ size : 60.8M (63752145 bytes)
└─ size err: 0
===== Storage Layer =====
total chunks : 92
├─ mem chunks : 0
└─ fs chunks : 92
├─ up : 35
└─ down : 57
Input pluginsdump
The dump provides insights for every input instance configured.
Status
Overall ingestion status of the plugin.
Entry
Sub-entry
Description
overlimit
If the plugin has been configured with Mem_Buf_Limit, this entry will report if the plugin is over the limit or not at the moment of the dump. If it is overlimit, it will print yes, otherwise no.
mem_size
Current memory size in use by the input plugin in-memory.
mem_limit
Limit set by Mem_Buf_Limit.
Tasks
When an input plugin ingest data into the engine, a Chunk is created. A Chunk can contains multiple records. Upon flush time, the engine creates a Task that contains the routes for the Chunk associated in question.
The Task dump describes the tasks associated to the input plugin:
Entry
Description
total_tasks
Total number of active tasks associated to data generated by the input plugin.
new
Number of tasks not assigned yet to an output plugin. Tasks are in new status for a very short period of time (most of the time this value is very low or zero).
running
Number of active tasks being processed by output plugins.
size
Amount of memory used by the Chunks being processed (Total chunks size).
Chunks
The Chunks dump tells more details about all the chunks that the input plugin has generated and are still being processed.
Depending of the buffering strategy and limits imposed by configuration, some Chunks might be up (in memory) or down (filesystem).
Entry
Sub-entry
Description
total_chunks
Total number of Chunks generated by the input plugin that are still being processed by the engine.
up_chunks
Total number of Chunks that are loaded in memory.
down_chunks
Total number of Chunks that are stored in the filesystem but not loaded in memory yet.
busy_chunks
Chunks marked as busy (being flushed) or locked. Busy Chunks are immutable and likely are ready to (or being) processed.
size
Amount of bytes used by the Chunk.
size err
Number of Chunks in an error state where it size could not be retrieved.
Storage layer dump
Calyptia Core Agent relies on a custom storage layer interface designed for hybrid buffering. The Storage Layer entry contains a total summary of Chunks registered by Calyptia Core Agent:
Entry
Sub-Entry
Description
total chunks
Total number of Chunks
mem chunks
Total number of Chunks memory-based
fs chunks
Total number of Chunks filesystem based
up
Total number of filesystem chunks up in memory
down
Total number of filesystem chunks down (not loaded in memory)
