ECS Metadata
The ECS Filter Enriches logs with AWS Elastic Container Service Metadata. The plugin can enrich logs with task, cluster and container metadata. The plugin uses the ECS Agent introspection API to obtain metadata. This filter only works with the ECS EC2 launch type. The filter only works when Calyptia Fluent Bit is running on an ECS EC2 Container Instance and has access to the ECS Agent introspection API. The filter is not supported on ECS Fargate. To obtain metadata on ECS Fargate, use the built-in FireLens metadata or the AWS for Calyptia Fluent Bit init project.
Configuration Parameters
The plugin supports the following configuration parameters:
Add
No default
ECS_Tag_Prefix
emptry string
Cluster_Metadata_Only
When enabled, the plugin will only attempt to attach cluster metadata values. This is useful if you want to attach cluster metadata to system/OS logs from processes that do not run as part of containers or ECS Tasks.
Off
ECS_Meta_Cache_TTL
The filter builds a hash table in memory mapping each unique container short ID to its metadata. This option sets a max TTL for objects in the hash table. You should set this if you have frequent container/task restarts. For example, your cluster runs short running batch jobs that complete in less than 10 minutes, there is no reason to keep any stored metadata longer than 10 minutes. So you would set this parameter to "10m".
1h (1 hour)
Supported Templating Variables for the ADD option
The following template variables can be used for values with the Add
option. See the tutorial below for examples.
$ClusterName
The ECS cluster name. Calyptia Fluent Bit is running on EC2 instance(s) that are part of this cluster.
Yes
$ContainerInstanceARN
The full ARN of the ECS EC2 Container Instance. This is the instance that Calyptia Fluent Bit is running on.
Yes
$ContainerInstanceID
The ID of the ECS EC2 Container Instance.
Yes
$ECSAgentVersion
The Version string of the ECS Agent that is running on the container instance.
Yes
$ECSContainerName
The name of the container from which the log originated. This is the name in your ECS Task Definition.
No
$DockerContainerName
The name of the container from which the log originated. This is the name obtained from Docker and is the name shown if you run docker ps
on the instance.
No
$ContainerID
The ID of the container from which the log originated. This is the full 64 character long container ID.
No
$TaskDefinitionFamily
The family name of the task definition for the task from which the log originated.
No
$TaskDefinitionVersion
The version/revision of the task definition for the task from which the log originated.
No
$TaskID
The ID of the ECS Task from which the log originated.
No
$TaskARN
The full ARN of the ECS Task from which the log originated.
No
Configuration File
Example 1: Attach Task ID and cluster name to container logs
The output log should be similar to:
Example 2: Attach customized resource name to container logs
The output log would be similar to:
Notice that the template variables in the value for the resource
key are separated by dot characters. Please see the section below about limitations in which characters can be used to separate template variables.
Example 3: Attach cluster metadata to non-container logs
This examples shows a use case for the Cluster_Metadata_Only
option- attaching cluster metadata to ECS Agent logs.
Limitations of record_accessor templating
Notice in example 2, that the template values are separated by dot characters. This is important; the Calyptia Fluent Bit record_accessor library has a limitation in the characters that can separate template variables- only dots and commas (.
and ,
) can come after a template variable. This is because the templating library must parse the template and determine the end of a variable.
The following would be invalid templates because the two template variables are not separated by commas or dots:
$TaskID-$ECSContainerName
$TaskID/$ECSContainerName
$TaskID_$ECSContainerName
$TaskIDfooo$ECSContainerName
However, the following are valid:
$TaskID.$ECSContainerName
$TaskID.ecs_resource.$ECSContainerName
$TaskID.fooo.$ECSContainerName
And the following are valid since they only contain one template variable with nothing after it:
fooo$TaskID
fooo____$TaskID
fooo/bar$TaskID
Last updated