Amazon CloudWatch Logs and Metrics

Amazon CloudWatch Logs and Metrics Destination Plugin

Amazon CloudWatch is a monitoring and observability service provided by Amazon Web Services (AWS). CloudWatch Logs allows you to monitor, store, and access log files from your EC2 instances, CloudTrail, Lambda, and other AWS resources. With CloudWatch Metrics, you can collect and analyze metric data from your resources to gain insights into their performance and health.
The CloudWatch Destination Plugin in Calyptia Core allows you to easily configure your pipeline to stream your log data and metrics to CloudWatch Logs and Metrics respectively.
Configuration Parameters
The following are configuration Parameters for Amazon CloudWatch Logs and Metrics Destination Plugin.
Key
Description
Region
The AWS region to send your logs or metrics to
CloudWatch Log Group Name
CloudWatch Log Group Name
CloudWatch Log Stream Name
CloudWatch Log Stream Name; Not compatible with Log Stream Prefix
The following are AWS Authentication configuration Parameters for Amazon Kinesis Firehouse Destination Plugin.
Key
Description
IAM Role ARN
ARN of an IAM role to assume (ex. for cross account access).
CloudWatch Logs and Metrics API Endpoint
Custom Endpoint for the FirehoSe API
STS API Endpoint
Custom Endpoint for the STS API
External ID for STS API
Specify an external ID for the STS API, can be used with the role_arn parameter if your role requires an external ID.
The following are configuration Parameters for Amazon CloudWatch Logs and Metrics Destination Plugin.
Key
Description
CloudWatch Log Stream Name Prefix
Prefix for CloudWatch Log Stream Name; the tag is appended to the prefix to form the stream name
CloudWatch Log Group Template
Template for CloudWatch Log Group name using record accessor syntax. Plugin falls back to the log_group_name configured if needed.
CloudWatch Log Stream Template
Template for CloudWatch Log Stream name using record accessor syntax. Plugin falls back to the log_stream_name or log_stream_prefix configured if needed.
Number of days to retain logs
If set to a number greater than zero, and newly create log group's retention policy is set to this many days. Valid values are: [1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731, 1827, 3653]
Log Format
An optional parameter that can be used to tell CloudWatch the format of the data. A value of json/emf enables CloudWatch to extract custom metrics embedded in a JSON payload
Log Key
By default, the whole log record will be sent to CloudWatch. If you specify a key name with this option, then only the value of that key will be sent to CloudWatch. For example, if you are using the Fluentd Docker log driver, you can specify log_key log and only the log message will be sent to CloudWatch.
Enable Auto Retry Requests
Immediately retry failed requests to AWS services once. This option does not affect the normal Fluent Bit retry mechanism with backoff. Instead, it enables an immediate retry with no delay for networking errors, which may help improve throughput when there are transient/random networking issues.
Enable Auto Create Log Group
Automatically create the log group (log streams will always automatically be created)
Metric namespace for CloudWatch EMF logs
Metric namespace for CloudWatch EMF logs
Metric Dimension List
Metric dimensions is a list of lists. If you have only one list of dimensions, put the values as a comma seperated string. If you want to put list of lists, use the list as semicolon seperated strings. If your value is 'd1,d2;d3', we will consider it as [[d1, d2],[d3]].
The following are Security and TLS Configuration configuration Parameters for Amazon CloudWatch Logs and Metrics Destination Plugin.
Key
Description
TLS
Enable or Disable TLS/SSL support
TLS Certificate Validation
Turn TLS/SSL certificate validation on / off, TLS must be on for this setting to be enabled.
TLS Debug Level
Set TLS debug verbosity level. It accepts the following values: 0 (No debug), 1 (Error), 2 (State change), 3 (Informational), and 4 Verbose
CA Certificate File Path
Absolute path to CA certificate file
Certificate File Path
Absolute path to Certificate file
Private key File Path
Absolute path to private Key file
Private Key Path Password
Optional password for tls.key_file file
TLS SNI Hostname Extension
Hostname to be used for TLS SNI Extension
The following are Advanced Networking configuration Parameters for Amazon CloudWatch Logs and Metrics Destination Plugin.
Key
Description
DNS Mode
Select the primary DNS connection type (TCP or UDP)
DNS Resolver
Select the primary DNS connection type (TCP or UDP)
Prefer IPv4
Prioritize IPv4 DNS results when trying to establish a connection
Keepalive
Enable or disable Keepalive support
Keepalive Idle Timeout
Set maximum time allowed for an idle Keepalive connection
Max Connect Timeout
Set maximum time allowed to establish a connection, this time includes the TLS handshake
Max Connect Timeout Log Error
On connection timeout, specify if it should log an error. When disabled, the timeout is logged as a debug message
Max Keepalive Recycle
Set maximum number of times a keepalive connection can be used before it is retired.
Source Address
Specify network address to bind for data traffic

Key	Description
Region	The AWS region to send your logs or metrics to
CloudWatch Log Group Name	CloudWatch Log Group Name
CloudWatch Log Stream Name	CloudWatch Log Stream Name; Not compatible with Log Stream Prefix

Key	Description
IAM Role ARN	ARN of an IAM role to assume (ex. for cross account access).
CloudWatch Logs and Metrics API Endpoint	Custom Endpoint for the FirehoSe API
STS API Endpoint	Custom Endpoint for the STS API
External ID for STS API	Specify an external ID for the STS API, can be used with the role_arn parameter if your role requires an external ID.

Key	Description
CloudWatch Log Stream Name Prefix	Prefix for CloudWatch Log Stream Name; the tag is appended to the prefix to form the stream name
CloudWatch Log Group Template	Template for CloudWatch Log Group name using record accessor syntax. Plugin falls back to the log_group_name configured if needed.
CloudWatch Log Stream Template	Template for CloudWatch Log Stream name using record accessor syntax. Plugin falls back to the log_stream_name or log_stream_prefix configured if needed.
Number of days to retain logs	If set to a number greater than zero, and newly create log group's retention policy is set to this many days. Valid values are: [1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731, 1827, 3653]
Log Format	An optional parameter that can be used to tell CloudWatch the format of the data. A value of json/emf enables CloudWatch to extract custom metrics embedded in a JSON payload
Log Key	By default, the whole log record will be sent to CloudWatch. If you specify a key name with this option, then only the value of that key will be sent to CloudWatch. For example, if you are using the Fluentd Docker log driver, you can specify log_key log and only the log message will be sent to CloudWatch.
Enable Auto Retry Requests	Immediately retry failed requests to AWS services once. This option does not affect the normal Fluent Bit retry mechanism with backoff. Instead, it enables an immediate retry with no delay for networking errors, which may help improve throughput when there are transient/random networking issues.
Enable Auto Create Log Group	Automatically create the log group (log streams will always automatically be created)
Metric namespace for CloudWatch EMF logs	Metric namespace for CloudWatch EMF logs
Metric Dimension List	Metric dimensions is a list of lists. If you have only one list of dimensions, put the values as a comma seperated string. If you want to put list of lists, use the list as semicolon seperated strings. If your value is 'd1,d2;d3', we will consider it as [[d1, d2],[d3]].

Key	Description
TLS	Enable or Disable TLS/SSL support
TLS Certificate Validation	Turn TLS/SSL certificate validation on / off, TLS must be on for this setting to be enabled.
TLS Debug Level	Set TLS debug verbosity level. It accepts the following values: 0 (No debug), 1 (Error), 2 (State change), 3 (Informational), and 4 Verbose
CA Certificate File Path	Absolute path to CA certificate file
Certificate File Path	Absolute path to Certificate file
Private key File Path	Absolute path to private Key file
Private Key Path Password	Optional password for tls.key_file file
TLS SNI Hostname Extension	Hostname to be used for TLS SNI Extension

Key	Description
DNS Mode	Select the primary DNS connection type (TCP or UDP)
DNS Resolver	Select the primary DNS connection type (TCP or UDP)
Prefer IPv4	Prioritize IPv4 DNS results when trying to establish a connection
Keepalive	Enable or disable Keepalive support
Keepalive Idle Timeout	Set maximum time allowed for an idle Keepalive connection
Max Connect Timeout	Set maximum time allowed to establish a connection, this time includes the TLS handshake
Max Connect Timeout Log Error	On connection timeout, specify if it should log an error. When disabled, the timeout is logged as a debug message
Max Keepalive Recycle	Set maximum number of times a keepalive connection can be used before it is retired.
Source Address	Specify network address to bind for data traffic

Amazon Kinesis Firehose

GCP Operations Suite (Stackdriver)

Last modified 2mo ago