2.x
Ask or search…
K

Amazon Kinesis Firehose

Amazon Kinesis Firehose destination plugin
Amazon Kinesis Firehose is a fully managed service provided by Amazon Web Services (AWS) for ingesting, transforming, and loading streaming data into AWS data stores and analytics tools. The Kinesis Firehose Destination Plugin in Calyptia Core lets you configure your pipeline to collect various data types such as logs, metrics, traces, and events and stream them directly to your Kinesis Firehose delivery stream for efficient and reliable delivery to your desired AWS data store or analytics tool.

Configuration parameters

The Amazon Kinesis Firehouse destination plugin provides these configuration parameters.

General

Key
Description
Region
The AWS region of your Kinesis Streams.
Kinesis Firehouse Name
Kinesis Firehouse Name.

AWS authentication

Key
Description
IAM Role ARN
ARN of an IAM role to assume (ex. for cross account access).
Firehose API Endpoint
Custom Endpoint for the FirehoSe API.
STS API Endpoint
Custom Endpoint for the STS API.
External ID for STS API
Specify an external ID for the STS API, can be used with the role_arn parameter if your role requires an external ID.

Advanced

Key
Description
Compression
Compression type for Firehose records. Each log record is individually compressed and sent to Firehose. 'gzip' and 'arrow' are the supported values. 'arrow' is only an available if Apache Arrow was enabled at compile time. Defaults to no compression.
Time Key
Add the timestamp to the record under this key. By default the timestamp from Fluent Bit will not be added to records sent to Kinesis.
Time Key Format
strftime compliant format string for the timestamp; for example, the default is '%Y-%m-%dT%H:%M:%S'. This option is used with time_key.
Log Key
By default, the whole log record will be sent to Kinesis. If you specify a key name with this option, then only the value of that key will be sent to Kinesis. For example, if you are using the Fluentd Docker log driver, you can specify log_key log and only the log message will be sent to Kinesis.
Enable Auto Retry Requests
Immediately retry failed requests to AWS services once. This option does not affect the normal Fluent Bit retry mechanism with backoff. Instead, it enables an immediate retry with no delay for networking errors, which may help improve throughput when there are transient/random networking issues.

Security and TLS

Key
Description
TLS
Enable or disable TLS/SSL support.
TLS Certificate Validation
Turn TLS/SSL certificate validation on or off. TLS must be on for this setting to be enabled.
TLS Debug Level
Set TLS debug verbosity level. Accepts these values: 0 (No debug), 1 (Error), 2 (State change), 3 (Informational), 4 (Verbose).
CA Certificate File Path
Absolute path to CA certificate file.
Certificate File Path
Absolute path to certificate file.
Private key File Path
Absolute path to private key file.
Private Key Path Password
Optional password for tls.key_file file.
TLS SNI Hostname Extension
Hostname to be used for TLS SNI extension.

Advanced networking

Key
Description
DNS Mode
Select the primary DNS connection type (TCP or UDP).
DNS Resolver
Select the primary DNS connection type (TCP or UDP).
Prefer IPv4
Prioritize IPv4 DNS results when trying to establish a connection.
Keepalive
Enable or disable Keepalive support.
Keepalive Idle Timeout
Set maximum time allowed for an idle Keepalive connection.
Max Connect Timeout
Set maximum time allowed to establish a connection, this time includes the TLS handshake.
Max Connect Timeout Log Error
On connection timeout, specify if it should log an error. When disabled, the timeout is logged as a debug message.
Max Keepalive Recycle
Set maximum number of times a keepalive connection can be used before it is retired.
Source Address
Specify network address to bind for data traffic.