Comment on page

Elasticsearch

Elasticsearch Destination Plugin

Elasticsearch is a search and analytics engine that is commonly used to store and analyze large volumes of machine-generated data, such as logs, metrics, and other telemetry data. The Elasticsearch Destination Plugin in Calyptia Core allows you to easily integrate your telemetry data with Elasticsearch, enabling you to store, search, and visualize your data using Elasticsearch's powerful indexing and querying capabilities.
With the Elasticsearch Output Plugin, you can configure your Calyptia Core pipeline to output your telemetry data to Elasticsearch.
This plugin provides a flexible and configurable way to transmit your data to Elasticsearch, allowing you to customize the indexing settings, document mappings, and other attributes of your data to suit your specific needs.
Configuration Parameters
The following are configuration Parameters for Elasticsearch Destination Plugin.
Key
Description
Host
IP address or hostname of the target Elasticsearch instance
Port
TCP port of the target Elasticsearch instance
Index
Index name
Logstash format
Enable Logstash format compatibility.
The following are Security and TLS configuration Parameters for Elasticsearch Destination Plugin.
Key
Description
TLS
Enable or Disable TLS/SSL support
TLS Certificate Validation
Turn TLS/SSL certificate validation on / off, TLS must be on for this setting to be enabled.
TLS Debug Level
Set TLS debug verbosity level. It accepts the following values: 0 (No debug), 1 (Error), 2 (State change), 3 (Informational), and 4 Verbose
CA Certificate File Path
Absolute path to CA certificate file
Certificate File Path
Absolute path to Certificate file
Private key File Path
Absolute path to private Key file
Private Key Path Password
Optional password for tls.key_file file
TLS SNI Hostname Extension
Hostname to be used for TLS SNI Extension
The following are Advanced configuration Parameters for Elasticsearch Destination Plugin.
Key
Description
Replace Dots
When enabled, replace field name dots with underscore, required by versions of Elasticsearch
Type
Type Name
Suppress Type Name
If true, mapping types are removed. for v7.0.0 or later
Buffer Size
Specify the buffer size used to read the response from the Elasticsearch HTTP service. This option is useful for debugging purposes where is required to read full responses, note that response size grows depending of the number of records inserted. To set an unlimited amount of memory set this value to 'false', otherwise the value must be according to the Unit Size specification
Path
Elasticsearch accepts new data on HTTP query path '/_bulk'. But it is also possible to serve Elasticsearch behind a reverse proxy on a subpath. This option defines such path on the fluent-bit side. It simply adds a path prefix in the indexing HTTP POST URI
Pipeline
Newer versions of Elasticsearch allows to setup filters called pipelines. This option allows to define which pipeline the database should use. For performance reasons is strongly suggested to do parsing and filtering on Fluent Bit side, avoid pipelines
Generate ID
When enabled, generate _id for outgoing records. This prevents duplicate records when retrying ES
Write Operation
Operation to use to write in bulk requests
ID Key
If set, _id will be the value of the key from incoming record.
Replace Dots
Use current time for index generation instead of message record
Logstash Prefix
When Logstash_Format is enabled, the Index name is composed using a prefix and the date, e.g: If Logstash_Prefix is equals to 'mydata' your index will become 'mydata-YYYY.MM.DD'. The last string appended belongs to the date when the data is being generated'
Logstash Prefix Key
When included: the value in the record that belongs to the key will be looked up and over-write the Logstash_Prefix for index generation. If the key/value is not found in the record then the Logstash_Prefix option will act as a fallback. Nested keys are supported through record accessor pattern
Logstash Date Format
Time format (based on strftime) to generate the second part of the Index name
Time Key
When Logstash_Format is enabled, each record will get a new timestamp field. The Time_Key property defines the name of that field
Time Key Format
When Logstash_Format is enabled, this property defines the format of the timestamp
Time Key Nanos
When Logstash_Format is enabled, enabling this property sends nanosecond precision timestamps
Include Tag Key
When enabled, it append the Tag name to the record
Tag Key
When Include_Tag_key is enabled, this property defines the key name of the tag in the message
The following are Basic Authentication configuration Parameters for Elasticsearch Destination Plugin.
Key
Description
HTTP Username
Basic Auth Username
HTTP Password
Basic Auth Password. Requires HTTP_User to be set
The following are AWS Authentication configuration Parameters for Elasticsearch Destination Plugin.
Key
Description
Enable AWS Authentication
Enable AWS Sigv4 Authentication
AWS Region
AWS Region of your service
AWS STS Endpoint
Custom endpoint for the AWS STS API, used with the AWS_Role_ARN option
AWS IAM Assume Role
ARN of an IAM role to assume (ex. for cross-account access)
AWS External ID
Specify an external ID for the STS API, can be used with the aws_role_arn parameter if your role requires an external ID.
The following are Elastic Cloud Authentication configuration Parameters for Elasticsearch Destination Plugin.
Key
Description
Elastic Cloud ID
Elastic cloud ID of the cluster to connect to
Elastic Cloud Authentication Credentials
Elastic cloud authentication credentials
The following are Advanced Networking configuration Parameters for Elasticsearch Destination Plugin.
Key
Description
DNS Mode
Select the primary DNS connection type (TCP or UDP)
DNS Resolver
Select the primary DNS connection type (TCP or UDP)
Prefer IPv4
Prioritize IPv4 DNS results when trying to establish a connection
Keepalive
Enable or disable Keepalive support
Keepalive Idle Timeout
Set maximum time allowed for an idle Keepalive connection
Max Connect Timeout
Set maximum time allowed to establish a connection, this time includes the TLS handshake
Max Connect Timeout Log Error
On connection timeout, specify if it should log an error. When disabled, the timeout is logged as a debug message
Max Keepalive Recycle
Set maximum number of times a keepalive connection can be used before it is retired.
Source Address
Specify network address to bind for data traffic
The following are Debugging configuration Parameters for Elasticsearch Destination Plugin.
Key
Description
Trace Output
When enabled print the Elasticsearch API calls to stdout (for diag only)
Trace Error
When enabled print the Elasticsearch exception to stderr (for diag only)

Key	Description
Host	IP address or hostname of the target Elasticsearch instance
Port	TCP port of the target Elasticsearch instance
Index	Index name
Logstash format	Enable Logstash format compatibility.

Key	Description
TLS	Enable or Disable TLS/SSL support
TLS Certificate Validation	Turn TLS/SSL certificate validation on / off, TLS must be on for this setting to be enabled.
TLS Debug Level	Set TLS debug verbosity level. It accepts the following values: 0 (No debug), 1 (Error), 2 (State change), 3 (Informational), and 4 Verbose
CA Certificate File Path	Absolute path to CA certificate file
Certificate File Path	Absolute path to Certificate file
Private key File Path	Absolute path to private Key file
Private Key Path Password	Optional password for tls.key_file file
TLS SNI Hostname Extension	Hostname to be used for TLS SNI Extension

Key	Description
Replace Dots	When enabled, replace field name dots with underscore, required by versions of Elasticsearch
Type	Type Name
Suppress Type Name	If true, mapping types are removed. for v7.0.0 or later
Buffer Size	Specify the buffer size used to read the response from the Elasticsearch HTTP service. This option is useful for debugging purposes where is required to read full responses, note that response size grows depending of the number of records inserted. To set an unlimited amount of memory set this value to 'false', otherwise the value must be according to the Unit Size specification
Path	Elasticsearch accepts new data on HTTP query path '/_bulk'. But it is also possible to serve Elasticsearch behind a reverse proxy on a subpath. This option defines such path on the fluent-bit side. It simply adds a path prefix in the indexing HTTP POST URI
Pipeline	Newer versions of Elasticsearch allows to setup filters called pipelines. This option allows to define which pipeline the database should use. For performance reasons is strongly suggested to do parsing and filtering on Fluent Bit side, avoid pipelines
Generate ID	When enabled, generate _id for outgoing records. This prevents duplicate records when retrying ES
Write Operation	Operation to use to write in bulk requests
ID Key	If set, _id will be the value of the key from incoming record.
Replace Dots	Use current time for index generation instead of message record
Logstash Prefix	When Logstash_Format is enabled, the Index name is composed using a prefix and the date, e.g: If Logstash_Prefix is equals to 'mydata' your index will become 'mydata-YYYY.MM.DD'. The last string appended belongs to the date when the data is being generated'
Logstash Prefix Key	When included: the value in the record that belongs to the key will be looked up and over-write the Logstash_Prefix for index generation. If the key/value is not found in the record then the Logstash_Prefix option will act as a fallback. Nested keys are supported through record accessor pattern
Logstash Date Format	Time format (based on strftime) to generate the second part of the Index name
Time Key	When Logstash_Format is enabled, each record will get a new timestamp field. The Time_Key property defines the name of that field
Time Key Format	When Logstash_Format is enabled, this property defines the format of the timestamp
Time Key Nanos	When Logstash_Format is enabled, enabling this property sends nanosecond precision timestamps
Include Tag Key	When enabled, it append the Tag name to the record
Tag Key	When Include_Tag_key is enabled, this property defines the key name of the tag in the message

Key	Description
HTTP Username	Basic Auth Username
HTTP Password	Basic Auth Password. Requires HTTP_User to be set

Key	Description
Enable AWS Authentication	Enable AWS Sigv4 Authentication
AWS Region	AWS Region of your service
AWS STS Endpoint	Custom endpoint for the AWS STS API, used with the AWS_Role_ARN option
AWS IAM Assume Role	ARN of an IAM role to assume (ex. for cross-account access)
AWS External ID	Specify an external ID for the STS API, can be used with the `aws_role_arn` parameter if your role requires an external ID.

Key	Description
Elastic Cloud ID	Elastic cloud ID of the cluster to connect to
Elastic Cloud Authentication Credentials	Elastic cloud authentication credentials

Key	Description
DNS Mode	Select the primary DNS connection type (TCP or UDP)
DNS Resolver	Select the primary DNS connection type (TCP or UDP)
Prefer IPv4	Prioritize IPv4 DNS results when trying to establish a connection
Keepalive	Enable or disable Keepalive support
Keepalive Idle Timeout	Set maximum time allowed for an idle Keepalive connection
Max Connect Timeout	Set maximum time allowed to establish a connection, this time includes the TLS handshake
Max Connect Timeout Log Error	On connection timeout, specify if it should log an error. When disabled, the timeout is logged as a debug message
Max Keepalive Recycle	Set maximum number of times a keepalive connection can be used before it is retired.
Source Address	Specify network address to bind for data traffic

Key	Description
Trace Output	When enabled print the Elasticsearch API calls to stdout (for diag only)
Trace Error	When enabled print the Elasticsearch exception to stderr (for diag only)

Dynatrace Logs

Forward

Last modified 2mo ago