OpenSearch
Send logs to Amazon OpenSearch Service
The opensearch output plugin, allows to ingest your records into an OpenSearch database. The following instructions assumes that you have a fully operational OpenSearch service running in your environment.
Configuration Parameters
The parameters index and type can be confusing if you are new to OpenSearch, if you have used a common relational database before, they can be compared to the database and table concepts. Also see the FAQ below
TLS / SSL
OpenSearch output plugin supports TTL/SSL, for more details about the properties available and general configuration, please refer to the TLS/SSL section.
write_operation
The write_operation can be any of:
Please note, Id_Key
or Generate_ID
is required in update, and upsert scenario.
Getting Started
In order to insert records into an OpenSearch service, you can run the plugin from the command line or through the configuration file:
Command Line
The opensearch plugin, can read the parameters from the command line in two ways, through the -p argument (property) or setting them directly through the service URI. The URI format is the following:
Using the format specified, you could start Calyptia Fluent Bit through:
which is similar to do:
Configuration File
In your main configuration file append the following Input & Output sections. You can visualize this configuration here
About OpenSearch field names
Some input plugins may generate messages where the field names contains dots. This opensearch plugin replaces them with an underscore, e.g:
becomes
FAQ
Calyptia Fluent Bit + Amazon OpenSearch Service
The Amazon OpenSearch Service adds an extra security layer where HTTP requests must be signed with AWS Sigv4. This plugin supports Amazon OpenSearch Service with IAM Authentication.
See here for details on how AWS credentials are fetched.
Example configuration:
Notice that the Port
is set to 443
, tls
is enabled, and AWS_Region
is set.
Action/metadata contains an unknown parameter type
Similarly to Elastic Cloud, OpenSearch in version 2.0 and above needs to have type option being removed by setting Suppress_Type_Name On
.
Without this you will see errors like:
Fluent-Bit + Amazon OpenSearch Serverless
Amazon OpenSearch Serverless is an offering that eliminates your need to manage OpenSearch clusters. All existing Calyptia Fluent Bit OpenSearch output plugin options work with OpenSearch Serverless. For Calyptia Fluent Bit, the only difference is that you must specify the service name as aoss
(Amazon OpenSearch Serverless) when you enable AWS_Auth
:
Data Access Permissions
When sending logs to OpenSearch Serverless, your AWS IAM entity needs OpenSearch Serverless Data Access permisions. Give your IAM entity the following data access permissions to your serverless collection:
With data access permissions, IAM policies are not needed to access the collection.
Last updated