Elasticsearch HTTP

Receive data from Elastic Beats, Agent, Logstash and other ES HTTP compatible agents

The Elasticsearch HTTP plugin allows you to recieve data from Beats, Elastic Agent, Logstash and other agents that can send data via an Elasticsearch HTTP bulk request. Using this plugin allows you to recieve the data in Calyptia Core to perform the following:

1. Reduce noisy or non-essential data

2. Modify, Enrich, or Redact incoming data

3. Route Elastic Agent, Beats, Logstash generated data to alternative backends (E.g. Azure Blob)

Configuration Parameters

The following are configuration Parameters for Elasticsearch HTTP Source Plugin.

Key	Description
Port	The TCP Port the pipeline should listen on.

The following are Advanced configuration Parameters for Elasticsearch HTTP Source Plugin.

Key	Description
Buffer Chunk Size	Set the chunk size for incoming messages
Buffer Max Size	This sets the chunk size for incoming incoming JSON messages. These chunks are then stored/managed in the space available by buffer_size.

The following are Security and TLS configuration Parameters for Elasticsearch HTTP Source Plugin.

Key	Description
TLS	Enable or Disable TLS/SSL support
TLS Certificate Validation	Turn TLS/SSL certificate validation on / off, TLS must be on for this setting to be enabled.
TLS Debug Level	Set TLS debug verbosity level. It accepts the following values: 0 (No debug), 1 (Error), 2 (State change), 3 (Informational), and 4 Verbose
CA Certificate File Path	Absolute path to CA certificate file
Certificate File Path	Absolute path to Certificate file
Private key File Path	Absolute path to private Key file
Private Key Path Password	Optional password for tls.key_file file
TLS SNI Hostname Extension	Hostname to be used for TLS SNI Extension

Elastic Agent Configuration

in the Elastic Agent configuratoin within the output for Elasticsearch two settings are essential to receive data. allow_older_versions and ilm . A sample configuration of Elastic Agent can be seen below

######################################
# Fleet configuration
######################################
outputs:
  default:
    type: elasticsearch
    hosts: [127.0.0.1:9200]
    allow_older_versions: true
    ilm: false