Deploying Pipelines

To make full use of Calyptia Core you need to create pipelines that allow you to define configuration related to listening to data, processing data, and sending that data to specific endpoints.

Calyptia Core pipeline configuration is fully based on Fluent Bit's configuration with a few select differences:

  • The SERVICE section is not required and is managed by Calyptia Core
  • Input Plugins that read from a local server (E.g. Tail, Systemd, etc.) are not currently supported
  • Filter Plugins that require data from a side-loaded file (E.g. GeoIP, CheckList, TensorFlow, etc.) require additional configuration

Defining a Pipeline

To create a pipeline you need to use the Calyptia Cloud CLI. First create a configuration, in the following example pipeline-es.conf we will read data from a TCP forward port and output that data to an Elasticsearch endpoint

[INPUT]
    Name          forward
    Host          0.0.0.0
    Port          24284

[OUTPUT]
    Name        es
    Match       *
    Host        hostname.us-east-2.es.amazonaws.com
    HTTP_User   es
    HTTP_Passwd 123123
    Port        443
    TLS         on

Deploying a Pipeline

After a pipeline configuration is created we can run the following to deploy

calyptia create pipeline --aggregator <AGGREGATOR ID> --config-file pipeline-es.conf

The pipeline auto-creates and we can retrieve the auto-generated name for the pipeline

 NAME                 AGE              
 honest-sunfire-cc22  125 milliseconds 

Validating a Pipeline

As part of a pipeline deployment, a pipeline is automatically validated.

Advanced options

Similar to Fluent Bit configuration there are many options you can also specify to customize the behavior and meet your requirements for your observability pipelines

These include the following

  • Resource Profiles
  • Secrets
  • Parsers
  • Referencing External Files

Updating a Pipeline

Pipelines can be updated at any time by running the following command

calyptia update pipeline <PIPELINE NAME>
Usage:
  calyptia update pipeline PIPELINE [flags]

Flags:
      --auto-create-ports      Automatically create pipeline ports from config if updated (default true)
      --config-file string     New Fluent Bit config file used by pipeline
  -h, --help                   help for pipeline
      --new-name string        New pipeline name
      --output-format string   Output format. Allowed: table, json (default "table")
      --replicas uint          New pipeline replica size

Deleting a Pipeline

You can run the following to delete a Pipeline

calyptia delete pipeline <PIPELINE NAME OR ID>

Did this page help you?