Creating and Deploying

Learn how to create and deploy a pipeline

To create a pipeline you need to use the Calyptia Cloud CLI. First create a configuration, in the following example pipeline-es.conf we will read data from a TCP forward port and output that data to an Elasticsearch endpoint

pipeline-es.conf

[INPUT]
    Name          forward
    Host          0.0.0.0
    Port          24284

[OUTPUT]
    Name        es
    Match       *
    Host        hostname.us-east-2.es.amazonaws.com
    HTTP_User   es
    HTTP_Passwd 123123
    Port        443
    TLS         on

Deploying a Pipeline

After a pipeline configuration is created we can run the following to deploy

calyptia create pipeline --aggregator <AGGREGATOR ID> --config-file pipeline-es.conf

The pipeline auto-creates and we can retrieve the auto-generated name for the pipeline

 NAME                 AGE              
 honest-sunfire-cc22  125 milliseconds 

Validating a Pipeline

As part of a pipeline deployment, a pipeline is automatically validated.

Advanced options

Similar to Fluent Bit configuration there are many options you can also specify to customize the behavior and meet your requirements for your observability pipelines

These include the following

• Resource Profiles

• Secrets

• Parsers

• Referencing External Files

Updating a Pipeline

Pipelines can be updated at any time by running the following command

calyptia update pipeline <PIPELINE NAME>

Usage:
  calyptia update pipeline PIPELINE [flags]

Flags:
      --auto-create-ports      Automatically create pipeline ports from config if updated (default true)
      --config-file string     New Fluent Bit config file used by pipeline
  -h, --help                   help for pipeline
      --new-name string        New pipeline name
      --output-format string   Output format. Allowed: table, json (default "table")
      --replicas uint          New pipeline replica size

Deleting a Pipeline

You can run the following to delete a Pipeline

calyptia delete pipeline <PIPELINE NAME OR ID>