Overview and creation

Learn how to create and deploy a custom pipeline

Custom pipelines allow you to read data from sources such as agents, network devices, syslog, APIs, and collect data from metric endpoints and object stores, such as S3.

Creating a custom pipeline is a few clicks away and each pipeline has the following advantages:

• Automatically load balanced when reading from a network source and multiple replicas selected.

• Self-healing in the case of unintended data.

• Buffering and Error-handling by default when the backend may not be functioning properly.

• Automatically monitored for performance and data statistics.

Creating your first pipeline

After deploying Calyptia Core you can navigate the Core Instance overview. On this overview page you can click create pipeline which allows you to choose from sources and destinations to start listening and routing data.

You can then click Save and Deploy to deploy the pipeline.

Starting with MockData is a good method to ensure your data is able to talk to your specific backend.

Arcade

Using Calyptia CLI

If you want to automate the creation of a pipeline, you can use Calyptia CLI with an existing configuration. These configurations can also be saved into your source control by navigating to Advanced Settings > Configuration, where you copy and paste.

For example, the following pipeline-es.conf file reads data from a TCP forward port and then outputs that data to an Elasticsearch endpoint:

pipeline-es.conf

[INPUT]
    Name          forward
    Host          0.0.0.0
    Port          24284

[OUTPUT]
    Name        es
    Match       *
    Host        hostname.us-east-2.es.amazonaws.com
    HTTP_User   es
    HTTP_Passwd 123123
    Port        443
    TLS         on

After a pipeline configuration is created, run the following command to deploy:

calyptia create pipeline --core-instance CORE_INSTANCE --config-file pipeline-es.conf

A manually deployed pipeline configuration is also validated against our configuration service. In some cases you may want to bypass the validation if you have a custom setting. To do this, use --skip-config-validation as a flag in the create command.

The pipeline auto-creates and you can retrieve the auto-generated name for the pipeline.

 NAME                 AGE
 honest-sunfire-cc22  125 milliseconds

As part of a pipeline deployment, a pipeline is automatically validated.