Overview and Creation

Learn how to create and deploy a custom pipeline

Custom pipelines allow you to read data from sources such as agents, network devices, syslog, APIs as well as collect data from metric endpoints and object stores, such as S3.
Creating a custom pipeline is a few clicks away and each pipeline has the advantage of the following:
Automatically load balanced when reading from a network source and multiple replicas selected
Self-healing in the case of unintended data
Buffering and Error-handling by default when the backend may not be functioning properly
Automatically monitored for performance and data statistics
Creating your first Pipeline
After deploying Calyptia Core you can navigate the Core Instance overview. On this overview page you can click create pipeline which allows you to choose from sources and destinations to start listening and routing data.
Once completed you can select Save and Deploy to deploy the pipeline
Starting with MockData is typically the easiest to ensure that your data is able to talk to your specific backend
Using the Calyptia CLI
If you want to automate the creation of a pipeline you can also use the Calyptia CLI with an existing configuration. These configurations can also be saved into your source control by navigating to Advanced Settings > Configuration where you copy and paste.
For example in the following pipeline-es.conf we will read data from a TCP forward port and output that data to an Elasticsearch endpoint
pipeline-es.conf
[INPUT]
    Name          forward
    Host          0.0.0.0
    Port          24284
​
[OUTPUT]
    Name        es
    Match       *
    Host        hostname.us-east-2.es.amazonaws.com
    HTTP_User   es
    HTTP_Passwd 123123
    Port        443
    TLS         on
After a pipeline configuration is created we can run the following to deploy
calyptia create pipeline --core-instance <CALYPTIA CORE NAME> --config-file pipeline-es.conf
Pipeline configuration that is deployed manually is also validated against our configuration service. In some cases you may wish to bypass the validation if you have a custom setting, this can be done using --skip-config-validation as a flag in the create command.
The pipeline auto-creates and we can retrieve the auto-generated name for the pipeline
 NAME                 AGE              
 honest-sunfire-cc22  125 milliseconds 
As part of a pipeline deployment, a pipeline is automatically validated.

Getting Started - Previous

Create your first pipeline

Next - Custom Pipelines

Modifying, Updating, Deleting

Last modified 2mo ago