Scaling

Increasing Performance for a Pipeline

When ingesting data via network sources or agents you may wish to scale the pipeline to process additional records. Thankfully with Calyptia Core this process is simple as all ingested data is automatically load balanced within a pipeline replica allowing

Within Advanced Settings simply increase the number of replicas to the desired amount.

Arcade

For replica count recommendations over 100 TB a day we recommend reaching out to the Calyptia team prior to scaling up to ensure proper stability.

Calyptia CLI

Scaling up an existing Calyptia Core pipeline can be performed using the update command

calyptia update pipeline <PIPELINE-NAME-OR-ID> --replicas <NUM OF REPLICAS>

To check the number of replicas for a given pipeline, run:

calyptia get pipelines --core-instance <CORE INSTANCE NAME>
 NAME                 REPLICAS  STATUS   AGE      
 pipeline-name               3  STARTED  15 hours