Scaling
Increasing Performance for a Pipeline
When ingesting data via network sources or agents you may wish to scale the pipeline to process additional records. Thankfully with Calyptia Core this process is simple as all ingested data is automatically load balanced within a pipeline replica allowing
Within Advanced Settings simply increase the number of replicas to the desired amount.
For replica count recommendations over 100 TB a day we recommend reaching out to the Calyptia team prior to scaling up to ensure proper stability.
Scaling up an existing Calyptia Core pipeline can be performed using the update command
calyptia update pipeline <PIPELINE-NAME-OR-ID> --replicas <NUM OF REPLICAS>
To check the number of replicas for a given pipeline, run:
calyptia get pipelines --core-instance <CORE INSTANCE NAME>
NAME REPLICAS STATUS AGE
pipeline-name 3 STARTED 15 hours
Last modified 2mo ago