Add powerful transformations to your pipelines
Calyptia core provides a set of Processing Rules that are built-in and ready to use. These enable you to transform your data in real time as it passes through your Telemetry pipeline. Examples can include:
- Adding a new field to every record
- Blocking extraneous fields
- Encoding / Decoding CSV or other formats
- Redacting or removing sensitive information
Processing Rules use a set of "actions" to perform transformations on the data. These actions can work with any data type, allowing you to transform various data formats.
When you use the Processing Rules, you specify one or more actions that will be applied to your data as it flows through your pipeline. Each action represents a specific data transformation unit, allowing you to perform complex transformations efficiently. The collection of actions you choose for your pipeline will inform the processing rule for your pipeline. \
This means that you can customize your pipeline to match the specific needs of your data and application, ensuring that you're getting the insights you need to make informed decisions.
You can specify processing rules for your pipeline using the user-friendly drag-and-drop interface.
You can add actions to your pipeline via the user interface
To add a processing rule to your pipeline via the user interface, select the pipeline to which you’d like to add a processing rule. From the pipeline overview page, click on the EDIT button at the top right corner.
Pipeline Overview Page
Click on the bare tile in the middle of your pipeline configuration from your Pipeline Configuration page.
Pipeline Configuration Page
You can start configuring your pipeline’s processing rule by adding an action. To add an action, click the ADD NEW tile at the right corner of your processing rule pop-up.
Processing Rules Interface
You can view a list of actions to include in your processing rule from the actions drop-down menu. Choose the actions you want to include and click on APPLY.
A list of actions you can add to a Processing Rules
You can add multiple actions to your pipeline process rules, and once you are satisfied with your configuration, select APPLY. Your pipeline will be updated immediately.
Overview of Processing Rules and How to Apply to a Pipeline
Use the toggle bar besides an Action to turn it on or off in your processing rule. This is especially helpful when testing out the transformation logic
Before applying your processing rules to your pipeline, you can test your actions by adding sample data to the Input Test box, then selecting RUN. View the result of your data transformation in the Output box.
An example of using a Processing Rule and testing the output
When using the input test, convert any text into JSON, or plain text using the toggle button beneath the input box. If you already have JSON click the toggle and paste in the data as is