Processing Rules
Add powerful transformations to your pipelines
Calyptia core provides a set of Processing Rules that are built-in and ready to use. These enable you to transform your data in real time as it passes through your Telemetry pipeline. Examples can include:
Adding a new field to every record
Blocking extraneous fields
Encoding / Decoding CSV or other formats
Redacting or removing sensitive information
Processing Rules use a set of "actions" to perform transformations on the data. These actions can work with any data type, allowing you to transform various data formats.
When you use the Processing Rules, you specify one or more actions that will be applied to your data as it flows through your pipeline. Each action represents a specific data transformation unit, allowing you to perform complex transformations efficiently. The collection of actions you choose for your pipeline will inform the processing rule for your pipeline. \
This means that you can customize your pipeline to match the specific needs of your data and application, ensuring that you're getting the insights you need to make informed decisions.
You can specify processing rules for your pipeline using the user-friendly drag-and-drop interface.
Adding Processing Rules to Your Pipeline
You can add actions to your pipeline via the user interface
To add a processing rule to your pipeline via the user interface, select the pipeline to which you’d like to add a processing rule. From the pipeline overview page, click on the EDIT button at the top right corner.
Click on the bare tile in the middle of your pipeline configuration from your Pipeline Configuration page.
You can start configuring your pipeline’s processing rule by adding an action. To add an action, click the ADD NEW tile at the right corner of your processing rule pop-up.
You can view a list of actions to include in your processing rule from the actions drop-down menu. Choose the actions you want to include and click on APPLY.
You can add multiple actions to your pipeline process rules, and once you are satisfied with your configuration, select APPLY. Your pipeline will be updated immediately.
Use the toggle bar besides an Action to turn it on or off in your processing rule. This is especially helpful when testing out the transformation logic
Before applying your processing rules to your pipeline, you can test your actions by adding sample data to the Input Test box, then selecting RUN. View the result of your data transformation in the Output box.
When using the input test, convert any text into JSON, or plain text using the toggle button beneath the input box. If you already have JSON click the toggle and paste in the data as is
Last updated
