2.x
Ask or search…
K

Processing Rules

Add powerful transformations to your pipelines
Calyptia Core provides a set of Processing Rules that are built-in and ready to use. These enable you to transform your data in real time as it passes through your Telemetry pipeline.
Examples can include:
  • Adding a new field to every record
  • Blocking extraneous fields
  • Encoding or decoding CSV or other formats
  • Redacting or removing sensitive information
Processing Rules use a set of actions to perform transformations on the data. These actions can work with any data type, allowing you to transform various data formats.
When you use the Processing Rules, you specify one or more actions that will be applied to your data as it flows through your pipeline. Each action represents a specific data transformation unit, allowing you to perform complex transformations efficiently. The collection of actions you choose for your pipeline will inform the processing rule for your pipeline.
This means that you can customize your pipeline to match the specific needs of your data and application, ensuring that you're getting the insights you need to make informed decisions.
You can specify processing rules for your pipeline using the drag-and-drop interface.

Adding Processing Rules to Your Pipeline

You can add actions to your pipeline by using the user interface
To add a processing rule to your pipeline with the user interface, select the pipeline to which you’d like to add a processing rule. From the pipeline overview page, click Edit in the top right corner.
Pipeline Overview Page
Click the bare tile in the middle of your pipeline configuration from your Pipeline Configuration page.
Pipeline Configuration Page
You can start configuring your pipeline's processing rule by adding an action. To add an action, click the Add New tile at the right corner of your processing rule pop-up.
Processing Rules Interface
You can view a list of actions to include in your processing rule from the actions dropdown. Choose the actions you want to include and click Apply.
A list of actions you can add to a Processing Rules
You can add multiple actions to your pipeline process rules, and after you're satisfied with your configuration, click Apply. Your pipeline will be updated immediately.
Overview of Processing Rules and How to Apply to a Pipeline
Use the toggle bar besides an Action to turn it on or off in your processing rule. This can be helpful when testing transformation logic.
Before applying your processing rules to your pipeline, you can test your actions by adding sample data to the Input Test box, then clicking Run. View the result of your data transformation in the Output box.
An example of using a Processing Rule and testing the output
When using the input test, convert any text into JSON, or plain text using the toggle button beneath the input box. If you already have JSON click the toggle and paste in the data as is.