Secrets
Learn how to use secrets in your custom pipeline configuration
When creating custom pipelines you may have settings that you do not wish to store in plain text or share with other users of the pipeline. A good way to prevent this is by using secrets within Calyptia Core.
Secrets within Calyptia Core are stored per pipeline and referenced in configuration paths via bracket. For example a secret that looks like the following
secrets
key=value
Can be referenced in a configuration via {{ secrets.key }}
Within the Pipeline Overview page you can navigate to the Advanced Settings page to modify and add secrets
In the following example we create the secret file from above in the UI
{% @arcade/embed flowId="gd3m0c9g0rgHvBhHhypr" url="https://app.arcade.software/share/gd3m0c9g0rgHvBhHhypr" %}
You can update a secret by select update within the Advanced Settings UI page.
Note that secrets can only be edited with a new value and previous values are neither displayed nor stored.
Calyptia Core stores all the secrets encrypted using RSA Public key cryptography.
Each deployment of Calyptia Core has its unique key pair (private, public) generated when the new instance registers, Calyptia Cloud does not store this private key on any form.
When a new instance of Calyptia Core gets registered, a new unique key pair is generated and the private key is stored in your operating environment. When a pipeline is deployed or updated and it requires a secret, that secret is fetched encrypted from Calyptia Core and decrypted by the keys within your environment using the in-memory private RSA key.
The following diagrams illustrate the secret generation and usage.
1067
1067
To reference secrets in the Calyptia CLI you need to create a new file in your local environment and then add them within your configuration. For example, in our
pipeline-es.conf we do not want to specify HTTP_Passwd in plain text. Instead we will use a secret with a variable es_http_passwd. This variable is defined in the secrets.env file below.pipeline-es.conf
[INPUT]
Name forward
Host 0.0.0.0
Port 24284
[OUTPUT]
Name es
Match *
Host hostname.us-east-2.es.amazonaws.com
HTTP_User es
HTTP_Passwd {{secrets.es_http_passwd}}
Port 443
TLS on
Now you need to define them in another file:
secrets.env
es_http_passwd=123123
When creating a pipeline we will need to ensure that we specify
--secrets-file such that the pipeline configuration will be able to use them. The command for that is below:calyptia create pipeline --aggregator <AGGREGATOR ID> --config-file pipeline-es.conf --secrets-file secrets.env
In the case that you need to update a secret, you can update them by first listing all secrets and then updating the secret with new values.
List all the secrets from a pipeline:
$> calyptia get pipeline_secrets --pipeline <PIPELINE ID> --show-ids
ID KEY AGO
73a941d6-e658-4f12-a175-97d9063c466f es_http_passwd 5 minutes
You can take the secret ID and use it to update its value:
calyptia update pipeline_secret <SECRET ID> <NEW VALUE>