Secrets

Learn how to use secrets in your pipeline configuration

How are secrets stored

Calyptia Core stores all the secrets encrypted using RSA Public key cryptography.

Each deployment of Calyptia Core has its unique key pair (private, public) generated when the new instance registers, Calyptia Cloud does not store this private key on any form.

When a new instance of Calyptia Core gets registered, a new unique key pair is generated and the private key is stored in the local cluster as an opaque secret. When a pipeline is deployed or updated and it requires a secret, that secret is fetched encrypted from Calyptia Cloud and decrypted by the CEFB instance using the in-memory private RSA key.

The following diagrams illustrate the secret generation and usage.

1067

1067

Using Secrets

To reference secrets you need to create a new file in your local environment and then add them within your configuration. For example, in our pipeline-es.conf we do not want to specify HTTP_Passwd in plain text. Instead we will use a secret with a variable es_http_passwd. This variable is defined in the secrets.env file below.

pipeline-es.conf

[INPUT]
    Name          forward
    Host          0.0.0.0
    Port          24284

[OUTPUT]
    Name        es
    Match       *
    Host        hostname.us-east-2.es.amazonaws.com
    HTTP_User   es
    HTTP_Passwd {{secrets.es_http_passwd}}
    Port        443
    TLS         on

Now you need to define them in another file:

secrets.env

es_http_passwd=123123

Creating a pipeline with a secret

When creating a pipeline we will need to ensure that we specify --secrets-file such that the pipeline configuration will be able to use them. The command for that is below:

calyptia create pipeline --aggregator <AGGREGATOR ID> --config-file pipeline-es.conf --secrets-file secrets.env

Update a Secret's Value

In the case that you need to update a secret, you can update them by first listing all secrets and then updating the secret with new values.

List all the secrets from a pipeline:

$> calyptia get pipeline_secrets --pipeline <PIPELINE ID> --show-ids
ID                                   KEY            AGO
73a941d6-e658-4f12-a175-97d9063c466f es_http_passwd 5 minutes

You can take the secret ID and use it to update its value:

calyptia update pipeline_secret <SECRET ID> <NEW VALUE>