Calyptia Core Agent
23.4
23.4
  • Calyptia Fluent Bit v23.04 Documentation
  • Differences with Open Source
  • Performance and Benchmarking
  • Concepts
    • Key Concepts
    • Buffering
    • Data Pipeline
      • Input
      • Parser
      • Filter
      • Buffer
      • Router
      • Output
  • Installation
    • Getting Started with Calyptia Fluent Bit
    • Supported Platforms
    • Linux Packages
      • Amazon Linux
      • Redhat / CentOS
      • Debian
      • Ubuntu
    • Docker
    • Kubernetes
    • macOS
    • Windows
    • EKS Add On
  • Administration
    • Configuring Calyptia Fluent Bit
      • Classic mode
        • Format and Schema
        • Configuration File
        • Variables
        • Commands
        • Upstream Servers
        • Record Accessor
      • YAML Configuration File
      • Unit Sizes
      • Multiline Parsing
    • Transport Security
    • Buffering & Storage
    • Backpressure
    • Scheduling and Retries
    • Networking
    • Memory Management
    • Monitoring
    • HTTP Proxy
    • Hot Reload
    • Troubleshooting
  • Local Testing
    • Validating your Data and Structure
    • Running a Logging Pipeline Locally
  • Data Pipeline
    • Inputs
      • Collectd
      • CPU Log Based Metrics
      • Disk I/O Log Based Metrics
      • Docker Log Based Metrics
      • Docker Events
      • Dummy
      • Elasticsearch
      • Exec
      • Exec Wasi
      • Fluent Bit Metrics
      • Forward
      • Head
      • HTTP
      • Kafka
      • Health
      • Kernel Logs
      • Memory Metrics
      • MQTT
      • Network I/O Log Based Metrics
      • NGINX Exporter Metrics
      • Node Exporter Metrics
      • Podman Metrics
      • Process Log Based Metrics
      • Prometheus Scrape Metrics
      • Random
      • Serial Interface
      • Standard Input
      • StatsD
      • Syslog
      • Systemd
      • Tail
      • TCP
      • Thermal
      • OpenTelemetry
      • Windows Event Log
      • Windows Event Log (winevtlog)
      • Windows Exporter Metrics
    • Parsers
      • Configuring Parser
      • JSON
      • Regular Expression
      • LTSV
      • Logfmt
      • Decoders
    • Filters
      • AWS Metadata
      • CheckList
      • ECS Metadata
      • Expect
      • GeoIP2 Filter
      • Grep
      • Kubernetes
      • Log to Metrics
      • Lua
      • Parser
      • Record Modifier
      • Modify
      • Multiline
      • Nest
      • Nightfall
      • Rewrite Tag
      • Standard Output
      • Throttle
      • Tensorflow
      • Wasm
    • Outputs
      • Amazon CloudWatch
      • Amazon Kinesis Data Firehose
      • Amazon Kinesis Data Streams
      • Amazon S3
      • Azure Blob
      • Azure Data Explorer
      • Azure Log Analytics
      • Counter
      • Datadog
      • Elasticsearch
      • File
      • FlowCounter
      • Forward
      • GELF
      • Google Cloud BigQuery
      • HTTP
      • InfluxDB
      • Kafka
      • Kafka REST Proxy
      • LogDNA
      • Loki
      • NATS
      • New Relic
      • NULL
      • Observe
      • OpenSearch
      • OpenTelemetry
      • PostgreSQL
      • Prometheus Exporter
      • Prometheus Remote Write
      • SkyWalking
      • Slack
      • Splunk
      • Stackdriver
      • Standard Output
      • Syslog
      • TCP & TLS
      • Treasure Data
      • Vivo Exporter
      • WebSocket
  • Calyptia Fluent Bit for Developers
    • Golang Output Plugins
    • WASM Filter Plugins
    • WASM Input Plugins
Powered by GitBook
On this page
  • Flexible configuration
  • Amazon Marketplace Integration
  • Prerequisites
  • Provision OIDC and IRSA
  • Enable Calyptia Core Agent add-on from AWS console
  • AWS CLI Deployment
  • Helm Chart Deployment
  • Other options
  1. Installation

EKS Add On

Using Calyptia Core Agent directly through AWS marketplace

PreviousWindowsNextConfiguring Calyptia Fluent Bit

Last updated 1 year ago

Flexible configuration

The Calyptia Core Agent builds on top of the vendor-neutral CNCF OSS Fluent Bit offering to support various inputs and outputs with inline processing too.

The default configuration of the add on provides basic output but the intention is this is customized to support whatever use case required.

Amazon Marketplace Integration

Calyptia Core Agent is provided as an either by directly running the container images or using the dedicated EKS Add On.

Prerequisites

  1. Ensure you have subscribed to the Calyptia Core Agent via .

  2. Install , and .

  3. Ensure you have access to an (or the ability to create one).

You need to replace $YOUR_CLUSTER_NAME and $AWS_REGION accordingly with your actual Amazon EKS cluster name and AWS region.

Provision OIDC and IRSA

The container requires ongoing access to the AWS Marketplace metering so to support this ensure the cluster has OIDC enabled and an appropriate IAM role attached to the service account (IRSA).

By default, the namespace and service account are set to calyptia-fluentbit. These can be changed for the Helm chart so update if appropriate but when using the EKS add on this will be the default.

Step 1: Create IAM policy

This service account needs to be assigned the AWS managed policy arn:aws:iam::aws:policy/AWSMarketplaceMeteringRegisterUsage (or an equivent one).

Here is an example IAM policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "aws-marketplace:RegisterUsage"
                ],
                "Effect": "Allow",
                "Resource": "*"
        }
    ]
}

Step 2: Ensure OIDC provider is set up

The Amazon EKS cluster needs to have the IAM OIDC provider enabled.

eksctl utils associate-iam-oidc-provider \
    --cluster "$YOUR_CLUSTER_NAME" --region "$AWS_REGION" --approve

Step 3: Create an IAM role for service account (IRSA)

The steps below help to automate these manual steps:

  1. Create an IAM role with the AWS-managed IAM policy above.

  2. Set up a trust relationship between the created IAM role with the service account automatically created by the add-on called calyptia-fluentbit.

eksctl create iamserviceaccount \
    --name calyptia-fluentbit \
    --namespace calyptia-fluentbit \
    --cluster $YOUR_CLUSTER_NAME --region $AWS_REGION \
    --attach-policy-arn arn:aws:iam::aws:policy/AWSMarketplaceMeteringRegisterUsage \
    --approve \
    --override-existing-serviceaccounts \
    --role-only \
    --role-name calyptia-fluentbit-role

Note we do not create a service account above, only the role - the service account will be created automatically by the add-on creation process (or by Helm).

Enable Calyptia Core Agent add-on from AWS console

After subscribing to the Calyptia Core Agent on AWS Marketplace and following the on-screen instructions successfully, you will be redirected to the Amazon EKS console.

To get started in the Amazon EKS console, go to your EKS clusters, and in the Add-ons tab, select Get more add-ons to find the Calyptia Core Agent EKS add-on in the cluster setting of your existing EKS clusters.

You can also use the search bar to find "Calyptia Core Agent" and follow the on-screen instructions to enable the Calyptia Core Agent add-on for your Amazon EKS clusters.

AWS CLI Deployment

Enable add-on

To enable the Calyptia Core Agent add-on for your Amazon EKS cluster run the following commands:

Please replace$ACOUNT_ID with your account ID to find the appropriate calyptia-fluent-bit role created earlier.

aws eks create-addon --addon-name calyptia_fluent-bit --cluster-name $YOUR_CLUSTER_NAME --region $AWS_REGION --service-account-role-arn arn:aws:iam::$ACCOUNT_ID:role/calyptia-fluentbit-role

To monitor the installation status, you can run the following command:

aws eks describe-addon --addon-name calyptia_fluent-bit --cluster-name $YOUR_CLUSTER_NAME --region $AWS_REGION

Disable add-on

To disable the add-on, you can run the following command:

aws eks delete-addon --addon-name calyptia_fluent-bit --cluster-name $YOUR_CLUSTER_NAME --region $AWS_REGION

Helm Chart Deployment

Create a service account associated with the appropriate policy.

eksctl create iamserviceaccount \
    --name calyptia-fluentbit \
    --namespace calyptia-fluentbit \
    --cluster $YOUR_CLUSTER_NAME --region $AWS_REGION \
    --attach-policy-arn arn:aws:iam::aws:policy/AWSMarketplaceMeteringRegisterUsage \
    --approve \
    --override-existing-serviceaccounts

Deploy the helm chart using the pre-created service account in the right region.

helm repo add calyptia https://helm.calyptia.com --force-update
helm repo update
helm upgrade --install calyptia-fluentbit calyptia/fluent-bit-eks-addon \
    --set serviceAccount.create=false,region="$AWS_REGION" \
    --wait

Calyptia Core integration

Remember to replace PROJECT_TOKEN with your actual Calyptia Core token.

helm upgrade --install calyptia/fluent-bit-eks-addon \
    --set serviceAccount.create=false,region="$AWS_REGION" \
    -set project_token="$PROJECT_TOKEN" \
    --wait

Modify version deployed

To deploy with different versions of the Calyptia Fluent Bit AWS marketplace offering, just configure the image.tag key:

helm upgrade --install calyptia-fluentbit calyptia/fluent-bit-eks-addon \
    --set serviceAccount.create=false,region="$AWS_REGION" \
    --set image.tag="23.4.5" \
    --wait

Other options

The container image can also be directly consumed in ECS or EKS pods, ensuring the IAM role is set up to allow it.

The AWS provides more details.

The AWS documentation provides more details.

AWS provides more details on how to set up these kinds of relationships .

You can learn more about direct deployment to Amazon EKS clusters from this .

Refer to the AWS documentation for the details on using the create-addon command and for an overview of role management. The service account create by the add-on has to be associated with the role created earlier so confirm the information is correct and matches your role identifier.

A is provided to deploy the EKS Add On as well as via the AWS CLI and UI.

The and tools are used below to simplify deployment.

The Helm chart also supports direct integration with by providing the token to use.

A separate helm chart is provided to deploy the AWS marketplace offering of Calyptia Fluent Bit directly. Refer to the installation instructions for the helm chart: .

AWS marketplace offering
AWS marketplace
kubectl
AWS CLI
eksctl
Amazon EKS cluster
Create and attach your first customer managed policy tutorial
Creating an IAM OIDC provider for your cluster
here
AWS blog post
here
Helm chart
eksctl
helm
Calyptia Core
https://github.com/calyptia/charts/tree/master/charts/aws-fluent-bit