Resource Profiles

When sending data you may need to adjust particular settings to balance performance vs. Reliability. To help make this process seamless Calyptia Core includes the concept of Resource Profiles.

These templates take optimal settings for Pipelines and allow you to specify aspects around buffering, as well as resource consumption.

These resource profiles are used during Pipeline Deployment, and by default, Calyptia includes three different profile specifications

Type

Description

Best Effort Low Resource
(best-effort-low-resource)

persistent storage, buffering configured, no checksum and up to a maximum of 5 MB of memory when processing backlog data.

High-Performance Optimal Throughput
(high-performance-optimal-throughput)

minimal persistent storage, In-memory only option, no buffering.

High-Performance Guaranteed Delivery
(high-performance-guaranteed-delivery)

larger persistent storage, buffering configured, larger memory for backlog data 256M

As per the current version, the resource profile only supports storage configurations, memory and CPU limits can be set in the profile, but those are currently set to unlimited when setting up the pods.

For extra infomration visit https://docs.fluentbit.io/manual/administration/buffering-and-storage

Setting Name

Fluentbit Configuration

Possible values

Description

StorageVolumeSize

Volume size in unit (Mi/Gi/Ti).

Size of the volume to be mounted in the pod for local storage processing. (GiB/MiB/TiB)

StorageSyncFull

storage.sync

Boolean (true/false)

Use normal synchronization mode (false) or full (true)

StorageBacklogMemLimit

storage.backlog.mem_limit

Memory in MB (i.e: 5M)

Maximum amount of memory to use when processing chunks from the backlog.

StorageMaxChunksUp

storage.max_chunks_up

Integer > 0

This property sets the maximum number of Chunks that can be up in memory.

Using a built-in Resource Profile during deployment

You can specify any of the above profiles above by using the following command and replace PROFILE with the reference name below

calyptia create pipeline --aggregator AGGREGATOR --resource-profile PROFILE

Defining a new resource profile

You can define your own resource profiles besides the 3 ones that Calyptia generates for you:

calyptia create resource_profile --aggregator AGGREGATOR --name NEW_NAME --spec JSON_FILE

The JSON file to define a resource profile follows this schema:

{
 "resources": {
  "storage": {
   "syncFull": false,
   "backlogMemLimit": "5M",
   "volumeSize": "250Mi",
   "maxChunksUp": 128,
   "maxChunksPause": false
  },
  "cpu": {
   "limit": "0",
   "request": "0"
  },
  "memory": {
   "limit": "0",
   "request": "0"
  }
 }
}
{
 "resources": {
  "storage": {
   "syncFull": false,
   "volumeSize": "128Mi",
   "maxChunksUp": 0,
   "maxChunksPause": false
  },
  "cpu": {
   "limit": "0",
   "request": "0"
  },
  "memory": {
   "limit": "0",
   "request": "0"
  }
 }
}
{
 "resources": {
  "storage": {
   "syncFull": false,
   "backlogMemLimit": "256M",
   "volumeSize": "512Mi",
   "maxChunksUp": 256,
   "maxChunksPause": false
  },
  "cpu": {
   "limit": "0",
   "request": "0"
  },
  "memory": {
   "limit": "0",
   "request": "0"
  }
 }
}

(0 means unlimited)

Listing resource profiles available

The resource profiles are defined within a new aggregator. You can list them with this command:

calyptia get resource_profiles --aggregator AGGREGATOR

Did this page help you?