Resource Profiles

When sending data you may need to adjust particular settings to balance performance vs. Reliability. To help make this process seamless Calyptia Core includes the concept of Resource Profiles.

These templates take optimal settings for Pipelines and allow you to specify aspects around buffering, as well as resource consumption.

These resource profiles are used during Pipeline Deployment, and by default, Calyptia includes three different profile specifications

Type	Description
Best Effort Low Resource (best-effort-low-resource)	persistent storage, buffering configured, no checksum and up to a maximum of 5 MB of memory when processing backlog data.
High-Performance Optimal Throughput (high-performance-optimal-throughput)	minimal persistent storage, In-memory only option, no buffering.
High-Performance Guaranteed Delivery (high-performance-guaranteed-delivery)	larger persistent storage, buffering configured, larger memory for backlog data 256M

As per the current version, the resource profile only supports storage configurations, memory and CPU limits can be set in the profile, but those are currently set to unlimited when setting up the pods.

For extra infomration visit https://docs.fluentbit.io/manual/administration/buffering-and-storage

Setting Name	Fluentbit Configuration	Possible values	Description
StorageVolumeSize	-	Volume size in unit (Mi/Gi/Ti).	Size of the volume to be mounted in the pod for local storage processing. (GiB/MiB/TiB)
StorageSyncFull	storage.sync	Boolean (true/false)	Use normal synchronization mode (false) or full (true)
StorageBacklogMemLimit	storage.backlog.mem_limit	Memory in MB (i.e: 5M)	Maximum amount of memory to use when processing chunks from the backlog.
StorageMaxChunksUp	storage.max_chunks_up	Integer > 0	This property sets the maximum number of Chunks that can be up in memory.

Using a built-in Resource Profile during deployment

You can specify any of the above profiles above by using the following command and replace PROFILE with the reference name below

calyptia create pipeline --aggregator AGGREGATOR --resource-profile PROFILE

Defining a new resource profile

You can define your own resource profiles besides the 3 ones that Calyptia generates for you:

calyptia create resource_profile --aggregator AGGREGATOR --name NEW_NAME --spec JSON_FILE

The JSON file to define a resource profile follows this schema:

best-effort-low-resourcehigh-performance-optimal-throughputhigh-performance-guaranteed-delivery

{
 "resources": {
  "storage": {
   "syncFull": false,
   "backlogMemLimit": "5M",
   "volumeSize": "250Mi",
   "maxChunksUp": 128,
   "maxChunksPause": false
  },
  "cpu": {
   "limit": "0",
   "request": "0"
  },
  "memory": {
   "limit": "0",
   "request": "0"
  }
 }
}

(0 means unlimited)

Listing resource profiles available

The resource profiles are defined within a new aggregator. You can list them with this command:

calyptia get resource_profiles --aggregator AGGREGATOR