Resource profiles

When sending data you may need to adjust particular settings to balance performance versus reliability. To help make this process seamless Calyptia Core includes the concept of Resource Profiles.

These templates take optimal settings for Pipelines and allow you to specify aspects around buffering, and resource consumption.

These resource profiles are used during Pipeline Deployment, and by default, Calyptia includes three different profile specifications.

Type	Description
Best Effort Low Resource (best-effort-low-resource)	Persistent storage, buffering configured, no checksum and up to a maximum of 5 MB of memory when processing backlog data.
High-Performance Optimal Throughput (high-performance-optimal-throughput)	Minimal persistent storage, In-memory only option, no buffering.
High-Performance Guaranteed Delivery (high-performance-guaranteed-delivery)	Larger persistent storage, buffering configured, larger memory for backlog data 256M.

As per the current version, the resource profile only supports storage configurations, memory and CPU limits can be set in the profile, but those are currently set to unlimited when setting up the pods.

For extra information, visit https://docs.fluentbit.io/manual/administration/buffering-and-storage

Setting Name	FluentBit Configuration	Possible values	Description
StorageVolumeSize	-	Volume size in unit (Mi/Gi/Ti).	Size of the volume to be mounted in the pod for local storage processing. (GiB/MiB/TiB)
StorageSyncFull	storage.sync	Boolean (true/false)	Use normal synchronization mode (false) or full (true)
StorageBacklogMemLimit	storage.backlog.mem_limit	Memory in MB (i.e: 5M)	Maximum amount of memory to use when processing chunks from the backlog.
StorageMaxChunksUp	storage.max_chunks_up	Integer > 0	This property sets the maximum number of Chunks that can be up in memory.

Using a built-in Resource Profile during deployment

You can specify any of the previously mentioned profiles by using the following command, and replace PROFILE with the following reference name:

calyptia create pipeline --core-instance CORE_INSTANCE --resource-profile PROFILE

Replace CORE_INSTANCE with the unique ID or name of the Calyptia Core instance.

Defining a new resource profile

You can define your own resource profiles besides the three that Calyptia generates for you:

calyptia create resource_profile --core-instance CORE_INSTANCE --name NEW_NAME --spec JSON_FILE

Replace CORE_INSTANCE with the unique ID or name of the Calyptia Core instance.

The JSON file to define a resource profile follows this schema:

best-effort-low-resourcehigh-performance-optimal-throughputhigh-performance-guaranteed-delivery

{
 "resources": {
  "storage": {
   "syncFull": false,
   "backlogMemLimit": "5M",
   "volumeSize": "250Mi",
   "maxChunksUp": 128,
   "maxChunksPause": false
  },
  "cpu": {
   "limit": "0",
   "request": "0"
  },
  "memory": {
   "limit": "0",
   "request": "0"
  }
 }
}

(0 means unlimited)

Listing resource profiles available

The resource profiles are defined within a new core instance. You can list them with this command:

calyptia get resource_profiles --core-instance CORE_INSTANCE

Replace CORE_INSTANCE with the unique ID or name of the Calyptia Core instance.