Resource Profiles
When sending data you may need to adjust particular settings to balance performance vs. Reliability. To help make this process seamless Calyptia Core includes the concept of Resource Profiles.
These templates take optimal settings for Pipelines and allow you to specify aspects around buffering, as well as resource consumption.
These resource profiles are used during Pipeline Deployment, and by default, Calyptia includes three different profile specifications
Type | Description |
|---|---|
Best Effort Low Resource
(best-effort-low-resource) | persistent storage, buffering configured, no checksum and up to a maximum of 5 MB of memory when processing backlog data. |
High-Performance Optimal Throughput
(high-performance-optimal-throughput) | minimal persistent storage, In-memory only option, no buffering. |
High-Performance Guaranteed Delivery
(high-performance-guaranteed-delivery) | larger persistent storage, buffering configured, larger memory for backlog data 256M |
As per the current version, the resource profile only supports storage configurations, memory and CPU limits can be set in the profile, but those are currently set to unlimited when setting up the pods.
Setting Name | Fluentbit Configuration | Possible values | Description |
|---|---|---|---|
StorageVolumeSize | - | Volume size in unit (Mi/Gi/Ti). | Size of the volume to be mounted in the pod for local storage processing. (GiB/MiB/TiB) |
StorageSyncFull | storage.sync | Boolean (true/false) | Use normal synchronization mode (false) or full (true) |
StorageBacklogMemLimit | storage.backlog.mem_limit | Memory in MB (i.e: 5M) | Maximum amount of memory to use when processing chunks from the backlog. |
StorageMaxChunksUp | storage.max_chunks_up | Integer > 0 | This property sets the maximum number of Chunks that can be up in memory. |
You can specify any of the above profiles above by using the following command and replace PROFILE with the reference name below
calyptia create pipeline --aggregator AGGREGATOR --resource-profile PROFILE
You can define your own resource profiles besides the 3 ones that Calyptia generates for you:
calyptia create resource_profile --aggregator AGGREGATOR --name NEW_NAME --spec JSON_FILE
The JSON file to define a resource profile follows this schema:
best-effort-low-resourcehigh-performance-optimal-throughputhigh-performance-guaranteed-delivery
{
"resources": {
"storage": {
"syncFull": false,
"backlogMemLimit": "5M",
"volumeSize": "250Mi",
"maxChunksUp": 128,
"maxChunksPause": false
},
"cpu": {
"limit": "0",
"request": "0"
},
"memory": {
"limit": "0",
"request": "0"
}
}
}
(0 means unlimited)
The resource profiles are defined within a new aggregator. You can list them with this command:
calyptia get resource_profiles --aggregator AGGREGATOR
Last modified 11d ago