In certain scenarios it would be ideal to estimate how much memory Calyptia Core Agent could be using, this is very useful for containerized environments where memory limits are a must.
In order to that we will assume that the input plugins have set the Mem_Buf_Limit option (you can learn more about it in the Backpressure section).
Input plugins append data independently, so in order to do an estimation, a limit should be imposed through the Mem_Buf_Limit option. If the limit was set to 10MB we need to estimate that in the worse case, the output plugin likely could use 20MB.
Calyptia Core Agent has an internal binary representation for the data being processed, but when this data reaches an output plugin, it will likely create its own representation in a new memory buffer for processing. The best examples are the InfluxDB and Elasticsearch output plugins, both need to convert the binary representation to their respective custom JSON formats before it can be sent to the backend servers.
So, if we impose a limit of 10MB for the input plugins and consider the worse case scenario of the output plugin consuming 20MB extra, as a minimum we need (30MB x 1.2) = 36MB.
Glibc and memory fragmentation
It is well known that in intensive environments where memory allocations happen in the orders of magnitude, the default memory allocator provided by Glibc could lead to high fragmentation, reporting a high memory usage by the service.
It's strongly suggested that in any production environment, Calyptia Core Agent should be built with jemalloc enabled (e.g. -DFLB_JEMALLOC=On). Jemalloc is an alternative memory allocator that can reduce fragmentation (among others things) resulting in better performance.
You can determine if Calyptia Core Agent has been built with Jemalloc using the following command: