One of the more common patterns for Fluent Bit and Fluentd is deploying in what is known as the forwarder/aggregator pattern. This pattern includes having a lightweight instance deployed on edge, generally where data is created, such as Kubernetes nodes or virtual machines. These forwarders do minimal processing and then use the forward protocol to send data to a dedicated instance. This dedicated instance, known as the aggregator, may perform more filtering and processing before routing to the appropriate backend(s).
When using Calyptia Core, one function is that of an aggregator. You can route logs, metrics, and security events through this instance and leverage the benefits of scaling, auto-healing, and high availability without having to manually configure this yourself. Sample use cases include routing messages to different endpoints depending on different message values, adding fields to every message sent, or redacting values for privacy/security concerns.
- Less resource utilization on the edge devices (maximize throughput)
- Allow processing to scale independently on the aggregator tier.
- Easy to add more backends (configuration change in aggregator vs. all forwarders)
- Dedicated resources required for an aggregation instance
Additionally, with this architecture you can also receive incoming network traffic from firewalls, network devices and other applications without requiring a lightweight instance deployed on the edge. Example use cases retrieving data from non-supported OSes (Solaris, AIX, HP-UX), filtering unnecessary data from high throughput firewalls, and enriching or processing data prior to sending to a final destination.
Updated 28 days ago