Azure Blob
Official and Microsoft Certified Azure Storage Blob connector
The Azure Blob output plugin allows ingesting your records into Azure Blob Storage service. This connector is designed to use the Append Blob and Block Blob API.
Our plugin works with the official Azure Service and also can be configured to be used with a service emulator such as Azurite.
Azure Storage Account
Before getting started, make sure you already have an Azure Storage account. As a reference, the following link explains step-by-step how to set up your account:
Configuration Parameters
We expose different configuration properties. The following table lists all the options available, and the next section has specific configuration details for the official service or the emulator.
account_name
Azure Storage account name. This configuration property is mandatory
shared_key
Specify the Azure Storage Shared Key to authenticate against the service. This configuration property is mandatory.
container_name
Name of the container that will contain the blobs. This configuration property is mandatory
blob_type
Specify the desired blob type. Calyptia Fluent Bit supports appendblob and blockblob.
appendblob
auto_create_container
If container_name does not exist in the remote service, enabling this option will handle the exception and auto-create the container.
on
path
Optional path to store your blobs. If your blob name is myblob, you can specify sub-directories where to store it using path, so setting path to /logs/kubernetes will store your blob in /logs/kubernetes/myblob.
emulator_mode
off
endpoint
tls
Enable or disable TLS encryption. Note that Azure service requires this to be turned on.
off
Getting Started
As mentioned above, you can either deliver records to the official service or an emulator. Below we have an example for each use case.
Configuration for Azure Storage Service
The following configuration example generates a random message with a custom tag:
[SERVICE]
flush 1
log_level info
[INPUT]
name dummy
dummy {"name": "Calyptia Fluent Bit", "year": 2020}
samples 1
tag var.log.containers.app-default-96cbdef2340.log
[OUTPUT]
name azure_blob
match *
account_name YOUR_ACCOUNT_NAME
shared_key YOUR_SHARED_KEY
path kubernetes
container_name logs
auto_create_container on
tls onAfter you run the configuration file above, you will be able to query the data using the Azure Storage Explorer. The example above will generate the following content in the explorer:
Configuring and using Azure Emulator: Azurite
Install and run Azurite
The quickest way to get started is to install Azurite using npm:
$ npm install -g azuritethen run the service:
$ azurite
Azurite Blob service is starting at http://127.0.0.1:10000
Azurite Blob service is successfully listening at http://127.0.0.1:10000
Azurite Queue service is starting at http://127.0.0.1:10001
Azurite Queue service is successfully listening at http://127.0.0.1:10001Configuring Calyptia Fluent Bit for Azurite
Azurite comes with a default account_name and shared_key, so make sure to use the specific values provided in the example below (do an exact copy/paste):
[SERVICE]
flush 1
log_level info
[INPUT]
name dummy
dummy {"name": "Calyptia Fluent Bit", "year": 2020}
samples 1
tag var.log.containers.app-default-96cbdef2340.log
[OUTPUT]
name azure_blob
match *
account_name devstoreaccount1
shared_key Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==
path kubernetes
container_name logs
auto_create_container on
tls off
emulator_mode on
endpoint http://127.0.0.1:10000after running that Calyptia Fluent Bit configuration you will see the data flowing into Azurite:
$ azurite
Azurite Blob service is starting at http://127.0.0.1:10000
Azurite Blob service is successfully listening at http://127.0.0.1:10000
Azurite Queue service is starting at http://127.0.0.1:10001
Azurite Queue service is successfully listening at http://127.0.0.1:10001
127.0.0.1 - - [03/Sep/2020:17:40:03 +0000] "GET /devstoreaccount1/logs?restype=container HTTP/1.1" 404 -
127.0.0.1 - - [03/Sep/2020:17:40:03 +0000] "PUT /devstoreaccount1/logs?restype=container HTTP/1.1" 201 -
127.0.0.1 - - [03/Sep/2020:17:40:03 +0000] "PUT /devstoreaccount1/logs/kubernetes/var.log.containers.app-default-96cbdef2340.log?comp=appendblock HTTP/1.1" 404 -
127.0.0.1 - - [03/Sep/2020:17:40:03 +0000] "PUT /devstoreaccount1/logs/kubernetes/var.log.containers.app-default-96cbdef2340.log HTTP/1.1" 201 -
127.0.0.1 - - [03/Sep/2020:17:40:04 +0000] "PUT /devstoreaccount1/logs/kubernetes/var.log.containers.app-default-96cbdef2340.log?comp=appendblock HTTP/1.1" 201 -Last updated
