GCP Big Query

GCP Big Query Destination Plugin

Google BigQuery is a fully-managed, cloud-native data warehouse that allows businesses to store, analyze, and query large datasets in a scalable and cost-effective manner. It is part of the Google Cloud Platform (GCP) suite of services.
BigQuery provides a serverless architecture, meaning that users do not need to worry about infrastructure provisioning, management, or tuning. It can process terabytes of data in seconds and petabytes of data in minutes, making it suitable for organizations that need to quickly process and analyze large amounts of data.
The GCP Big Query Suite Destination Plugin in Calyptia Core allows you to easily configure your pipeline to send your log data and metrics to GCP Big Query.
Configuration Parameters
The following are configuration Parameters for GCP Big Query Destination Plugin.
Key
Description
Google Service Credentials Path
The Service Credentials file allows Calyptia Core to communicate directly with Google Cloud Services, please read the following on how to set up service credentials: https://cloud.google.com/logging/docs/agent/logging/authorization#create-service-account
Google Project Id
The project id containing the BigQuery dataset to stream into. If the service file is provided then the project id is taken from there
Existing Data Set ID
The dataset id of the BigQuery dataset to write into. This dataset must exist in your project.
Existing Table ID
The table id of the BigQuery table to write into. This table must exist in the specified dataset and the schema must match the output.
The following are Advanced configuration Parameters for GCP Big Query Destination Plugin.
Key
Description
Skip Invalid Rows
If on then insert all valid rows of a request, even if invalid rows exist. The default value is false, which causes the entire request to fail if any invalid rows exist.
Ignore Unknown Values
Accept rows that contain values that do not match the schema. The unknown values are ignored. Default is false, which treats unknown values as errors.
Enable Workload Identity Federation
Enables workload identity federation as an alternative authentication method. Cannot be used with service account credentials file or environment variable. AWS is the only identity provider currently supported
GCP Region for BigQuery
GCP Region for BigQuery
GCP Project Number
GCP project number where the identity provider was created. Used to construct the full resource name of the identity provider.
GCP Pool Id
GCP workload identity pool where the identity provider was created. Used to construct the full resource name of the identity provider.
GCP Provider Id
GCP workload identity provider. Used to construct the full resource name of the identity provider. Currently only AWS accounts are supported.
The following are Security and TLS configuration Parameters for GCP Big Query Destination Plugin.
Key
Description
TLS
Enable or Disable TLS/SSL support
TLS Certificate Validation
Turn TLS/SSL certificate validation on / off, TLS must be on for this setting to be enabled.
TLS Debug Level
Set TLS debug verbosity level. It accepts the following values: 0 (No debug), 1 (Error), 2 (State change), 3 (Informational), and 4 Verbose
CA Certificate File Path
Absolute path to CA certificate file
Certificate File Path
Absolute path to Certificate file
Private key File Path
Absolute path to private Key file
Private Key Path Password
Optional password for tls.key_file file
TLS SNI Hostname Extension
Hostname to be used for TLS SNI Extension
The following are Advanced Networking configuration Parameters for GCP Big Query Destination Plugin.
Key
Description
DNS Mode
Select the primary DNS connection type (TCP or UDP)
DNS Resolver
Select the primary DNS connection type (TCP or UDP)
Prefer IPv4
Prioritize IPv4 DNS results when trying to establish a connection
Keepalive
Enable or disable Keepalive support
Keepalive Idle Timeout
Set maximum time allowed for an idle Keepalive connection
Max Connect Timeout
Set maximum time allowed to establish a connection, this time includes the TLS handshake
Max Connect Timeout Log Error
On connection timeout, specify if it should log an error. When disabled, the timeout is logged as a debug message
Max Keepalive Recycle
Set maximum number of times a keepalive connection can be used before it is retired.
Source Address
Specify network address to bind for data traffic

Key	Description
Google Service Credentials Path	The Service Credentials file allows Calyptia Core to communicate directly with Google Cloud Services, please read the following on how to set up service credentials: https://cloud.google.com/logging/docs/agent/logging/authorization#create-service-account
Google Project Id	The project id containing the BigQuery dataset to stream into. If the service file is provided then the project id is taken from there
Existing Data Set ID	The dataset id of the BigQuery dataset to write into. This dataset must exist in your project.
Existing Table ID	The table id of the BigQuery table to write into. This table must exist in the specified dataset and the schema must match the output.

Key	Description
Skip Invalid Rows	If on then insert all valid rows of a request, even if invalid rows exist. The default value is false, which causes the entire request to fail if any invalid rows exist.
Ignore Unknown Values	Accept rows that contain values that do not match the schema. The unknown values are ignored. Default is false, which treats unknown values as errors.
Enable Workload Identity Federation	Enables workload identity federation as an alternative authentication method. Cannot be used with service account credentials file or environment variable. AWS is the only identity provider currently supported
GCP Region for BigQuery	GCP Region for BigQuery
GCP Project Number	GCP project number where the identity provider was created. Used to construct the full resource name of the identity provider.
GCP Pool Id	GCP workload identity pool where the identity provider was created. Used to construct the full resource name of the identity provider.
GCP Provider Id	GCP workload identity provider. Used to construct the full resource name of the identity provider. Currently only AWS accounts are supported.

Key	Description
TLS	Enable or Disable TLS/SSL support
TLS Certificate Validation	Turn TLS/SSL certificate validation on / off, TLS must be on for this setting to be enabled.
TLS Debug Level	Set TLS debug verbosity level. It accepts the following values: 0 (No debug), 1 (Error), 2 (State change), 3 (Informational), and 4 Verbose
CA Certificate File Path	Absolute path to CA certificate file
Certificate File Path	Absolute path to Certificate file
Private key File Path	Absolute path to private Key file
Private Key Path Password	Optional password for tls.key_file file
TLS SNI Hostname Extension	Hostname to be used for TLS SNI Extension

Key	Description
DNS Mode	Select the primary DNS connection type (TCP or UDP)
DNS Resolver	Select the primary DNS connection type (TCP or UDP)
Prefer IPv4	Prioritize IPv4 DNS results when trying to establish a connection
Keepalive	Enable or disable Keepalive support
Keepalive Idle Timeout	Set maximum time allowed for an idle Keepalive connection
Max Connect Timeout	Set maximum time allowed to establish a connection, this time includes the TLS handshake
Max Connect Timeout Log Error	On connection timeout, specify if it should log an error. When disabled, the timeout is logged as a debug message
Max Keepalive Recycle	Set maximum number of times a keepalive connection can be used before it is retired.
Source Address	Specify network address to bind for data traffic

GCP Operations Suite (Stackdriver)

Azure Blob Storage

Last modified 2mo ago