GCP Operations Suite (Stackdriver)

GCP Operations Suite (Stackdriver) Destination Plugin

Google Cloud Operations Suite, formerly known as Stackdriver, is a cloud-based management platform designed to provide observability and insight into the performance, uptime, and overall health of applications and infrastructure running on Google Cloud Platform (GCP) and other cloud environments.
The suite includes a range of integrated tools such as logging, monitoring, trace, debugging, and error reporting to help developers and IT teams identify, troubleshoot, and resolve issues quickly. The platform also provides real-time metrics and alerts, allowing teams to proactively monitor and optimize their applications and services for optimal performance and reliability.
The Google Cloud Operations Suite Destination Plugin in Calyptia Core allows you to easily configure your pipeline to stream your log data and metrics to Google Cloud Operations Suite.
Configuration Parameters
The following are configuration Parameters for GCP Operations Suite (Stackdriver) Destination Plugin.
Key
Description
Google Service Credentials Path
The Service Credentials file allows Calyptia Core to communicate directly with Google Cloud Services, please read the following on how to set up service credentials: https://cloud.google.com/logging/docs/agent/logging/authorization#create-service-account
GCP Service Account Email
Account email associated to the service. Only available if no credentials file has been provided.
GCP Service Account Secret
Private key content associated with the service account. Only available if no credentials file has been provided.
Metadata Server
Private key content associated with the service account. Only available if no credentials file has been provided.
The following are Advanced configuration Parameters for GCP Operations Suite (Stackdriver) Destination Plugin.
Key
Description
Region Location
The GCP or AWS region in which to store data about the resource. If the resource type is one of the generic_node or generic_task, then this field is required.
Namespace Identifier
A namespace identifier, such as a cluster name or environment. If the resource type is one of the generic_node or generic_task, then this field is required.
Node ID Identifier
A unique identifier for the node within the namespace, such as hostname or IP address. If the resource type is generic_node, then this field is required.
Job Identifier
An identifier for a grouping of related task, such as the name of a microservice or distributed batch. If the resource type is generic_task, then this field is required.
Task ID Identifier
A unique identifier for the task within the namespace and job, such as a replica index identifying the task within the job. If the resource type is generic_task, then this field is required.
GCP Project ID
The GCP project that should receive these logs.Defaults to the project ID of the google_service_credentials file, or the project_id from Google's metadata.google.internal server.
Resource Type
Set resource type of data. Supported resource types: k8s_container, k8s_node, k8s_pod, global, generic_node, generic_task, and gce_instance.
Kubernetes Cluster Name
The name of the cluster that the container (node or pod based on the resource type) is running in. If the resource type is one of the k8s_container, k8s_node or k8s_pod, then this field is required.
Kubernetes Cluster Location
The physical location of the cluster that contains (node or pod based on the resource type) the container. If the resource type is one of the k8s_container, k8s_node or k8s_pod, then this field is required.
Labels Key
The value of this field is used by the Stackdriver output plugin to find the related labels from jsonPayload and then extract the value of it to set the LogEntry Labels.
Labels
Optional list of comma separated of strings specifying key=value pairs. The resulting labels will be combined with the elements in obtained from labels_key to set the LogEntry Labels. Elements from labels will override duplicate values from labels_key.
Log Name Key
The value of this field is used by the Stackdriver output plugin to extract logName from jsonPayload and set the logName field.
Tag Prefix
Set the tag_prefix used to validate the tag of logs with k8s resource type. Without this option, the tag of the log must be in format of k8s_container(pod/node).* in order to use the k8s_container resource type. Now the tag prefix is configurable by this option (note the ending dot).
Severity Key
Specify the name of the key from the original record that contains the severity information.
Enable Cloud Trace Integration
Rewrite the trace field to include the projectID and format it for use with Cloud Trace. When this flag is enabled, the user can get the correct result by printing only the traceID (usually 32 characters).
Custom K8s Regex
Set a custom regex to extract field like pod_name, namespace_name, container_name and docker_id from the local_resource_id in logs. This is helpful if the value of pod or node name contains dots.
The following are Security and TLS configuration Parameters for GCP Operations Suite (Stackdriver) Destination Plugin.
Key
Description
TLS
Enable or Disable TLS/SSL support
TLS Certificate Validation
Turn TLS/SSL certificate validation on / off, TLS must be on for this setting to be enabled.
TLS Debug Level
Set TLS debug verbosity level. It accepts the following values: 0 (No debug), 1 (Error), 2 (State change), 3 (Informational), and 4 Verbose
CA Certificate File Path
Absolute path to CA certificate file
Certificate File Path
Absolute path to Certificate file
Private key File Path
Absolute path to private Key file
Private Key Path Password
Optional password for tls.key_file file
TLS SNI Hostname Extension
Hostname to be used for TLS SNI Extension
The following are configuration Parameters for GCP Operations Suite (Stackdriver) Destination Plugin.
Key
Description
DNS Mode
Select the primary DNS connection type (TCP or UDP)
DNS Resolver
Select the primary DNS connection type (TCP or UDP)
Prefer IPv4
Prioritize IPv4 DNS results when trying to establish a connection
Keepalive
Enable or disable Keepalive support
Keepalive Idle Timeout
Set maximum time allowed for an idle Keepalive connection
Max Connect Timeout
Set maximum time allowed to establish a connection, this time includes the TLS handshake
Max Connect Timeout Log Error
On connection timeout, specify if it should log an error. When disabled, the timeout is logged as a debug message
Max Keepalive Recycle
Set maximum number of times a keepalive connection can be used before it is retired.
Source Address
Specify network address to bind for data traffic

Key	Description
Google Service Credentials Path	The Service Credentials file allows Calyptia Core to communicate directly with Google Cloud Services, please read the following on how to set up service credentials: https://cloud.google.com/logging/docs/agent/logging/authorization#create-service-account
GCP Service Account Email	Account email associated to the service. Only available if no credentials file has been provided.
GCP Service Account Secret	Private key content associated with the service account. Only available if no credentials file has been provided.
Metadata Server	Private key content associated with the service account. Only available if no credentials file has been provided.

Key	Description
Region Location	The GCP or AWS region in which to store data about the resource. If the resource type is one of the generic_node or generic_task, then this field is required.
Namespace Identifier	A namespace identifier, such as a cluster name or environment. If the resource type is one of the generic_node or generic_task, then this field is required.
Node ID Identifier	A unique identifier for the node within the namespace, such as hostname or IP address. If the resource type is generic_node, then this field is required.
Job Identifier	An identifier for a grouping of related task, such as the name of a microservice or distributed batch. If the resource type is generic_task, then this field is required.
Task ID Identifier	A unique identifier for the task within the namespace and job, such as a replica index identifying the task within the job. If the resource type is generic_task, then this field is required.
GCP Project ID	The GCP project that should receive these logs.Defaults to the project ID of the google_service_credentials file, or the project_id from Google's metadata.google.internal server.
Resource Type	Set resource type of data. Supported resource types: k8s_container, k8s_node, k8s_pod, global, generic_node, generic_task, and gce_instance.
Kubernetes Cluster Name	The name of the cluster that the container (node or pod based on the resource type) is running in. If the resource type is one of the k8s_container, k8s_node or k8s_pod, then this field is required.
Kubernetes Cluster Location	The physical location of the cluster that contains (node or pod based on the resource type) the container. If the resource type is one of the k8s_container, k8s_node or k8s_pod, then this field is required.
Labels Key	The value of this field is used by the Stackdriver output plugin to find the related labels from jsonPayload and then extract the value of it to set the LogEntry Labels.
Labels	Optional list of comma separated of strings specifying key=value pairs. The resulting labels will be combined with the elements in obtained from labels_key to set the LogEntry Labels. Elements from labels will override duplicate values from labels_key.
Log Name Key	The value of this field is used by the Stackdriver output plugin to extract logName from jsonPayload and set the logName field.
Tag Prefix	Set the tag_prefix used to validate the tag of logs with k8s resource type. Without this option, the tag of the log must be in format of k8s_container(pod/node).* in order to use the k8s_container resource type. Now the tag prefix is configurable by this option (note the ending dot).
Severity Key	Specify the name of the key from the original record that contains the severity information.
Enable Cloud Trace Integration	Rewrite the trace field to include the projectID and format it for use with Cloud Trace. When this flag is enabled, the user can get the correct result by printing only the traceID (usually 32 characters).
Custom K8s Regex	Set a custom regex to extract field like pod_name, namespace_name, container_name and docker_id from the local_resource_id in logs. This is helpful if the value of pod or node name contains dots.

Key	Description
TLS	Enable or Disable TLS/SSL support
TLS Certificate Validation	Turn TLS/SSL certificate validation on / off, TLS must be on for this setting to be enabled.
TLS Debug Level	Set TLS debug verbosity level. It accepts the following values: 0 (No debug), 1 (Error), 2 (State change), 3 (Informational), and 4 Verbose
CA Certificate File Path	Absolute path to CA certificate file
Certificate File Path	Absolute path to Certificate file
Private key File Path	Absolute path to private Key file
Private Key Path Password	Optional password for tls.key_file file
TLS SNI Hostname Extension	Hostname to be used for TLS SNI Extension

Key	Description
DNS Mode	Select the primary DNS connection type (TCP or UDP)
DNS Resolver	Select the primary DNS connection type (TCP or UDP)
Prefer IPv4	Prioritize IPv4 DNS results when trying to establish a connection
Keepalive	Enable or disable Keepalive support
Keepalive Idle Timeout	Set maximum time allowed for an idle Keepalive connection
Max Connect Timeout	Set maximum time allowed to establish a connection, this time includes the TLS handshake
Max Connect Timeout Log Error	On connection timeout, specify if it should log an error. When disabled, the timeout is logged as a debug message
Max Keepalive Recycle	Set maximum number of times a keepalive connection can be used before it is retired.
Source Address	Specify network address to bind for data traffic

Amazon CloudWatch Logs and Metrics

GCP Big Query

Last modified 2mo ago