Google Data Studio is a visualization and reporting tool, and an integral part of Google Marketing Platform. It enables an enhanced approach to decision-making, allowing your entire organization to readily access and use marketing insights.

Data Studio lets you create beautiful, informative reports that are customizable and easy to understand and share. Empowered with the information they need, your teams can make better decisions at the exact moment they need to. With Data Studio, you can build reports and dashboards with just a few clicks.

GKE usage metering is helpful for scenarios like the following:

Tracking per-tenant resource requests and actual resource consumption in a multi tenant cluster where each tenant operates within a given Namespace.

Determining the resource consumption of a workload running in a given cluster, by assigning a unique label to the Kubernetes objects associated with the workload.

Identifying workloads whose resource requests differ significantly from their actual resource consumption so that you can more efficiently allocate resources for each workload.

This article will go through the process of setting up Google Data Studio to visualize Google Kubernetes Engine usage and cost metrics.

Prerequisites

Before you use usage metering, you must meet the following prerequisites:

1] Enable the BigQuery API for your project.

2] Create a Google Kubernetes Engine cluster v1.11.4-gke.8 or higher, v1.12.6-gke.8 or higher, or v1.13.0 or higher.

Setup Billing export for BigQuery in Google Kubernetes Engine.

1] In the BigQuery UI under your project, create a new dataset for the GKE usage metric.

2] Enable the GKE usage metering for your GKE cluster and provide the dataset which you’ve created earlier for GKE usage metric.

To verify that GKE metric is enable for your cluster run the below command in the Google cloud shell.

gcloud beta container clusters describe <cluster-name> \ --format="value(resourceUsageExportConfig)"

Setup Billing export for BigQuery in Billing account data.

1] In the BigQuery UI under your project, create a new dataset for the billing account data.

2] In the Billing section, go to the linked billing account and select your project.

2] Click on the Billing export section, select export data to BigQuery.

3] From the Project list, select the project where your BigQuery dataset which you have created earlier for billing account data.

Note: The table within the billing dataset will take up to 5 hours to appear after exporting to BigQuery and start to be populated. When the table appears, note its name.

It will be named something similar to

gcp_billing_export_v1_<var>BILLING_ACCOUNT_ID</var>

Setup the Data Studio Dashboard.

1] Enable Data Studio if it is not already enabled.

2] Create Data Source.

We need to create two separate data sources for creating a GKE usage metric dashboard.

— Replace PROJECT_ID with your project's ID (not the project name).

— Replace USAGE_METERING_DATASET with the dataset where your kubernetes cluster writes its usage metering.

— Replace BILLING_DATASET and BILLING_TABLE with the dataset and table where Cloud Billing data is exported.

1} Create a data source for BreakdownQuery.

In the Data Studio UI, create a new data source and select custom query and copy the below snippet.

SELECT

resource_usage.cluster_name,

resource_usage.cluster_location,

resource_usage.namespace,

resource_usage.labels,

resource_usage.resource_name,

resource_usage.sku_id,

resource_usage.start_time AS usage_start_time,

resource_usage.end_time AS usage_end_time,

resource_usage.usage.amount * gcp_billing_export.rate AS cost

FROM

`PROJECT_ID.USAGE_METERING_DATASET.gke_cluster_resource_usage` AS resource_usage

LEFT JOIN (

SELECT

sku.id AS sku_id,

SUM(cost) / SUM(usage.amount) AS rate,

MIN(usage_start_time) AS min_usage_start_time,

MAX(usage_end_time) AS max_usage_end_time

FROM

`PROJECT_ID.BILLING_DATASET.BILLING_TABLE`

WHERE

project.id = "PROJECT_ID"

GROUP BY

sku_id) AS gcp_billing_export

ON

resource_usage.sku_id = gcp_billing_export.sku_id

WHERE

resource_usage.start_time >= gcp_billing_export.min_usage_start_time

AND resource_usage.end_time <= gcp_billing_export.max_usage_end_time

Note: This above code joins the joins the billing data with the GKE usage metrics.

2} Create a data source for GKEProjectCostFromGCP.

In the Data Studio UI, create a new data source and select custom query and copy the below snippet.

SELECT

*

FROM

`PROJECT_ID.BILLING_DATASET.BILLING_TABLE`

WHERE

project.id = "PROJECT_ID"

3] Create the Data Studio dashboard.

We created a dashboard you can copy into your project. When you copy the dashboard, you are prompted to select three data sources you just created.

Click this dashboard link. Click the Copy button, in the top right of the screen. For the data sources, select the data sources that you created before. Use BreakdownQuery in the first field and GKEProjectCostFromGCP in the second. Click Create Report.

After the Google Data Studio report is created you can visualize the GKE usage based on different dimensions and analyze it.