Basics of Operator Metering Reports

Let’s start by describing some basics of Operator Metering reports, from data sources, to report generations, and scheduled reports. These objects are all expressed using YAML and stored as definitions using the Kubernetes command line. Starting with data source, think of these as tables that are defined based on Prometheus queries. Operator Metering comes with some predefined report generations, for example “namespace-cpu-request”, which can also be used as a data source.

Defining kube-pod-labels data source

If a data source doesn’t exist you can define one as seen in our example where we are creating a pod label data source which provides us access to the pod labels found in Prometheus.

Report generations are essentially the SQL SELECT statements that are going to populate your CSV or JSON output. In a report generation you define a name, the data sources, the columns with type and units and lastly the query against the defined data sources that obtains the records with the defined columns. In the example below you can see the definition of the report generation with its name, columns, and query; here you can see how it gathers the pod, namespace (or project), node, and CPU core limit for the current time interval.

Report Generation for “pod-cpu-limit-raw”

Now we’ve walked through data sources and report generations you can see how they build upon each other, however, we don’t have an executing report yet. Operator Metering supports the concept of both one-time (or on-demand) reports and scheduled reports.

Defining a hourly scheduled report

These reports can have a defined start and end period, if you are defining a one-time report you would specify “runImmediately: true”, for scheduled reports you specify it as such with a period like hourly, daily, etc. In the example on the left you can see that no start or end period has been supplied, which means this report will run indefinitely. The concept of an indefinitely running report is appealing, however, the drawback of not specifying a start and end period is that when requesting results of a report you may hit performance limits as the size of the data requested grows continuously.

Combining Data in Reports

As I mentioned above, Operator Metering is simple and usable. If you have any familiarity with SQL you can probably intuit how to combine data. I briefly mentioned in the last section that a report generation can be used as a data source for another report generation; you can also supply multiple data sources to be used within a report generation. Continuing with the view that a data source is just a table then you would correctly infer that you can JOIN data from these tables, which can be seen in this example report generation which is built using nine previously defined report queries.

Aggregation and Windowing

With the power to combine data, one can see that report sizes can grow very quickly if they need to be gathered at an hourly rate. The growth of data could eventually lead to performance issues, and for some of the views you may not need the fine grain hourly data but be more interested in something like an average or maximum, Operator Metering supports roll-up reports to provide just this functionality.

However, if you do need a fine grain level of detail your initial approach maybe to create blocks of reports with defined start and end periods to limit the amount of data that will be returned for a report. An approach with blocks of reports is feasible, however, some naming scheme would be needed in order to access the data, additionally you would need to continue to create report blocks as time marches forward, this could be automated but also leaves opportunity for mistakes and missing data. For our use case, the development team came upon the concept of windowing data with a look back, which comes with the additional performance benefit of essentially constructing a view on an existing table.