Prometheus

References

https://prometheus.io/docs/concepts/data_model/
https://prometheus.io/docs/concepts/metric_types/
https://prometheus.io/docs/concepts/jobs_instances/
https://prometheus.io/docs/prometheus/latest/querying/basics/
https://satyanash.net/software/2021/01/04/understanding-prometheus-range-vectors.html
https://prometheus.io/docs/prometheus/latest/querying/functions/#predict_linear

 

Jobs and instances

In Prometheus terms, an endpoint you can scrape is called an instance, usually corresponding to a single process. A collection of instances with the same purpose, a process replicated for scalability or reliability for example, is called a job.

 

Metric names and labels

Prometheus fundamentally stores all data as time series: streams of timestamped values belonging to the same metric and the same set of labeled dimensions. Besides stored time series, Prometheus may generate temporary derived time series as the result of queries.

Every time series is uniquely identified by its metric name and optional key-value pairs called labels.

The change of any label's value, including adding or removing labels, will create a new time series.

 

Samples

Samples form the actual time series data. Each sample consists of:

• a float64 or native histogram value
• a millisecond-precision timestamp

 

Metric

Prometheus metrics are numerical data points used to monitor the performance and behavior of systems over time. They are collected as time-series data, allowing users to analyze trends and identify issues in their applications and infrastructure.

The Prometheus client libraries offer four core metric types. These are currently only differentiated in the client libraries (to enable APIs tailored to the usage of the specific types) and in the wire protocol. The Prometheus server does not yet make use of the type information and flattens all data into untyped time series. This may change in the future.

• Counter - A counter is a cumulative metric that represents a single monotonically increasing counter whose value can only increase or be reset to zero on restart.
• Gauge - A gauge is a metric that represents a single numerical value that can arbitrarily go up and down.
• Histogram - A histogram samples observations (usually things like request durations or response sizes) and counts them in configurable buckets. It also provides a sum of all observed values.
• Summary - Similar to a histogram, a summary samples observations (usually things like request durations and response sizes). While it also provides a total count of observations and a sum of all observed values, it calculates configurable quantiles over a sliding time window.

 

Querying

When you send a query request to Prometheus, it can be an instant query, evaluated at one point in time, or a range query at equally-spaced steps between a start and an end time.

Expression language data types:

In Prometheus's expression language, an expression or sub-expression can evaluate to one of four types:

• Instant vector - a set of time series containing a single sample for each time series, all sharing the same timestamp
• Range vector - a set of time series containing a range of data points over time for each time series
• Scalar - a simple numeric floating point value
• String - a simple string value; currently unused

Depending on the use case (e.g. when graphing vs. displaying the output of an expression), only some of these types are legal as the result of a user-specified expression. For instant queries, any of the above data types are allowed as the root of the expression. Range queries only support scalar-typed and instant-vector-typed expressions.

Vector

Since Prometheus is a timeseries database, all data is in the context of some timestamp. The series that maps a timestamp to recorded data is called a timeseries. In Prometheus lingo, a set of related timeseries is called a vector. Vectors allow us to specify further dimensions called "labels" so that we can mark data.

Prometheus further defines two types of vectors, depending on the what the timestamps map to:

Instant vector

• A set of timeseries where every timestamp maps to a single data point at that "instant".

Range vector

• A set of timeseries where every timestamp maps to a "range" of data points, recorded some duration into the past. These cannot exist without a specified duration called the "range", which is used to build the list of values for every timestamp.

With all the definitions in place, we establish two ideas regarding these vector types:

• Instant vectors can be charted; Range vectors cannot. This is because charting something involves displaying a data point on the y-axis for every timestamp on the x-axis. Instant vectors have a single value for every timestamp, while range vectors have many of them. For the purpose of charting a metric, it is undefined how to show multiple data points for a single timestamp in a timeseries.
• Instant vectors can be compared and have arithmetic performed on them; Range vectors cannot. This is also due to the way comparison and arithmetic operators are defined. For every timestamp, if we have multiple values, we don't know how to add or compare them to another timeseries of a similar nature.

One can convert a range vector into a single vector by appending a range selection [5m] to an instance vector, which produces a range vector, and then using a function to subtract the first data poing from the last data point, like "increase()": increase(http_requests_total{code="200",handler="/api/v1/query"}[15m]) produces an instance vector which can be charted.

Time Series Selectors

Instant Vector Selectors

Instant vector selectors allow the selection of a set of time series and a single sample value for each at a given timestamp (point in time). In the simplest form, only a metric name is specified, which results in an instant vector containing elements for all time series that have this metric name.

The value returned will be that of the most recent sample at or before the query's evaluation timestamp (in the case of an instant query) or the current step within the query (in the case of a range query).

arista_interface_counter{counter="octets",direction="out",interface_name="Port-Channel8",target="r3"}

arista_interface_counter is the metric name
counter, interface_name, target, are the labels

 

Range Vector Selectors

Range vector literals work like instant vector literals, except that they select a range of samples back from the current instant.

The range is a left-open and right-closed interval, i.e. samples with timestamps coinciding with the left boundary of the range are excluded from the selection, while samples coinciding with the right boundary of the range are included in the selection.

In this example, we select all the values recorded less than 5m ago for all time series that have the metric name arista_interface_counter:

arista_interface_counter{counter="octets",direction="out",interface_name="Port-Channel8",target="r3"}[5m]

 

Subquery

Subquery allows you to run an instant query for a given range and resolution. The result of a subquery is a range vector. Subqueries are run against a Scalar or instant vector. Subqueries use the syntax [<range> : <resolution>]. Range is required, and it states how far back to collector data from. Resolutions defaults to the scrape interval is not specified (it's optional). All data point from now - range until now will be pulled at $resolution intervals.

Assume the scrap interval is 1 minute.

The following returns an instant vector (from the last/most recent data point): arista_interface_counter{...}

The following returns a range vector which starts 20 minutes ago, pulling every data point since then (data at every scrape interval): arista_interface_counter{...}[20m]

The following returns a range vector starting 20 minutes ago, pulling every other data point (resolution is specified as 2m, instead of default scrape interval of 1m): arista_interface_counter{...}[20m:2m]

The following takes a range vector of all data points starting 10 minutes ago (every 1 minute, default resolution == scrape interval) and calculates the average rate of increase, returning an instance vector: rate(arista_interface_counter{...}[10m])

The following takes a range vector of data points starting 10 minutes ago, taking every other data point (2m resolution), calculates the average rate of increase and returns an instant vector. This process is repeated starting 5 minutes ago, 4 minutes ago, 3 minutes ago, and so on (5 mintues ago in 1 minute intervals) which produces a range vector (the average rate of increase for the "last 10 minutes" starting at 5 different times). This [5m:1m] is the subquery that produces a range vector from a series of instant vectors: rate(arista_interface_counter{...}[10m:2m])[5m:1m]

Offset modifier

The offset modifier allows changing the time offset for individual instant and range vectors in a query.

This returns the 5-minute rate that http_requests_total had a week ago:

rate(http_requests_total[5m] offset 1w)

When querying for samples in the past, a negative offset will enable temporal comparisons forward in time:

rate(http_requests_total[5m] offset -1w)

Note that this allows a query to look ahead of its evaluation time.

 

@ modifier

The @ modifier allows changing the evaluation time for individual instant and range vectors in a query. The time supplied to the @ modifier is a Unix timestamp and described with a float literal.

For example, the following expression returns the value of http_requests_total at 2021-01-04T07:40:00+00:00:

http_requests_total @ 1609746000

http_requests_total @ 1609746000 offset 5m

 

predict_linear

predict_linear(v range-vector, t scalar) predicts the value of time series t seconds from now, based on the range vector v, using simple linear regression. The range vector must have at least two float samples in order to perform the calculation. When +Inf or -Inf are found in the range vector, the predicted value will be NaN.

predict_linear should only be used with gauges and only works for float samples. Elements in the range vector that contain only histogram samples are ignored entirely. For elements that contain a mix of float and histogram samples, only the float samples are used as input, which is flagged by an info-level annotation.

predict_linear((irate(arista_interface_counter{target="r3", interface_name="Port-Channel8", counter="octets", direction="out"}[2m])*8)[1d:5m], 1d)

predict_linear((irate(arista_interface_counter{target="r3", interface_name="Port-Channel8", counter="octets", direction="out"}[2m])*8)[15m:2m], 1m)

---