How can I group labels in a Prometheus query?

If I have a metric with the following labels:

my_metric{group="group a"} 100
my_metric{group="group b"} 100
my_metric{group="group c"} 100
my_metric{group="misc group a"} 1
my_metric{group="misc group b"} 2
my_metric{group="misc group c"} 1
my_metric{group="misc group d"} 1

Is it possible to do a query or even a label_replace that combines the 'misc' groups together?

(I realize that the metric cardinality needs to be improved, and I've updated the app to fix it. However it left me with this question for if I wanted to fix the metrics via a query later)

3 Answers

Yes, you can you use label replace to group all the misc together:

sum by (new_group) ( label_replace( label_replace(my_metric, "new_group", "$1", "group", ".+"), "new_group", "misc", "group", "misc group.+" )
)

The inner label_replace copies all values from group into new_group, the outer overwrites those which match "misc group.+" with "misc", and we then sum by the "new_group" label. The reason for using a new label is the series would no longer be unique if we just overwrote the "group" label, and the sum wouldn't work.

It's even easier

sum by (group) (my_metric)

You can use a regex query:

my_metric{group=~"misc group.+"}

That will give you everything where group starts with "misc group".