Repository navigation
Handle custom histogram buckets #126
Description
Some of the Autometrics libraries (Rust, Go) already support customizing the histogram buckets and others are likely to follow suit. One implication of this is that if the library produces metrics with different than default histogram buckets, the Autometrics alerting ruleset won't work. To make it easy to handle these cases am could be able to generate and apply these custom rulesets.
Ideally am would be able to infer from the code or at runtime if custom histogram buckets are used and suggest to apply them to the Prometheus config running under the hood. Alternatively, an easier path to implement would be a flag + am.toml field that would allow to set these manually
Activity
hatchan commentedon Aug 24, 2023
This can be picked up together with #22
gagbo commentedon Aug 24, 2023
I think runtime detection is safer, because sometimes you won't be able to infer the buckets from the code (if an environment variable controls the buckets).
Also, technically the alerting ruleset will still work if you only change the buckets of the histogram, it will just be less accurate (meaning the alert might not trigger at exactly
90%or whatever SLO success rate); what will break completely the alerting rules (as "no alert would trigger at all"), is to change the percentile objective to something unsupported (so something outside90%,95%,99%, and99.9%iirc). It's still something we want to tackle for sure anyway!IvanMerrill commentedon Sep 5, 2023
I would strongly suggest that there needs to be a way to enforce a rule for there to be a bucket boundary on the SLO latency target. The loss of accuracy that @gagbo describes could be a big deal. For example:
SLO target is 95% at <= 150ms
A bucket exists for >100ms & <200ms
All responses take 110ms
The 95% percentile would be calculated as 195ms, and an alert would trigger.
Without enforcing the boundaries to match the SLO target it's easy for people to create a situation where their well-behaved code would permanently trigger an alert.