Open table of contents
Conclusion
Separate short-term collection and evaluation from long-term storage and querying.
Context
Long-term metrics add retention, query and deduplication responsibilities. The boundary between collection and storage becomes central.
Design and verification scope
Assess the following responsibilities and boundaries when designing and verifying a configuration.
- Prometheus retention
- Object storage
- Compaction
- Queries
- Replica labels
- HA
- Downsampling
Decision rationale
Use the relationship between Prometheus retention and Downsampling to compare the responsibilities of the selected approach and alternatives. Separate retained constraints from what the new boundary can change.
Trade-offs
Compare the implementation, maintenance and review work introduced by Object storage with the control it provides. Include failure paths, operator effort and conditions in which the approach should not be adopted.
Limitations
Record retention, label cardinality, deduplication, queries during store failures and cost. Topology alone does not establish performance or loss rates.
Related case context
These cases provide attributed design context. They do not establish that the proposed experiments or configurations were delivered in those engagements.