Open table of contents
Conclusion
Create a notification path that survives failures of the system it observes.
Context
When monitoring and alert delivery depend on the same cluster, an outage can silence both. Provide an external liveness check.
Design and verification scope
Assess the following responsibilities and boundaries when designing and verifying a configuration.
- External probes
- Dead-man switches
- Alert delivery
- Cluster failure
- Recovery
Decision rationale
Use the relationship between External probes and Recovery to compare the responsibilities of the selected approach and alternatives. Separate retained constraints from what the new boundary can change.
Trade-offs
Compare the implementation, maintenance and review work introduced by Dead-man switches with the control it provides. Include failure paths, operator effort and conditions in which the approach should not be adopted.
Limitations
Review observations and notifications during cluster, notification endpoint and DNS failures, including independence of the external path.
Related case context
These cases provide attributed design context. They do not establish that the proposed experiments or configurations were delivered in those engagements.