Rethinking UX for AI-driven alerting
Alerting UX is shifting from manually configured static thresholds toward statistical systems that understand trends, seasonality, and group behavior. Forecasting, anomaly detection, and outlier detection reduce maintenance and nuisance alerts, while algorithmic feeds can discover problems users never explicitly configured. The post argues that supervised feeds—trained by user feedback—could become the most significant evolution in monitoring. ## Traditional Alerting UX Most alerts are built from four dimensions: - **Scope:** The hosts, containers, services, or other targets being monitored. - **Metric:** The value tracked, such as free disk space. - **Thresholds:** Static warning or critical values that trigger alerts. - **Time:** A duration or time window during which the condition must occur. Static threshold alerts remain common, with many taking the form: “If free disk space equals zero, alert.” ## Problems with Static Thresholds - Static thresholds do not adapt to system growth, changing workloads, or temporary events such as holiday traffic. - They require regular review and maintenance to remain useful. - Warning thresholds often serve as manual “heads-ups” so engineers can inspect a graph and estimate whether intervention is necessary. - Large numbers of warning alerts create false positives and alert fatigue. - Monitoring systems must be explicitly told which scopes and metrics to watch, leading to duplicated configuration and ongoing maintenance. ## Algorithmic Alerting Statistical alerting introduces three primary methods: - **Forecasting** - Uses historical data to predict when a metric will cross a threshold. - Changes “alert when disk reaches zero” into “alert if disk will reach zero within 24 hours.” - Lets teams specify how much remediation time they need. - Can eliminate separate warning thresholds because the forecast provides advance notice. - **Anomaly detection** - Predicts what should be happening now based on historical behavior. - Considers configurable confidence intervals and seasonality, such as daily or weekly patterns. - Alerts when current behavior deviates significantly from the expected range. - **Outlier detection** - Compares members of a group that should behave similarly. - Flags an individual server or service whose behavior differs from its peers. - Does not depend on historical behavior. These methods make thresholds and time behavior more flexible, but they still require users to define the metrics and scopes in advance. ## Algorithmic Feeds Algorithmic feeds apply similar statistical techniques without requiring detailed alert configuration. - They can monitor systems without predefined individual scopes or metrics. - They are especially useful for unpredictable anomalies and outliers. - Examples include Slack Highlights and Datadog Watchdog. - Feeds shift monitoring from **opt-in alerting**—where users specify what to watch—to discovering noteworthy activity automatically. - The post presents this shift as potentially the largest change in alerting UX, while noting that algorithmic feeds are still immature. ## Supervised Algorithmic Feeds Once a monitoring system generates a stream of events, user feedback can help train it to surface more relevant information. - The model is compared to social media feeds, where actions such as “likes” guide future recommendations. - This suggests a future in which engineers can teach monitoring systems which anomalies and events matter to them. - The provided excerpt ends while introducing this concept, so it does not describe the specific feedback mechanisms or implementation details. Monitoring is likely moving toward adaptive systems that combine statistical detection with user-guided prioritization. Teams should use forecasting and anomaly-based alerts where appropriate, while treating algorithmic feeds as a complementary way to discover issues outside manually configured monitoring.
Read original(opens in new tab)