On October 20, we experienced roughly three hours of degraded service in the US region.

• From 13:30 to 14:40 UTC, querying, the Honeycomb UI, the management API, and triggers experienced elevated latency and error rates, with the highest impact between 13:55 and 14:10 UTC. During this time, customers may have experienced slow or failed queries and a sluggish UI, and some trigger alerts may have been late or missed.
• From 13:26 to approximately 16:10 UTC, SLO processing was delayed, so SLO data may have appeared out of date and burn alerts may have been delayed or missed during that time.
• From 13:57 to 14:09 UTC, about half of requests to the event ingest API received 5xx errors, with lesser impact between 13:48 and 13:55 UTC (1% error rate) and for approximately a minute at 14:15 UTC (1.9% error rate).

Telemetry rejected during these periods was not ingested unless the sender successfully retried it. Customers may see gaps in their data for those periods, including in triggers and SLOs that cover it.

Our on-call engineers were first paged at 13:43 UTC by an alert about triggers not running on time, and we spun up our internal incident response as further alerts followed.

Our investigation found that the primary database’s CPU was saturated by a large number of long-running read queries. We performed a database failover to relieve the load. This briefly eased the pressure, but the same queries soon reappeared on the new primary, so it did not resolve the problem. We then reduced database load by terminating those queries and temporarily pausing SLO processing, which restored the other affected services. Once a database-side timeout was in place on the responsible query, we resumed SLO processing, which then caught up, and we repaired historical SLO data.

These queries came from a check our SLO processing runs whenever a Honeycomb environment's settings change. In a vast majority of cases, it completes in a few milliseconds. For the 0.1% of our environments with an unusually large number of SLOs and calculated fields, however, the database ran it in a very inefficient way, and a routine update to one such environment's settings made the check too slow to ever complete. These queries continued running after the requesting service timed out, and repeated attempts increased the load on a database shared by several services.

To prevent this from happening again, in addition to the database-side timeout, we changed how the database executes the query for environments like this one, which brought its runtime down from minutes to a few milliseconds. We have also adjusted our alerting thresholds to ensure that an on-call engineer is made aware of future database load changes quickly.