Resolved -
This incident is now resolved.
We are also noting that some other parts of honeycomb (the query UI) may have seen spurious failures as well during the impacted period (13:22—13:44 UTC).
Mar 10, 07:14 PDT
Monitoring -
We have stabilized the cluster and are monitoring the state while investigating to understand what triggered the degradation. Overall, 0.04% traffic was rejected with an error due to this degradation during the 20 minute window we were impaired.
Mar 10, 06:56 PDT
Investigating -
We are currently seeing elevated errors in connecting to some of our databases, which results in low amounts of Ingest traffic being rejected. We are looking into correcting the situation.
Mar 10, 06:50 PDT