We have identified the problem, related to a deployment interaction across components that invalidated build artifacts that were in use, which had a small cascade on their callers, causing further query issues.
We are confident that the issue is under control.
Posted Sep 23, 2021 - 13:25 PDT
Investigating
Further querying issues have surfaced. We are currently working to isolate the problem and mitigate query impact.
Posted Sep 23, 2021 - 13:04 PDT
Monitoring
The system has recovered and appears to be stable. We are still investigating to pinpoint the exact fault that was at play to prevent it from re-occuring, but everything is functional.
Posted Sep 23, 2021 - 12:54 PDT
Investigating
We have detected problems with our ability to perform queries and run triggers. Ingest and SLOs appear unaffected. We are still investigating and preemptively rolling back our latest release.
Posted Sep 23, 2021 - 12:46 PDT
This incident affected: ui.honeycomb.io - US1 Querying.