Triggers and SLO Burn Alerts Failed
Incident Report for Honeycomb
Resolved
Missing SLO data has been backfilled. SLO calculations should be correct going forward.
Posted Jun 28, 2021 - 11:21 PDT
Update
We have a fix verified to restore the approximately 3 hours of missing SLO data, but will not be running it until tomorrow morning (28 Jun).
Posted Jun 27, 2021 - 17:36 PDT
Identified
The issue has been identified and a fix is being implemented.
Posted Jun 27, 2021 - 12:10 PDT
Investigating
Due to a database failure, many triggers and SLO burn alerts did not fire between 12:02–15:07 UTC. Customers may also notice that SLO burndown and compliance calculations are incorrect during this time window. We are actively working to correct SLO calculations and will provide an update when that process is complete. No incoming event data was lost.
Posted Jun 27, 2021 - 12:10 PDT