We believe the system to be stable. Further corrective changes will be rolled out at 5PM PT later today, which may cause an extra few minutes of disruption. We choose to push them further out in the day to hopefully impact fewer people.
Posted Aug 22, 2022 - 11:56 PDT
Monitoring
Ingest has been stabilized. We've narrowed down the issue around database burden during cache invalidations and are looking for ways to make that impact manageable for the next disruption.
Posted Aug 22, 2022 - 11:20 PDT
Identified
We have noticed ingestion delays over our entire fleet. Performances seem to be stabilizing already but we are looking for ways to confirm they can remain stable overall.
Posted Aug 22, 2022 - 11:12 PDT
This incident affected: api.honeycomb.io - US1 Event Ingest.