We now feel confident service is operational at its usual levels.
Posted Apr 21, 2023 - 08:45 PDT
Monitoring
We believe the beta feature in question being turned off brought back all expected stability. We are seeing ingest looking healthy, and if it keeps stable for another half hour, we'll consider this degradation to be over.
Posted Apr 21, 2023 - 08:13 PDT
Update
Our investigation has led us toward a beta feature that was turned on for a subset of customers and that may have caused instability. Until our subject matter experts are available, we are going to turn that beta feature off to see if it re-stabilizes ingest in general.
Posted Apr 21, 2023 - 07:54 PDT
Update
We have identified two potential independent sources of instability. We're currently rolling out a mitigation against one of them in an attempt to bring stability closer to its expected level and are shifting our attention to properly identifying the other source.
Posted Apr 21, 2023 - 06:47 PDT
Investigating
We have noticed a trickle of errors on the ingest pipeline that goes a bit above our target error budget over the last few hours, even if it's still functional and won't show errors for most customers. We are looking for ways to bring performance back to normal.
Posted Apr 21, 2023 - 06:00 PDT
This incident affected: api.honeycomb.io - US1 Event Ingest.