We continue to work toward restoring the Activity Log. We’ve made meaningful progress on the underlying issues, but have had to pivot on our remediation plan due to previously unknown hard retention limits. Our engineering team is working through the next steps required to restore Activity Log processing. Next updates to come during Monday US business hours
Posted Dec 12, 2025 - 15:20 PST
Update
We continue to work toward restoring the Activity Log. We’ve made meaningful progress on the underlying issues, but have had to pivot on our remediation plan due to previously unknown hard retention limits. Our engineering team is actively working through the next steps required to restore Activity Log processing.
Posted Dec 11, 2025 - 16:09 PST
Update
We continue to work toward fully restoring the Activity Log. We’ve made meaningful progress on the underlying issues and continue to work through our remediation plan. Our engineering team is actively working through the next steps required to restore Activity Log processing.
Posted Dec 11, 2025 - 10:02 PST
Update
We continue to work toward fully restoring the Activity Log. While functionality has not yet improved, we’ve made progress on the underlying issue and have begun the next phase of remediation. Our engineering team is pausing for the evening and will resume working through the remaining steps required to resume Activity Log processing during US business hours tomorrow. We will provide another update tomorrow morning.
Posted Dec 10, 2025 - 17:08 PST
Update
We continue to work toward fully restoring the Activity Log. While functionality has not yet improved, we’ve made progress on the underlying issue and have begun the next phase of remediation. Our engineering team is actively working through the remaining steps required to resume Activity Log processing.
Posted Dec 10, 2025 - 09:34 PST
Update
We continue to work toward fully restoring the Activity Log. While functionality has not yet improved, we have a clear path to recovery, and remediation work is underway. Our next steps involve coordinated infrastructure changes that will allow us to resume Activity Log processing. We’ll provide another update during US business hours Wednesday.
Posted Dec 09, 2025 - 14:40 PST
Update
We are continuing to work on a fix for this issue.
Posted Dec 08, 2025 - 18:05 PST
Update
We fixed the issue that was causing subsets of temporarily missing data in queries against certain partitions. We continue to work on restoring full functionality to Activity Log. We'll provide another update during US business hours Tuesday.
Posted Dec 08, 2025 - 18:01 PST
Update
We continue to work on restoring full functionality to Activity Log and repair the partition causing certain customers to have subsets of unqueryable data. We will next provide an update during US business hours Tuesday.
Posted Dec 08, 2025 - 17:02 PST
Update
The Activity Log is operational but is experiencing ingestion delays, which means recent activity may not be reflected immediately. We are working to restore full functionality. Currently, a small subset of data for certain customers may not be queryable due to a partition needing repair. We are working to return the data to queryability. After investigation of the impact starting on Friday to SLO and Trigger functionality, we're providing the update below with the details.
Overview SLOs and Triggers were affected by Kafka and ingest outages beginning Friday, December 5 at 8:40 PM UTC. Trigger evaluations are now behaving normally except for some baseline triggers. SLOs with evaluation periods that include missing ingest or are on the affected partition will reflect that currently unqueryable data in their ongoing calculations. SLOs - No burn alerts were fired between December 5, 2025 at 8:40 PM UTC and December 6, 2025 at 7:15 PM UTC due to a processing delay. - Now that ingest is restored, burn alerts should behave normally again minus a subset that are also impacted by the small amount of currently unqueryable data. Triggers - When ingest was down on December 6, no events were received, so all EU Trigger evaluations were impacted. - Baseline Triggers with lookback windows that query data in the affected partition returning will also be affected by the small amount of currently unqueryable data. - Non-baseline triggers (which can only look back 24 hours) should behave normally going forward.
We will provide an update at the end of the workday (5PM PT) or as the situation changes.
Posted Dec 08, 2025 - 13:52 PST
Update
Our Kafka cluster is still operational and we are continuing to work towards restoring full reliability. We will resume work during business hours to restore full Activity Log functionality. We will continue to update as work progresses.
Posted Dec 07, 2025 - 12:23 PST
Update
The Kafka cluster is still operational, but is not back to full capacity and resiliency. Work will continue tomorrow to ensure sufficient capacity, as well as during business hours to resume full Activity Log functionality. We will provide more updates tomorrow and as-needed if there are any changes to availability.
Posted Dec 06, 2025 - 18:18 PST
Update
We have gotten the Kafka cluster to a stable-enough state, however it will still require remediation before the Activity Log is fully functional and it is back to a level of service sustainable until business hours. We will provide another update in 2 hours or as the situation changes.
Posted Dec 06, 2025 - 16:10 PST
Update
Ingest has been fully re-enabled, and SLOs and Triggers are now up-to-date.
Posted Dec 06, 2025 - 14:08 PST
Identified
We have resumed ingest to api.eu1.honeycomb.io as we were able to restore partial service to the Kafka cluster.
Posted Dec 06, 2025 - 13:51 PST
Update
We are continuing to investigate issues with the Kafka cluster. We will continue providing updates every 2 hours unless there are significant changes.
Posted Dec 06, 2025 - 13:23 PST
Update
We are continuing to investigate issues with the Kafka cluster.
Posted Dec 06, 2025 - 11:20 PST
Update
External API access is restored, event ingestion is still impacted
Posted Dec 06, 2025 - 08:15 PST
Update
We have temporarily disabled event ingestion for the EU Region, in service of restoring full functionality to our EU Kafka fleet. External API access will also be disabled. Additionally, during the ongoing outage, our Service Level Objectives feature has been impacted and down since 12:30 PM Pacific time on Friday, December 5th. SLO data will not be correct until our systems catch up and we rebuild the cache
Posted Dec 06, 2025 - 07:19 PST
Update
Automated systems are still working to catch up and Activity Log will remain offline until that that completes. We will post another update Saturday whether or not the Activity Log outage has been remediated.
Posted Dec 05, 2025 - 23:30 PST
Update
Ingest, querying, SLOs, and Trigger alerting is back to normal. Activity Log is still impacted, we have identified the cause and are working to resolve. We are still evaluating the scope of the outage for ingest and expect to have a full answer for that posted during US business hours on Monday. We will post at least one more update this evening.
Posted Dec 05, 2025 - 16:04 PST
Update
We are continuing the investigation after business hours to stabilize ingest and determine what work will be needed to fully recover. Known impact will be updated here as the situation changes. We will post at least one more update this evening.
Posted Dec 05, 2025 - 15:46 PST
Update
We have identified that 0.23% of datasets are fully affected, and a larger percentage are seeing intermittent ingestion and query failures (500s at the API level). We are also investigating a replication error in our ingestion pipeline.
Posted Dec 05, 2025 - 14:33 PST
Update
A subset of customer environments may see higher than usual error rates when sending events to api.eu1.honeycomb.io, and notifications for SLOs and Triggers may be delayed for that subset. We are continuing to investigate the issue.