DNS Configuration Issue
Incident Report for Honeycomb
All systems now confirmed operational
Posted Nov 30, 2023 - 12:15 PST
In an attempt to make our DNS mechanism better and safer, we deployed a change that instead appears to have drastically dropped our ability to do DNS lookups.
While we don’t have a full understanding of how that happened, we have rolled back the change and everything is back to functional.

Impact of the incident:
- SLO processing was delayed by 2 minutes, but has since recovered
- Queries and triggers were significantly impacted for 12 minutes
- We had a 19 second period where 14% of ingest events were impacted
Posted Nov 30, 2023 - 12:03 PST
This incident affected: api.honeycomb.io - US1 Event Ingest, ui.honeycomb.io - US1 Querying, ui.honeycomb.io - US1 App Interface, and ui.honeycomb.io - US1 Trigger & SLO Alerting.