In an attempt to make our DNS mechanism better and safer, we deployed a change that instead appears to have drastically dropped our ability to do DNS lookups.
While we don’t have a full understanding of how that happened, we have rolled back the change and everything is back to functional.
Impact of the incident:
- SLO processing was delayed by 2 minutes, but has since recovered
- Queries and triggers were significantly impacted for 12 minutes
- We had a 19 second period where 14% of ingest events were impacted