In an attempt to make our DNS mechanism better and safer, we deployed a change that instead appears to have drastically dropped our ability to do DNS lookups. While we don’t have a full understanding of how that happened, we have rolled back the change and everything is back to functional.
Impact of the incident: - SLO processing was delayed by 2 minutes, but has since recovered - Queries and triggers were significantly impacted for 12 minutes - We had a 19 second period where 14% of ingest events were impacted