Resolved -
All services have been fully recovered.
Impact window: 17:55–18:17 UTC
Regions affected: Primary impact in US. The EU region was affected to a much lesser degree.
During this period, the following effects may have occurred:
Ingest: Some data was not ingested, resulting in complete or partial data loss for events sent during the window.
Triggers: Triggers may have failed to fire within the impact window.
SLOs: SLI values across the affected window are skewed by the missing data and may show artificial dips or accelerated budget burn.
Service Maps: Maps covering the outage window are incomplete. Services and dependencies may be under-counted, as their traces were not ingested.
We apologize for the disruption. Please reach out if you have any questions about how this may have affected your data.
Jun 24, 12:04 PDT
Update -
Triggers, SLOs and Service Maps are recovered. We are continuing to monitoring the Ingest Service
Jun 24, 11:31 PDT
Monitoring -
We have rolled back the deploy and ingest service has recovered. There has been an ingest outage from 17:55 - 18:14 UTC. We have also observed an delay for Service Maps.
Jun 24, 11:22 PDT
Investigating -
We are investigating an issue with delayed ingest in the US and EU region. Our engineers are rolling back a recent deploy. SLOs and Trigger evaluations may also be delayed.
Jun 24, 11:14 PDT