Resolved -
We can confirm that the fix was applied uniformly to all impacted and potentially impacted environments; no teams should see ingest errors related to this issue anymore.
Apr 10, 10:24 PDT
Update -
We are continuing to work on the systemic solution and monitor. We'll leave the incident open until we're able to confirm the issue won't recur.
Apr 10, 08:21 PDT
Update -
The manual fix has been applied to all environments.
Apr 10, 07:32 PDT
Monitoring -
We've applied the manual fix to nearly all affected datasets (less than 1% remaining). The systemic solution is still in progress.
Apr 10, 07:19 PDT
Update -
We've identified a manual fix and are deploying that by hand to each affected environment, while also investigating a systemic solution.
Apr 10, 06:55 PDT
Update -
We have further narrowed down that only teams on Classic environments are currently seeing issues. We are figuring out mitigation mechanisms at the moment.
Apr 10, 06:46 PDT
Identified -
We have confirmed reports of some teams getting issues ingesting their data. We have identified a probable source behind this behavior and are currently trying to correct it.
Apr 10, 06:41 PDT