Our fix has been identified and deployed, and there is no indication of further customer impact.
Posted Sep 23, 2024 - 14:37 PDT
Monitoring
We've found the right set of fixes required to stabilize everything. Queries should work as usual but we're keeping an eye on service stability.
Posted Sep 23, 2024 - 13:32 PDT
Update
we have rolled back our change as narrowly and quickly as we could, but we are finding it was connected to more elements of our infrastructure. We are still gradually rolling back more components until we stabilize querying of older data.
Posted Sep 23, 2024 - 13:22 PDT
Identified
We have identified an issue with a recent build that causes queries and triggers that look back further than 2 hours to sometimes fail. Retrying queries should work but may take multiple attempts. We are currently preparing a fix.