Honeycomb Query Engine
Incident Report for Honeycomb
Postmortem

Over the weekend, our data storage and querying fleet experienced a partial degradation starting Sunday afternoon. As a result of a resource leak, some data was unable to be properly compressed on halves of our mirrored pairs, causing our metadata to be out of sync with our data. This incorrect metadata resulted in partial results being returned for some queries against the affected data.

After discovery, we identified the resource leak, and repaired the incorrect metadata causing queries to return partial results. No customer data was lost as a result of this incident.

Posted Feb 27, 2020 - 10:24 PST

Resolved
This incident has been resolved. All data that was unavailable to the query engine is now available to query normally.
Posted Feb 25, 2020 - 10:06 PST
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Feb 24, 2020 - 19:05 PST
Investigating
We are currently investigating an issue in our query engine.
You may receive the error message: "One or more nodes failed to return data. Your results may be incomplete".
We are working to repair the issue.
Posted Feb 24, 2020 - 12:07 PST
This incident affected: ui.honeycomb.io - US1 Querying.