DB migration slowing the interface down a bit
Service maps are now caught up, so we are moving this incident to resolved.
Query and application performance have returned to normal, but as a result of the incident we are behind on ingesting the events that build our service maps. We will provide a final update once service maps have caught up.
We have failed over the affected database which relieved the database load issue we encountered. Unfortunately, this caused a short period of query failures while we restarted a service that did not gracefully reconnect to the DB. Querying and performance should now be normal.
We've fixed everything query side and are still seeing issues. We are currently diving deeper into our database to see what we can find out about the unexpected performance degradation. We have disabled service map updates in the meanwhile to preserve full capacity for other features.
We've identified unrelated queries and specific ingest patterns that might be related to performance issues. We're still stable albeit a bit slower at times, but we're looking at whether addressing them brings more performance back.
Performance seems to be recovering and SLO/trigger performance is back within normal limits, but there is still a slightly elevated amount of query errors, including on the homepage.
The database migration has finished, but internal processes on the host have brought it to full utilization. We're investigating strategies to mitigate the effects until it catches up.
A database migration has had a slightly larger impact than we expected. The interface might be slower and some triggers could take a few extra seconds to run in aggregate, but we expect everything to be fully functional and to ride it out until the end of the migration.
This incident affected: ui.honeycomb.io - US1 Querying, ui.honeycomb.io - US1 App Interface, and ui.honeycomb.io - US1 Trigger & SLO Alerting.