Slowdown when loading insights and dashboards

Incident Report for Keypup

Resolved

Incident resolved.
Posted May 24, 2023 - 13:43 UTC

Update

The fix for our slow queries has been released to production and our data team is currently monitoring the results.
Posted May 24, 2023 - 09:02 UTC

Update

A fix for the slow queries has been developed by our data team and is currently being tested in our staging environment.
Posted May 23, 2023 - 17:20 UTC

Update

Our team has identified an area of improvement in our reporting service. We will test a fix today and deploy based on performance results.
Posted May 23, 2023 - 06:28 UTC

Update

We've proceeded with an upgrade of our reporting datastore. We will be monitoring the performance of reporting queries over the next 12 hours.
Posted May 22, 2023 - 18:58 UTC

Update

Backlog of data updates has been processed. However we are still investigating slow queries for insights.
Posted May 22, 2023 - 17:57 UTC

Monitoring

The second fix has been released. The overall performance of our data processing pipeline is now back to normal. We are currently processing the backlog of updates for the day.
Posted May 22, 2023 - 15:58 UTC

Identified

A first fix has been applied to reduce the processing time of records. A second fix is currently in testing.
Posted May 22, 2023 - 15:08 UTC

Update

The investigation is still in progress. The team is refining its investigations around specific areas of our data processing pipeline.
Posted May 22, 2023 - 13:32 UTC

Update

The team is still investigating the issue. We have identified some areas that may be creating excessive load on our data processing pipeline.
Posted May 22, 2023 - 11:10 UTC

Investigating

We've noticed that insights and dashboards take a significant amount of time to load. Our team is currently investigating.
Posted May 22, 2023 - 10:01 UTC
This incident affected: Data & Reporting.