RESOLVED: SaaS disruption NL4C06 degraded performance

Incident Report for TOPdesk SaaS Status page

Postmortem

On June 26, 2024, beginning at approximately 09:30 CET, we started receiving customer reports of slow or unresponsive TOPdesk environments hosted on the NL4 datacenter, specifically within NL4 container 6. Concurrently, our monitoring system flagged issues with CPU on NL4.

In response to the customer reports and system warnings, our Cloud Engineers started an investigation. They identified the root cause as an overload of the host machine, resulting in the reported slowness.

By 11:30 CET, our team implemented a solution: a failover to a less busy host. This action resulted in immediate improvements in the SQL metrics, as noted by our monitoring system.

We are pleased to announce that the main issue was fully resolved by 11:42 CET. The overall performance degradation lasted approximately 2 hours and 12 minutes.

In light of this incident, we have begun an internal investigation aimed at reducing the load on the host machine to prevent similar issues from recurring.

We value your understanding and patience as we strive to offer a more reliable service.

Posted Jul 09, 2024 - 11:57 CEST

Resolved

After carefully monitoring our dashboard metrics and checking in with a few of our customers, we are pleased to inform you that the issue regarding degraded performance has been resolved.

We will proceed to evaluate this incident internally, and once completed, a detailed post-mortem will be published shortly thereafter.

Customers are encouraged to contact support if they continue to experience issues. Thank you for your patience and we apologize for the inconvenience this issue has caused.

Posted Jun 26, 2024 - 12:22 CEST

Monitoring

The measurements implemented have provided an indication that the system has regained stability.

Customers are encouraged to contact support if they continue to experience issues.

We will keep monitoring our dashboards and provide you with an update as soon as possible.

Posted Jun 26, 2024 - 12:01 CEST

Update

We have identified a potential cause and our engineers have taken measures to decrease the load on the host that was causing the problems with degraded performance. We will keep monitoring the system and provide you with an update as soon as possible.

Posted Jun 26, 2024 - 11:42 CEST

Investigating

We are currently experiencing problems on the NL4 hosting location. As a result, you may experience degraded performance while using TOPdesk.

We are aware of the problem and are working on a solution.

Our apologies for the inconvenience. At the time of writing this we are not able to give you an estimate on when the issue will be resolved. We aim to update this status page every 30 minutes until the issue has been resolved.

E-mail updates will be sent when the issue has been resolved. You can subscribe on the status page (https://status.topdesk.com) for additional updates.

To inform TOPdesk you are affected by this issue, please visit https://my.topdesk.com/tas/public/ssp/ . Please refer to incident TDR24 06 7821.

Posted Jun 26, 2024 - 11:11 CEST

This incident affected: NL4 SaaS hosting location.