RESOLVED: Performance / availability issues

Incident Report for TOPdesk SaaS Status page

Postmortem

On March 22, 2024, multiple TOPdesk hosting locations experienced periodic performance and availability issues. Although we have been unable to definitively identify the root cause of the issue in this particular case, we can confirm that it is highly unlikely that this originates with the TOPdesk application and / or network due to the synchronicity and regularity of the 'spikes' in error frequency across multiple data centers.

Immediate steps were taken to mitigate the impact and we escalated this issue to our hosting partners, as the errors we were receiving in the logging indicated an issue with our Content Delivery Network. Unfortunately they were unable to provide any further guidance on the cause of this issue.

Whilst we are not continuing to investigate this specific incident, we are committed to learning from such experiences to enhance our response and service delivery in the future. We have no reason to believe that this issue will repeat moving forward, and we have no evidence to suggest this continued after the incident on March 22nd.

Posted Apr 19, 2024 - 15:03 CEST

Resolved

We can confirm that since our previous update, there have been no further performance "spike", all system checks have returned to normal and the performance issues we were experiencing have been resolved. As this issue appears to be resolved, we will close this Major Incident. We appreciate your patience during this time and apologise for any inconvenience caused.

The evidence gathered during our investigation strongly indicates that the root cause of this issue lies with one of our hosting partners, and we are currently working with them to fully evaluate this issue. We will update this status page with a Root Cause Analysis (RCA) report once in due course, however we are unable to provide a time frame on exactly when this will be published this at this moment in time.

Posted Mar 22, 2024 - 16:02 CET

Monitoring

The temporary "spikes" of poor performance mentioned in the previous update have now not occurred for over an hour and our logging indicates normal levels of performance for customers. We are currently monitoring this situation to ensure these issues do not repeat.

At this moment in time the best information we have available indicates that the root cause of this issue lies with one of our hosting partners, however we are continuing to investigate this issue in the background and will provide a further update once we are able to confirm this or have new information to share.

Posted Mar 22, 2024 - 14:48 CET

Update

We're noticing small improvements in the performance of TOPdesk instances based on our monitoring.

However, this might be due to a natural reduction in traffic with TOPdesk related to customers usage.

We are seeing short bursts of slower performance every 30 minutes.

Our sincere apologies for any inconvienience this is causing. Our team is working hard to figure out what's causing this and fix it.

We will aim to update further within the next hour, unless new information comes to light in the meantime

Posted Mar 22, 2024 - 14:21 CET

Update

Our team of engineers are continuing the investigation into the issues of degraded performance and unavailable environments.

It's important to note that these issues are not impacting all of our customers, but a segment of them.

We will ensure to post an update on our progress as soon as possible.

Posted Mar 22, 2024 - 13:11 CET

Investigating

We are currently experiencing problems on multiple hosting locations as a result your TOPdesk environment may not be available or experiencing slow performance.

We are aware of the problem and are working on a solution.

Our apologies for the inconvenience. At the time of writing this we are not able to give you an estimate on when your environment will be available. We aim to update this status page every 30 minutes until the issue has been resolved.

E-mail updates will be sent when the issue has been resolved. You can subscribe on the status page (https://status.topdesk.com) for additional updates.

To inform TOPdesk you are affected by this issue, please visit https://my.topdesk.com/tas/public/ssp/ . Please refer to incident TDR24 03 7898.

Posted Mar 22, 2024 - 12:32 CET

This incident affected: AU1 SaaS hosting location, BR1 SaaS Hosting location, CA1 SaaS hosting location, EU1 SaaS hosting location, NL3 SaaS hosting location, UK1 SaaS hosting location, and US1 SaaS hosting location.