UK1 Performance issues
Incident Report for TOPdesk SaaS Status page
Update
An update on UK1 performance issues:

Since the uptick in performance related incidents we received on Monday morning, we have received very few notifications about poor performance from customers hosted on UK1. We have investigated all updates logged since Monday morning and this has provided us with further evidence to support the position stated in our previous update that ongoing performance issues are related to individual instances and causes, as opposed to a larger issue with our infrastructure, as was previously the case.

We are continuing to communicate with customers reporting performance trouble and investigating all available information. Action points we are currently working on can be broadly summarised as:

- Case management team assessing previous updates to further establish common links between periods of slowness.

- We are assessing the balance of resources across our Application servers. If an uneven load is found we are actively contacting customers to move environments in order to guarantee that certain servers are not overloaded.

- Connections to the micro services are being investigated with the help of a third party.

If you continue to experience sub-standard performance on your TOPdesk instance please update your ticket.
Posted Feb 25, 2021 - 16:04 CET
Update
On Monday morning (22/2) we received further incoming reports of performance problems.

These reports were distributed over the entire data center at different times, and we have investigated each update both individually and collectively. At time of writing, the evidence we have collected strongly indicates that nearly all the problems reported on Monday can be related to individual causes. We have discussed these causes internally and we will be following up on multiple action points. All customers who reported issues will be contacted to discuss their specific case.

Please note that the best information we have available at this time indicates that substandard performance for customers hosted in UK1 would most likely have a specific cause and isn't related to a wider issue with our infrastructure, as was previously the case. To assist us with our investigation, if you continue to experience sub-standard performance on your TOPdesk instance please update your ticket and our Major Incident team for this issue will investigate further.

Apologies for the inconvenience.
Posted Feb 23, 2021 - 17:37 CET
Update
An update on the ongoing performance issues on our UK1 data center.

Experts from our SaaS Operations team, Development and the Technical Support teams met to discuss to ongoing problems on the UK1 data center.

Our investigation has allowed us to eliminate one potential cause, and we are now looking at the infrastructure of the micro services as the probable cause. This is being actively investigated. Although we are seeing some improvement in the performance metrics today, there is still evidence to show temporary periods of poor performance. Again, we are actively investigating multiple data points around these spikes.

Should you continue to experience poor performance, please contact us as this gives us further information to investigate.
Posted Feb 17, 2021 - 16:45 CET
Update
We are continuing to investigate this issue.
Posted Feb 16, 2021 - 12:40 CET
Update
Updated the component status on the status page to reflect the actual availability.
Posted Feb 16, 2021 - 11:15 CET
Update
An update on the recurring performance issues on our UK1 data center.

Since Monday February 1st, we have had multiple reports from customers hosted on our UK1 data center encountering intermittent - but consistent and disruptive - performance issues. A group consisting of representatives from UK / Technical Support, SaaS Support, Devops and Development have been investigating this issue and have determined multiple possible causes, all of which are being actively investigated with high priority.

Specifically, the two key areas are the main focus of our investigation:

1) An over exhaustion of resources and poor query processing efficiency on our database servers.
2) Connectivity problems with one of the vital micro services.

Furthermore, due in part to the number of reports coming to us, communication regarding this issue up to now has not met our usual standard, with some customers reactively chasing updates as opposed to us proactively providing updates, as we aim for. Moving forward with this issue, we are creating an entry on our status page as well as a published major on my.topdesk.com (TDR21 02 444).

If you experience problems with the performance of your TOPdesk SaaS environment, please open the above issue on my.topdesk.com, click the 'I am affected by this disruption' button, and register your ticket with the time of the problem. If you already have a ticket open; please update your ticket with the time of the problem as this will help us more quickly identify the potential cause and focus our investigative resources accordingly.

We will aim to keep all customers we are aware of being impacted by this issue updated on a regular basis via email and / or updates on the status page. Of course, if you have any further questions or have any additional concerns about this issue please contact us. 

Apologies for the inconvenience this has caused.
Posted Feb 15, 2021 - 19:25 CET
Investigating
We are currently experiencing problems on our UK1 data center. As a result the performance of your TOPdesk environment may be degraded.

Our apologies for the inconvenience. We aim to update this status page every 30 minutes until the issue has been resolved.

E-mail updates will be sent when the issue has been resolved. You can subscribe on the status page (https://status.topdesk.com) for additional updates.

To inform TOPdesk you are affected by this issue, please visit https://my.topdesk.com/tas/public/ssp/ . Please refer to incident TDR21 02 444.
Posted Feb 15, 2021 - 19:24 CET
This incident affects: UK1 SaaS hosting location.