The ongoing major incident for the performance problems experienced by some customers hosted in our UK1 location will be closed.
Last month, a number of customers contacted TOPdesk Support reporting poor performance of their TOPdesk instances hosted in our UK1 data center. Due to the significant number of incidents logged in a short period of time, our major incident process was started. As previously communicated via this status page, this issue was originally caused by technical issue relating to a part of our SaaS network responsible for authentication and permission-granting. We have not observed any further issues of this nature after February 17th.
However, since February 17th we have continued to see some customers continuing to report instances of poor performance on UK1. Our case management team have investigated all tickets and questions we have received and found that nearly all the reports could be explained by overloaded sub systems such as the application server and the database server. Over the past few weeks we have also worked with a 3rd party to investigate the possibility of a further potential structural issue with UK1, and have found no evidence of this. We have communicated with all customers who have reported poor performance to us since February 17th individually, and focused our attention on resolving this by reallocating server resources, reducing the load on certain servers and analyzing the performance of the used machines. This is an ongoing process and we will continue to investigate all reports of poor performance and communicate with impacted customers moving forward.
As there is no longer any evidence to suggest that ongoing performance issues are related to a larger issue with our SaaS network, and as only a relatively small sub-set of our customers are continuing to report poor performance of their TOPdesk SaaS systems, we will close this major incident. Please note that this does not indicate a reduction in the priority of this matter, and we will continue to research into the remaining open performance problems as part of our Problem Management process.
If you continue to experience poor performance of your TOPdesk SaaS instance please update your ticket or contact TOPdesk Support and we will investigate further.
If you have any further questions about this update or anything else, please contact us and we will be happy to help.
Posted Mar 10, 2021 - 12:54 CET
Update
Further to our previous update, we are continuing to investigate all reported incidents of poor performance on customers hosted in our UK1 data center.
Since 17th February, almost all incidents reported to us have been found to have been caused by / related to specific TOPdesk instances, for example CPU spikes due to intensive, frequent OData feed extractions. Amongst other action points, we have been proactively working to improve the efficiency of these queries and better balance the load across our infrastructure.
Other lines of inquiry open at this time relate to continuing to improve the distribution of load on the application servers, and on the hardware which host part of the micro services on the UK1 data center. Although the information we have gathered over the past two weeks indicate that any examples of ongoing poor performance is not related to a structural issue with our SaaS network, we are continuing to work with a 3rd party to fully investigate this possibility.
We will update this issue with new information once we have it, but in the meantime please continue to update your ticket if you experience poor performance of your TOPdesk SaaS instance, or if you have any questions.
Posted Mar 02, 2021 - 16:00 CET
Update
An update on UK1 performance issues:
Since the uptick in performance related incidents we received on Monday morning, we have received very few notifications about poor performance from customers hosted on UK1. We have investigated all updates logged since Monday morning and this has provided us with further evidence to support the position stated in our previous update that ongoing performance issues are related to individual instances and causes, as opposed to a larger issue with our infrastructure, as was previously the case.
We are continuing to communicate with customers reporting performance trouble and investigating all available information. Action points we are currently working on can be broadly summarised as:
- Case management team assessing previous updates to further establish common links between periods of slowness.
- We are assessing the balance of resources across our Application servers. If an uneven load is found we are actively contacting customers to move environments in order to guarantee that certain servers are not overloaded.
- Connections to the micro services are being investigated with the help of a third party.
If you continue to experience sub-standard performance on your TOPdesk instance please update your ticket.
Posted Feb 25, 2021 - 16:04 CET
Update
On Monday morning (22/2) we received further incoming reports of performance problems.
These reports were distributed over the entire data center at different times, and we have investigated each update both individually and collectively. At time of writing, the evidence we have collected strongly indicates that nearly all the problems reported on Monday can be related to individual causes. We have discussed these causes internally and we will be following up on multiple action points. All customers who reported issues will be contacted to discuss their specific case.
Please note that the best information we have available at this time indicates that substandard performance for customers hosted in UK1 would most likely have a specific cause and isn't related to a wider issue with our infrastructure, as was previously the case. To assist us with our investigation, if you continue to experience sub-standard performance on your TOPdesk instance please update your ticket and our Major Incident team for this issue will investigate further.
Apologies for the inconvenience.
Posted Feb 23, 2021 - 17:37 CET
Update
An update on the ongoing performance issues on our UK1 data center.
Experts from our SaaS Operations team, Development and the Technical Support teams met to discuss to ongoing problems on the UK1 data center.
Our investigation has allowed us to eliminate one potential cause, and we are now looking at the infrastructure of the micro services as the probable cause. This is being actively investigated. Although we are seeing some improvement in the performance metrics today, there is still evidence to show temporary periods of poor performance. Again, we are actively investigating multiple data points around these spikes.
Should you continue to experience poor performance, please contact us as this gives us further information to investigate.
Posted Feb 17, 2021 - 16:45 CET
Update
We are continuing to investigate this issue.
Posted Feb 16, 2021 - 12:40 CET
Update
Updated the component status on the status page to reflect the actual availability.
Posted Feb 16, 2021 - 11:15 CET
Update
An update on the recurring performance issues on our UK1 data center.
Since Monday February 1st, we have had multiple reports from customers hosted on our UK1 data center encountering intermittent - but consistent and disruptive - performance issues. A group consisting of representatives from UK / Technical Support, SaaS Support, Devops and Development have been investigating this issue and have determined multiple possible causes, all of which are being actively investigated with high priority.
Specifically, the two key areas are the main focus of our investigation:
1) An over exhaustion of resources and poor query processing efficiency on our database servers. 2) Connectivity problems with one of the vital micro services.
Furthermore, due in part to the number of reports coming to us, communication regarding this issue up to now has not met our usual standard, with some customers reactively chasing updates as opposed to us proactively providing updates, as we aim for. Moving forward with this issue, we are creating an entry on our status page as well as a published major on my.topdesk.com (TDR21 02 444).
If you experience problems with the performance of your TOPdesk SaaS environment, please open the above issue on my.topdesk.com, click the 'I am affected by this disruption' button, and register your ticket with the time of the problem. If you already have a ticket open; please update your ticket with the time of the problem as this will help us more quickly identify the potential cause and focus our investigative resources accordingly.
We will aim to keep all customers we are aware of being impacted by this issue updated on a regular basis via email and / or updates on the status page. Of course, if you have any further questions or have any additional concerns about this issue please contact us.
Apologies for the inconvenience this has caused.
Posted Feb 15, 2021 - 19:25 CET
Investigating
We are currently experiencing problems on our UK1 data center. As a result the performance of your TOPdesk environment may be degraded.
Our apologies for the inconvenience. We aim to update this status page every 30 minutes until the issue has been resolved.
E-mail updates will be sent when the issue has been resolved. You can subscribe on the status page (https://status.topdesk.com) for additional updates.