On Thursday 14th of March between 09:20 UTC and 11:20 UTC, a number of customers reported significant user performance and in some cases availability and login issues when using TOPdesk SaaS. Shortly after receiving the first reports of an issue, a high-priority investigation was launched by our UK Support team who soon determined that all customers reporting issues were hosted on a specific part of our infrastructure within our UK1 hosting location (container 3). Some customers hosted in this container may have also noticed less severe performance issues during the evening of March 13th too.
An analysis of monitoring data from the Application and SQL servers within this container (uk1c03), as well as database logging from individual TOPdesk instances impacted by this issue revealed that the root cause of this issue to be high CPU caused by database deadlocking. Due to the inability of the SQL server to process new DB transactions, a backlog of pending queries built up, eventually resulting in high levels of CPU. Customers on this particular server may have noted system performance gradually deteriorating during this time period.
We were able to isolate the queries in question and restore access. Although some customers still had issues isolated to specific functionalities, these were resolved after restarts of the instances in question. Although we are unable to categorically state why this deadlock occurred, the evidence available suggests this was related to the mail import service / functionality. The incident is now resolved, but there has been escalated internally for further investigation.