Slowness while working in TOPdesk (NL3 hosting location)

Incident Report for TOPdesk SaaS Status page

Postmortem

On Monday 23 November 2020 at 09:20 CET the first customers in the NL3 hosting location reported slowness while working in TOPdesk. Over the course of the day, several similar reports were logged.

TOPdesk investigated the issue and noticed a high CPU usage on several application servers. A major incident was created to group the requests and inform all customers.

We investigated if a common root cause could be found, but we didn't find any. No Operating system updates, malfunctioning hardware, or changes in the TOPdesk product could explain the slowness customers were experiencing. The problem was only found on 1 hosting location, and was evenly distributed over several unrelated machines, while only 10% of the hosting location appeared to have been affected.

We did notice some additional activity in the Java Virtual Machine used to run TOPdesk, but couldn't pinpoint the root cause.

On Tuesday there were very few reports of slowness. We closed the major by the end of Tuesday as no common cause could be identified and all affected customers reported better performance.

Follow-up actions

We identified 3 points of improvement were additional tools and information could help investigate or prevent these issues in the future. One task was already in progress, and will be continued with a high priority. We’ve forwarded the other ideas to our Performance experts for validation.

Posted Dec 07, 2020 - 16:03 CET

Resolved

A clear root cause for the high CPU load of some servers could not be found.

From our metrics we see that the CPU load today is at a normal level compared to yesterday. Our metrics also show the performance for affected environments to be better today. There are minimal new reports of slowness today. This major will be now closed, but an internal investigation will continue to mitigate this issue in the future.

If you however still experience slowness, please let us know by updating your ticket.

Posted Nov 24, 2020 - 16:36 CET

Update

The root cause of the high CPU is still being investigated.

The current performance is today better than yesterday.

If you have encountered slowness today, please let us know by updating your ticket.

Posted Nov 24, 2020 - 13:09 CET

Update

We will continue with our investigation into the slowness while working in TOPdesk environments on the NL3 hosting location today. Please update your ticket in My TOPdesk if you still experience a slow TOPdesk environment.

Posted Nov 24, 2020 - 08:58 CET

Update

The root cause of the high CPU is still being investigated. The current performance seems to be better than this morning.

We will continue to investigate the root cause tomorrow.

If you encounter slowness tomorrow morning, please let us know by updating your incident.

Posted Nov 23, 2020 - 16:47 CET

Update

TOPdesk is continuing to investigate the root cause of the high CPU load.

So far we've been unable to find a link between the virtual machines that show a higher load and a (possibly malfunctioning) hardware component. There also does not appear to be a relation with recent Operating System updates.

We're reviewing if the same high CPU load also occurs in other hosting locations. Next to that, we're reviewing all changes in the TOPdesk product over the last week.

We will post another update on this status page before 17:00 (5PM) CET today.

Posted Nov 23, 2020 - 13:57 CET

Update

We see a higher CPU load on several servers compared to previous weeks. However, no root cause for this problem has been found yet. We'll continue to investigate this issue.

Since the investigation looks like it will take more time than usual, updates will not be posted every 30 minutes. We will post another update on this topic before 14:00 (2PM) today.

Posted Nov 23, 2020 - 11:59 CET

Investigating

Several customers reported that their TOPdesk environment is much slower than usual.

We are aware of the problem and working on a solution.

Our apologies for the inconvenience. We aim to update this status page every 30 minutes until the issue has been resolved.

E-mail updates will be sent when the issue has been resolved. You can subscribe on the status page (https://status.topdesk.com) for additional updates.

To inform TOPdesk you are affected by this issue, please visit https://my.topdesk.com/tas/public/ssp/ . Please refer to incident TDR20 11 6794.

Posted Nov 23, 2020 - 11:13 CET

This incident affected: NL3 SaaS hosting location.