Disruption in NL2 hosting location
Incident Report for TOPdesk SaaS Status page
Postmortem

On July 1st at 11:00 (AM) CEST our monitoring system indicated all connections to the NL2 datacenter were dropped. An investigation was started and we soon noticed all servers in the NL2 datacenter were running, but could not access the internet.

The issue was escalated and we included the hosting provider in our troubleshooting steps. At 11:33 the primary network switch was restarted, after which TOPdesk environments were reachable again. We continued to monitor the situation and no remaining issues were identified.

Root cause

The issue was caused by a problem on the primary firewall for the NL2 hosting location. As hardware issues can occur, a fail-over system is in place and should have taken over. However, the fail-over system was not configured correctly and did not trigger. After manually restarting the primary firewall, the failover system kicked in and all connectivity was restored.

Follow-up actions

We’ve already taken steps to reduce the load on the primary firewall in the NL2 hosting location. We’re contacting all customers in the NL2 datacenter to move their environment to a new location with a more reliable firewall set-up.

Posted Jul 07, 2020 - 16:44 CEST

Resolved
At 11:00 our monitoring system indicated all connections to the NL2 datacenter were dropped. An investigation was started and we soon noticed all servers in the NL2 datacenter were running, but could not access the internet.

The issue was escalated and we included the hosting provider in our troubleshooting steps. At 11:33 the primary network switch was restarted, after which TOPdesk environments were reachable again. We continued to monitor the situation and no remaining issues were identified.

We will evaluate this disruption, and aim to post a full root cause analysis on our status page in 1 week.
Posted Jul 01, 2020 - 13:13 CEST
Monitoring
We found the cause of the disruption and resolved the issue. All TOPdesk environments and services are back online.

We're checking if all services work as expected and will continue to monitor the situation.
Posted Jul 01, 2020 - 11:47 CEST
Update
Servers in the NL2 datacenter are running, but cannot connect to the internet. We're working with the hosting provider and internet provider to troubleshoot the connection issue.
Posted Jul 01, 2020 - 11:35 CEST
Investigating
We are currently experiencing problems on the NL2 hosting location. As a result your TOPdesk environment may not be available.

We are aware of the problem and working on a solution.

Our apologies for the inconvenience. The current status can be found on our TOPdesk Status Page: https://status.topdesk.com

To inform TOPdesk you are affected by this issue, please visit https://my.topdesk.com/tas/public/ssp/ . Please refer to incident 20 07 152.
Posted Jul 01, 2020 - 11:11 CEST
This incident affected: NL2 SaaS hosting location.