On July 1st at 11:00 (AM) CEST our monitoring system indicated all connections to the NL2 datacenter were dropped. An investigation was started and we soon noticed all servers in the NL2 datacenter were running, but could not access the internet.
The issue was escalated and we included the hosting provider in our troubleshooting steps. At 11:33 the primary network switch was restarted, after which TOPdesk environments were reachable again. We continued to monitor the situation and no remaining issues were identified.
Root cause
The issue was caused by a problem on the primary firewall for the NL2 hosting location. As hardware issues can occur, a fail-over system is in place and should have taken over. However, the fail-over system was not configured correctly and did not trigger. After manually restarting the primary firewall, the failover system kicked in and all connectivity was restored.
Follow-up actions
We’ve already taken steps to reduce the load on the primary firewall in the NL2 hosting location. We’re contacting all customers in the NL2 datacenter to move their environment to a new location with a more reliable firewall set-up.