Update - We are monitoring the situation and we see no problems with performance or the reporting service.
We are planning to close the major around noon. If you encounter any problems, please update your ticket.
Sep 25, 10:09 CEST
Update - Our SaaS Operations team has been working on a fix. Currently the performance is back to what it should be and we are closely monitoring the Duration Reports & Odata feed functionalities. Tomorrow we will evaluate the situation and determine the strategy for the next few days.
We ask you to not use any heavy Odata feed extractions until we can safely say that the fix works.
We strive to have all functionalities fully operational with solid performance by Monday. However we are cautious and cannot guarantee this yet.
Sep 24, 17:10 CEST
Update - Hereby more information regarding the current problems with performance and the reporting (Odata) services.
The root cause of the problem is that the virtual machines which are hosting our new services architecture have become overloaded. The is due to an increase in the overall traffic and an increase in the usage of the reporting layer. To remedy the performance issues we have temporarily disabled the reporting service (Duration reporting and the Odata feed).
We realize that the Odata feed can be a vital part of your process and therefor we are researching if we can reactivate the reporting service.
For a long term solution we will evaluate the resources allocation in the new services architecture, more on this will follow in the RCA.
Sep 24, 13:55 CEST
Update - Our SaaS Operations team continues to work on the problem.
It seems that multiple causes have put a too high load on the hardware of the services architecture.
Currently we are in contact with the supplier of the hardware to find a fitting solution.
Later today we will publish a more detailed update on the situation, the cause and the steps we will take to remedy this.
Sep 24, 10:55 CEST
Update - The SaaS Operations team has decided to disable the Odata feed for a longer period of time. We will enable the Odata feed as soon as possible but stability and performance have our priority. When we have an estimated time we will post this on the status page.
Along with closely monitoring the situation, our Operations team has contacted our supplier to make sure that the SaaS environments will remain stable after we enable all the features.
Sep 23, 17:22 CEST
Monitoring - Our SaaS team has been closely monitoring the resources on the NL3 data center. We have implemented a temporary fix and the performance should be back to normal.
A part of this temporary fix is to disable the reporting service. The reporting service is part of the new architecture and will affect Duration reports and Odata reports, the standard reports in TOPdesk will continue to work.
We are planning to enable the reporting service again after working hours with reduced resources.
A long term solution will involve changes made by our development teams, these changes will take more time and will be mentioned in the RCA.
Sep 23, 13:05 CEST
Update - We are closely monitoring the services and the resources and we are seeing that the impact on performance has been reduced. We will continue to monitor the situation.
We are not enabling the reporting service yet (for Duration reports and the Odata feed), we want to make sure that the issues are resolved before enabling these services. Later today we will post a decision regarding the re-enabling of the reporting service.
Sep 23, 11:06 CEST
Update - The SaaS Operations team is closely monitoring the resources of the services.
Within an hour we will give an update.
Sep 23, 09:56 CEST
Identified - A summary of the problems we have been having on our NL3 data center:
On Monday we noticed one of our services was requiring a lot more resources than normal, which negatively affected the performance of your TOPdesk environment. During the day it seemed like the problems were hardware related and the major incident was closed.
Later on this same day the problems started coming back around noon. It was not clear whether the issues had the same cause, thus we created a new major incident. We investigated until the end of the day and continued to monitor the resources.
On Tuesday around peak times (10:00 AM) we noticed the same problems with the services and the resources. A more in depth investigation found a likely culprit, the reporting service. We planned to disable this service on Wednesday during the next peak times, however later on this Tuesday the problems came back.
Around 14:00 our SaaS Operations team noticed the issues were affecting the authentication service. To resolve this, resources have been re-balanced to have this key service function again. Furthermore, we have disabled the reporting service for now. This means that Duration reports as well as the Odata feed will not work at this moment.
The plan to monitor the new situation closely on Wednesday morning still stands, we will closely monitor the resources and we will determine a strategy to bring this service back online as soon as possible with stability as the most important factor.
Sep 22, 17:02 CEST
Update - We have now briefly experienced problems with the authentication service, this will have resulted in temporary downtime.
In order to swiftly react we are disabling the service responsible for reporting. This means that the duration reporting functionalities will not be working in order to make sure that our other services stay online. The standard TOPdesk reports will continue to work.
We will update you with a more detailed plan.
Sep 22, 14:55 CEST
Update - Our SaaS operations team suspects that the problems from today are originating from specific key services which are affecting the performance.
If you have information about a certain time period of degraded performance, please let us know by updating your ticket so we can correlate this to our monitoring.
Sep 22, 11:57 CEST
Update - The performance problems which started yesterday (2020-09-21) still appear to affect environments on the NL3 data center.
We will contact the callers of newly created tickets to investigate the cause of the problems.
If your environment is affected, please create a new ticket from the major on my.topdesk.com.
Sep 22, 10:53 CEST
Update - We are continuing to investigate this issue.
Sep 21, 18:22 CEST
Update - We are continuing to investigate this issue.
Sep 21, 17:54 CEST
Investigating - A few customers in our NL3 hosting location reported performance issues. We're investigating if a common cause can be found for these problems.
It is different from the problem experienced this morning and we expect to give more updates as soon as we have more information or find out the root cause of it.
Sep 21, 16:20 CEST