Update - An update on UK1 performance issues:
Since the uptick in performance related incidents we received on Monday morning, we have received very few notifications about poor performance from customers hosted on UK1. We have investigated all updates logged since Monday morning and this has provided us with further evidence to support the position stated in our previous update that ongoing performance issues are related to individual instances and causes, as opposed to a larger issue with our infrastructure, as was previously the case.
We are continuing to communicate with customers reporting performance trouble and investigating all available information. Action points we are currently working on can be broadly summarised as:
- Case management team assessing previous updates to further establish common links between periods of slowness.
- We are assessing the balance of resources across our Application servers. If an uneven load is found we are actively contacting customers to move environments in order to guarantee that certain servers are not overloaded.
- Connections to the micro services are being investigated with the help of a third party.
If you continue to experience sub-standard performance on your TOPdesk instance please update your ticket.
Feb 25, 16:04 CET
Since the uptick in performance related incidents we received on Monday morning, we have received very few notifications about poor performance from customers hosted on UK1. We have investigated all updates logged since Monday morning and this has provided us with further evidence to support the position stated in our previous update that ongoing performance issues are related to individual instances and causes, as opposed to a larger issue with our infrastructure, as was previously the case.
We are continuing to communicate with customers reporting performance trouble and investigating all available information. Action points we are currently working on can be broadly summarised as:
- Case management team assessing previous updates to further establish common links between periods of slowness.
- We are assessing the balance of resources across our Application servers. If an uneven load is found we are actively contacting customers to move environments in order to guarantee that certain servers are not overloaded.
- Connections to the micro services are being investigated with the help of a third party.
If you continue to experience sub-standard performance on your TOPdesk instance please update your ticket.
Feb 25, 16:04 CET
Update - On Monday morning (22/2) we received further incoming reports of performance problems.
These reports were distributed over the entire data center at different times, and we have investigated each update both individually and collectively. At time of writing, the evidence we have collected strongly indicates that nearly all the problems reported on Monday can be related to individual causes. We have discussed these causes internally and we will be following up on multiple action points. All customers who reported issues will be contacted to discuss their specific case.
Please note that the best information we have available at this time indicates that substandard performance for customers hosted in UK1 would most likely have a specific cause and isn't related to a wider issue with our infrastructure, as was previously the case. To assist us with our investigation, if you continue to experience sub-standard performance on your TOPdesk instance please update your ticket and our Major Incident team for this issue will investigate further.
Apologies for the inconvenience.
Feb 23, 17:37 CET
These reports were distributed over the entire data center at different times, and we have investigated each update both individually and collectively. At time of writing, the evidence we have collected strongly indicates that nearly all the problems reported on Monday can be related to individual causes. We have discussed these causes internally and we will be following up on multiple action points. All customers who reported issues will be contacted to discuss their specific case.
Please note that the best information we have available at this time indicates that substandard performance for customers hosted in UK1 would most likely have a specific cause and isn't related to a wider issue with our infrastructure, as was previously the case. To assist us with our investigation, if you continue to experience sub-standard performance on your TOPdesk instance please update your ticket and our Major Incident team for this issue will investigate further.
Apologies for the inconvenience.
Feb 23, 17:37 CET
Update - An update on the ongoing performance issues on our UK1 data center.
Experts from our SaaS Operations team, Development and the Technical Support teams met to discuss to ongoing problems on the UK1 data center.
Our investigation has allowed us to eliminate one potential cause, and we are now looking at the infrastructure of the micro services as the probable cause. This is being actively investigated. Although we are seeing some improvement in the performance metrics today, there is still evidence to show temporary periods of poor performance. Again, we are actively investigating multiple data points around these spikes.
Should you continue to experience poor performance, please contact us as this gives us further information to investigate.
Feb 17, 16:45 CET
Experts from our SaaS Operations team, Development and the Technical Support teams met to discuss to ongoing problems on the UK1 data center.
Our investigation has allowed us to eliminate one potential cause, and we are now looking at the infrastructure of the micro services as the probable cause. This is being actively investigated. Although we are seeing some improvement in the performance metrics today, there is still evidence to show temporary periods of poor performance. Again, we are actively investigating multiple data points around these spikes.
Should you continue to experience poor performance, please contact us as this gives us further information to investigate.
Feb 17, 16:45 CET
Update - We are continuing to investigate this issue.
Feb 16, 12:40 CET
Feb 16, 12:40 CET
Update - Updated the component status on the status page to reflect the actual availability.
Feb 16, 11:15 CET
Feb 16, 11:15 CET
Update - An update on the recurring performance issues on our UK1 data center.
Since Monday February 1st, we have had multiple reports from customers hosted on our UK1 data center encountering intermittent - but consistent and disruptive - performance issues. A group consisting of representatives from UK / Technical Support, SaaS Support, Devops and Development have been investigating this issue and have determined multiple possible causes, all of which are being actively investigated with high priority.
Specifically, the two key areas are the main focus of our investigation:
1) An over exhaustion of resources and poor query processing efficiency on our database servers.
2) Connectivity problems with one of the vital micro services.
Furthermore, due in part to the number of reports coming to us, communication regarding this issue up to now has not met our usual standard, with some customers reactively chasing updates as opposed to us proactively providing updates, as we aim for. Moving forward with this issue, we are creating an entry on our status page as well as a published major on my.topdesk.com (TDR21 02 444).
If you experience problems with the performance of your TOPdesk SaaS environment, please open the above issue on my.topdesk.com, click the 'I am affected by this disruption' button, and register your ticket with the time of the problem. If you already have a ticket open; please update your ticket with the time of the problem as this will help us more quickly identify the potential cause and focus our investigative resources accordingly.
We will aim to keep all customers we are aware of being impacted by this issue updated on a regular basis via email and / or updates on the status page. Of course, if you have any further questions or have any additional concerns about this issue please contact us.
Apologies for the inconvenience this has caused.
Feb 15, 19:25 CET
Since Monday February 1st, we have had multiple reports from customers hosted on our UK1 data center encountering intermittent - but consistent and disruptive - performance issues. A group consisting of representatives from UK / Technical Support, SaaS Support, Devops and Development have been investigating this issue and have determined multiple possible causes, all of which are being actively investigated with high priority.
Specifically, the two key areas are the main focus of our investigation:
1) An over exhaustion of resources and poor query processing efficiency on our database servers.
2) Connectivity problems with one of the vital micro services.
Furthermore, due in part to the number of reports coming to us, communication regarding this issue up to now has not met our usual standard, with some customers reactively chasing updates as opposed to us proactively providing updates, as we aim for. Moving forward with this issue, we are creating an entry on our status page as well as a published major on my.topdesk.com (TDR21 02 444).
If you experience problems with the performance of your TOPdesk SaaS environment, please open the above issue on my.topdesk.com, click the 'I am affected by this disruption' button, and register your ticket with the time of the problem. If you already have a ticket open; please update your ticket with the time of the problem as this will help us more quickly identify the potential cause and focus our investigative resources accordingly.
We will aim to keep all customers we are aware of being impacted by this issue updated on a regular basis via email and / or updates on the status page. Of course, if you have any further questions or have any additional concerns about this issue please contact us.
Apologies for the inconvenience this has caused.
Feb 15, 19:25 CET
Investigating - We are currently experiencing problems on our UK1 data center. As a result the performance of your TOPdesk environment may be degraded.
Our apologies for the inconvenience. We aim to update this status page every 30 minutes until the issue has been resolved.
E-mail updates will be sent when the issue has been resolved. You can subscribe on the status page (https://status.topdesk.com) for additional updates.
To inform TOPdesk you are affected by this issue, please visit https://my.topdesk.com/tas/public/ssp/ . Please refer to incident TDR21 02 444.
Feb 15, 19:24 CET
Our apologies for the inconvenience. We aim to update this status page every 30 minutes until the issue has been resolved.
E-mail updates will be sent when the issue has been resolved. You can subscribe on the status page (https://status.topdesk.com) for additional updates.
To inform TOPdesk you are affected by this issue, please visit https://my.topdesk.com/tas/public/ssp/ . Please refer to incident TDR21 02 444.
Feb 15, 19:24 CET
About This Site
This page shows the current status of our TOPdesk SaaS infrastructure.
You can subscribe for updates for your relevant hosting locations at the top right of the page. To see which hosting location is used for your environment, see the SaaS hosting details on My TOPdesk.
This page displays disruptions affecting groups of customers. For detailed information regarding your TOPdesk SaaS environment, see the My TOPdesk monitoring page.
Also see
TOPdesk Support contact details | TOPdesk SaaS information website | TOPdesk product release notes
AU1 SaaS hosting location
?
Operational
BR1 SaaS Hosting location
Operational
CA1 SaaS hosting location
Operational
EU1 SaaS hosting location
Operational
NL3 SaaS hosting location
?
Operational
UK1 SaaS hosting location
Operational
US1 SaaS hosting location
Operational
My TOPdesk
?
Operational
General notifications
?
Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
No data exists for this day.
had a major outage.
had a partial outage.
Scheduled Maintenance
Introducing a SIEM for the TOPdesk SaaS network
Jan 25, 12:00-13:00 CET
To improve the security of our TOPdesk SaaS network we will start using a Security Information and Event Management (SIEM) system. This system is designed to process all server logs and system security events to detect potential threats.
The SIEM system will first be introduced in our Canadian (CA1) hosting location. Once all tests have been completed successfully, we will continue rolling out this solution to other hosting locations.
We already have a partnership in place with Fox-IT to monitor traffic to TOPdesk SaaS environments for the Intrusion Detection System (IDS) in the NL3 hosting location. Fox-IT will also monitor the alerts from the SIEM system, and contacts the TOPdesk SaaS hosting team when a potential threat is detected. In case of a security incident, TOPdesk Support will inform affected customers within 24 hours of detection.
By monitoring the server logs and security alerts the SIEM can operate effectively without direct access to customer data. This means implementing the SIEM does not grant Fox-IT additional access to your data.
A network diagram highlighting the changes can be found here;
https://page.topdesk.com/hubfs/TOPdesk%20SaaS%20Network%20and%20DMZ%202020-12.png
For more information about our security measures, see https://page.topdesk.com/saas-information#security
Note that during implementation the SIEM and our current IDS (on NL3) will operate simultaneously. Only once the SIEM has been deployed in the NL3 hosting location we might remove the IDS, as it serves the same purpose.
Posted on Dec 22, 15:46 CET
The SIEM system will first be introduced in our Canadian (CA1) hosting location. Once all tests have been completed successfully, we will continue rolling out this solution to other hosting locations.
We already have a partnership in place with Fox-IT to monitor traffic to TOPdesk SaaS environments for the Intrusion Detection System (IDS) in the NL3 hosting location. Fox-IT will also monitor the alerts from the SIEM system, and contacts the TOPdesk SaaS hosting team when a potential threat is detected. In case of a security incident, TOPdesk Support will inform affected customers within 24 hours of detection.
By monitoring the server logs and security alerts the SIEM can operate effectively without direct access to customer data. This means implementing the SIEM does not grant Fox-IT additional access to your data.
A network diagram highlighting the changes can be found here;
https://page.topdesk.com/hubfs/TOPdesk%20SaaS%20Network%20and%20DMZ%202020-12.png
For more information about our security measures, see https://page.topdesk.com/saas-information#security
Note that during implementation the SIEM and our current IDS (on NL3) will operate simultaneously. Only once the SIEM has been deployed in the NL3 hosting location we might remove the IDS, as it serves the same purpose.
Posted on Dec 22, 15:46 CET