On June 27th, the EU1 hosting location experienced a significant disruption that impacted the performance and availability of SaaS environments for many customers. Users reported issues such as endless loading when saving cards, and in some cases, the service was entirely unresponsive. We understand how vital our platform is to your daily operations and sincerely apologize for this inconvenience.
The root cause of the disruption was a capacity issue with our messaging service at the EU1 hosting location. The messaging service experienced memory issues due to reaching its capacity limits, which resulted in a backlog of requests and contributed to degraded performance and responsiveness in customer environments.
After our cloud engineers implemented a fix, the queued up tasks were processed gradually, resulting in customer environments returning to normal. We continued to closely monitor the situation to ensure stability and full recovery for all customers.
We take this incident very seriously and have already identified several improvements to prevent similar issues in the future. In addition, we already have an ongoing project to resize the EU1 hosting location so that resources are properly managed and monitored. This will help prevent similar issues with the messaging service running out of memory and improve the overall resilience of our platform.
We sincerely apologize for this service interruption and thank you for your patience and understanding. If you have any further questions or concerns, please reach out to our support team.