We would like to share more details about the events that occurred with Phrase between 10:40 AM and 11:05 AM CET on November 30th, 2023 which led to a performance disruption of the Phrase Strings Translation Editor and what Phrase engineers are doing to prevent these issues from happening again.
10:54 AM CET: We received PagerDuty and Datadog alerts. The engineers started investigating the issue.
11:00 AM CET: The engineers uploaded a new version manually.
11:05 AM CET: The Engineering team confirmed that the issue has been resolved. The Phrase Strings Translation Editor is fully functional again.
We changed how we handled the uploads during the deployment process of the Phrase Strings Translation Editor. There was an error which caused the Translation Editor to not be displayed or function properly.
Ensure uploads are handled properly for the deployment process. A hot fix was released on November 30, 2023 at 11:05 CET. The long-term solution for this bug was released on November 30, 2023 at 12:05 CET.
Firstly, we want to apologize. We know how critical our services are to your business. Phrase as a whole will do everything to learn from this incident and use it to drive improvements across our services. As with any significant operational issue, Phrase engineers will be working tirelessly over the next coming days and weeks on improving their understanding of the incident and determining how to make changes that improve our services and processes.