Performance Disruption of Phrase TMS (EU & US) on June 3 from 16:39 to 17:02 CEST

Introduction

We would like to share more details about a service disruption that affected Phrase TMS on June 3, 2026. Between 16:39 and 17:02 CEST, customers experienced errors and degraded performance when accessing Phrase TMS in both the EU and US regions. The engineering team identified and resolved the issue within approximately 23 minutes. We apologize for the disruption and are committed to preventing similar incidents in the future.

Timeline

Jun 3, 2026 @ 16:39 CEST – Phrase TMS began returning errors for customers in both EU and US regions. Our monitoring/alerting detected this and on-call engineers were alerted and began investigation immediately.

Jun 3, 2026 @ 16:42 CEST – The root cause was identified as a backwards-incompatible database schema change applied during a recent deployment. A status page update was posted to inform customers that the team was investigating.

Jun 3, 2026 @ 16:53 CEST – A remediation was prepared to revert the incompatible schema change directly in the database.

Jun 3, 2026 @ 17:00 CEST – The schema change was successfully reverted in the EU region. Phrase TMS in the EU region recovered and began serving requests normally.

Jun 3, 2026 @ 17:02 CEST – The same remediation was applied in the US region. Full service was restored across both regions.

Jun 3, 2026 @ 17:48 CEST – The status page incident was marked as resolved after continued monitoring confirmed stable operation.

Root Cause

During a planned deployment, a database schema change was applied that renamed a column in an internal database table. This change was not backwards-compatible with the version of the application currently running in production.

Our production deployment process vs the QA one applies database schema changes separately before rolling out updated application instances — a standard and normally safe approach for additive changes. However, a column rename immediately invalidates queries from the currently running application that reference the original column name. As this did not cause any issues in QA the problem was not immediately caught.

As a result, during the window between the schema change being applied and the new application version completing its rollout, all running application instances executed database queries referencing a column that no longer existed, causing all affected requests to fail with errors.

Actions to Prevent Recurrence

The root cause was a backwards-incompatible schema change applied in a single step during a rolling deployment. The following actions have been taken to prevent a recurrence:

• Code review guidelines updated: Engineering review guidelines now require reviewers to raise a blocking finding for any database schema migration that is not backwards-compatible with the previously deployed application version. This explicitly covers column renames, column drops, table renames, and constraint additions that would cause running application instances to fail.
• Safe schema change pattern enforced: The engineering team is required to use an incremental, multi-step approach for destructive schema changes: the new structure is added alongside the existing one in a first deployment, the application transitions across subsequent deployments, and the old structure is removed only once no running application instance references it.