Introduction

We would like to share more details about the events that occurred with Phrase between 06:57 AM CEST and 07:09 AM CEST on June 18, 2024 which led to a performance disruption of the IDM (EU) component and what Phrase engineers are doing to prevent these issues from reoccurring.

Timeline

06:41 AM CEST: Planned underlying infrastructure upgrade complete and post upgrade steps are being applied.

06:57: AM CEST: The rescheduling of IDM resources started rendering the IDM partially unavailable.

07:09 AM CEST: IDM rescheduling complete and service is fully available.

Root Cause

The rescheduling of IDM resources caused a partial unavailability of the IDM application on a load balancer level. During service stop and start, some requests reached already terminating processes or not yet fully initialized containers. 

Actions to Prevent Recurrence

• Reschedule/deployment procedure and automation will be improved to prevent availability issues on the load balancer level.
• Monitoring of application and infrastructure behind the load balancer will be improved to ensure application instances in the correct state are active on the load balancer level.