Resolved -
This incident has been resolved.
Jul 20, 14:07 CEST
Monitoring -
We continue to make progress resolving the duplicated metrics consumption issue first identified on July 16. The root cause has been identified and addressed. Since our last update, our team has corrected the duplicated data for all previously blocked accounts. We are now working through a remaining set of affected accounts and expect this cleanup to continue over the coming days. No further duplication is occurring, this work addresses historical data only.
Jul 17, 15:02 CEST
Identified -
We have successfully deduplicated metrics consumption for all blocked organizations. We are continuing work to develop per-organization deduplication tooling to ensure accurate billing cycle data, with the full team scaling these fixes tomorrow morning.
Jul 16, 21:04 CEST
Monitoring -
We have stopped the progression of the issue and are now working to identify the full scope of impact, including which organizations are affected or blocked. Affected accounts will be prioritized for resolution through a deduplication process, though this will take some time to complete.
Jul 16, 19:31 CEST
Identified -
We have identified the root cause as instability on a specific broker, which is causing repeated consumer group rebalancing in our Metrics service and leading to duplicate processing of metric updates. This may result in some customers experiencing incorrect consumption calculations or unexpected account blocking. We continue working on a fix.
Jul 16, 18:29 CEST
Investigating -
We are currently investigating an issue affecting our Metrics service that may result in duplicated metrics consumption for some customers. Our team is actively working to identify the affected components and determine the root cause. We will provide further updates as more information becomes available.
Jul 16, 17:47 CEST