Coinbase reviews July service outage: Kubernetes configuration conflict caused multiple trading services to be suspended, user funds were not affected

PANews July 22日消息, Coinbase’s official published a post-incident review report on the July 14 service outage, disclosing in detail the incident’s causes, the recovery process, and subsequent improvement measures. Coinbase said the outage was caused by a Kubernetes resource name conflict during a routine configuration update, which led to the failure of critical network components and affected transfers, card transactions, and on-chain services, but users’ funds were never at risk. At 12:34 PM Eastern Time on July 14, Coinbase deployed a routine configuration change to a shared production Kubernetes cluster that hosts core infrastructure services. This update was part of a migration to a new service deployment model, aimed at improving system performance and reliability, and was reviewed via the standard approval process, and was considered a low-risk change.

However, because the pre-production checks did not detect any resource name conflict, the configuration update unexpectedly modified Kubernetes resources related to the Istio Ingress Gateway in the cluster, causing the ingress gateway to become unavailable. By 12:37, all traffic entering the cluster was interrupted, preventing internal services from accessing multiple infrastructure components.

Coinbase said the affected asynchronous workflows covered almost the entire funds flow system, including trading settlement, asset transfers, and card transaction authorization. When infrastructure services could not be accessed, these processes were paused across the platform. Affected business included: retail users being unable to complete off-chain transactions, deposits, and withdrawals; some in-progress transactions showing as stalled, and automatically completing after service restoration; Coinbase Card debit card transactions temporarily failing, while credit card payments remained usable, but some card management features were unavailable; the Coinbase DEX on-chain exchange service on Base and Solana being paused; transfers and settlement failures or delays for Coinbase Exchange and Prime institutional clients; and Coinbase Developer Platform customers being unable to complete account opening, fund transfers, and funding services.

Coinbase said the company paused the affected workflows during recovery and continued to provide updates on progress via the Retail, Exchange, and Prime status pages. At 13:20 PM Eastern Time, the ingress gateway restored operations; at 13:23, the incident was essentially mitigated. Afterwards, the system began processing the backlog. Most services recovered quickly, and the remaining queues were cleared within a few hours. Regarding this incident, Coinbase said it will take the following measures to improve system reliability: expand deployment-phase protection mechanisms to automatically detect and block Kubernetes resource name conflicts, preventing configuration updates from mistakenly impacting other components; strengthen redundancy between deployment tools and underlying infrastructure to ensure that a failure does not simultaneously affect recovery tools; regularly review and drill emergency access mechanisms to improve recovery speed in the event of major outages.

Coinbase said the company’s goal is always to achieve zero-downtime operation, and it will continue to restructure its infrastructure to ensure that even if similar incidents occur in the future, it can roll back and recover more quickly. The company also apologized to users for the impact and advised users who still have related issues to contact the official support team.