Coinbase Recap of the July 14 Service Outage: A Configuration Error Caused a 50-Minute Failure; User Funds Were Not Affected

Odaily Planet Daily News: Coinbase released a post-incident review report for the July 14 service outage, saying the incident was caused by a critical misconfiguration of key network components triggered by a routine configuration update. The outage lasted about 50 minutes, during which user funds remained safe at all times.

Coinbase said that at 12:34 PM Eastern Time on July 14 (12:34 PM Beijing time), the company deployed a routine configuration change to a shared production Kubernetes cluster that hosts core infrastructure services. The update was originally planned to migrate to a new service deployment approach to improve system performance and reliability.

However, because a resource name conflict was not detected by pre-production environment checking mechanisms, the configuration change unintentionally affected Kubernetes resources related to Istio’s Ingress Gateway in the cluster, causing the network component to become unavailable. About 3 minutes later, all inbound traffic to the cluster was interrupted, preventing internal services from accessing multiple infrastructure components.

Coinbase noted that many asynchronous workflows rely on these infrastructure services, including transaction settlement, fund transfers, and card transaction authorization processes, so multiple business lines were impacted. Affected services included:

Retail users were unable to complete off-chain transactions, deposits, and withdrawals; some in-progress transactions appeared stuck, and were completed after service restoration;

Coinbase Card debit card transactions temporarily failed; credit card payments were not affected, but some card management functions were unavailable;

Coinbase DEX paused on-chain exchange services on Base and Solana;

Coinbase Exchange and Prime institutional customers saw transfer and settlement failures or delays;

Coinbase Developer Platform customers could not complete account opening, fund transfers, and funding services.

Coinbase said the Ingress Gateway was restored at 13:20 Eastern Time, and the incident was mitigated at 13:23. Afterwards, the system began processing the backlog, and most services returned to normal quickly, with the remaining queue being completed in the following hours. During the recovery process, Coinbase faced two additional challenges. First, the deployment tooling itself depended on the affected Ingress Gateway running, preventing the normal rollback procedure from executing, creating a circular dependency of “repair tool affected by the fault.” Second, although the emergency recovery process (break-glass access) was effective, permission verification and manual review processes increased recovery time.

Coinbase said this incident did not involve any asset safety risk, but it exposed that large financial infrastructure still needs continuous optimization in automated deployments, system dependencies, and disaster recovery mechanisms. The company will further strengthen system resilience to reduce the probability of similar incidents happening again.