Odaily Planet Daily News Conduit released a post-incident analysis report on the Degen Chain crash.
On Friday, May 10th (PST time), Conduit will increase the batch size of Degen and Proof of Play Apex to 10MB to reduce costs, which delays the data from being batched and published to their parent chain. On Sunday, around 1 pm on May 12th, this configuration was restored to fix the batch release. This resulted in a reorganization on two networks as the batch was released after a 24-hour mandatory inclusion window. Arbitrum Nitro will insert any inbox messages before any transactions in the batch and replay these transactions with a new timestamp. After the reorganization, due to the depth not being handled well by geth, the node will come back with a damaged database. This requires resynchronizing the data directory from genesis. The synchronization time for each network exceeds 40 hours, with a replay rate of approximately 100M gas/s. Once the node is resynchronized, Conduit will attempt various transaction replay schemes, although not all transactions can be recovered as some transactions rely on precise timestamps. After consulting with each rollup team, Conduit discusses and parallelly tries various strategies to keep the network online and restore it to its pre-reorganization state. Degen Chain went live on May 14th at 7:30 PM, approximately 54 hours after the network outage. The Apex chain, based on Proof of Play, was restored around the same time, but only became available to the public on May 15th at 4 PM after implementing another recovery plan. Conduit has improved its alerts and monitoring for the Orbit chain to cover this situation and is committed to working with Offchain Labs to enhance observability for all Orbit chain operators. The team will continue to invest in and research mechanisms to better simulate Mainnet conditions and transaction payloads in the test environment. Degen Chain Explorer is displaying the latest status of Degen Chain as usual.
The content is for reference only, not a solicitation or offer. No investment, tax, or legal advice provided. See Disclaimer for more risks disclosure.
Odaily Planet Daily News Conduit released a post-incident analysis report on the Degen Chain crash.
On Friday, May 10th (PST time), Conduit will increase the batch size of Degen and Proof of Play Apex to 10MB to reduce costs, which delays the data from being batched and published to their parent chain.
On Sunday, around 1 pm on May 12th, this configuration was restored to fix the batch release. This resulted in a reorganization on two networks as the batch was released after a 24-hour mandatory inclusion window. Arbitrum Nitro will insert any inbox messages before any transactions in the batch and replay these transactions with a new timestamp.
After the reorganization, due to the depth not being handled well by geth, the node will come back with a damaged database. This requires resynchronizing the data directory from genesis. The synchronization time for each network exceeds 40 hours, with a replay rate of approximately 100M gas/s.
Once the node is resynchronized, Conduit will attempt various transaction replay schemes, although not all transactions can be recovered as some transactions rely on precise timestamps. After consulting with each rollup team, Conduit discusses and parallelly tries various strategies to keep the network online and restore it to its pre-reorganization state.
Degen Chain went live on May 14th at 7:30 PM, approximately 54 hours after the network outage. The Apex chain, based on Proof of Play, was restored around the same time, but only became available to the public on May 15th at 4 PM after implementing another recovery plan.
Conduit has improved its alerts and monitoring for the Orbit chain to cover this situation and is committed to working with Offchain Labs to enhance observability for all Orbit chain operators. The team will continue to invest in and research mechanisms to better simulate Mainnet conditions and transaction payloads in the test environment.
Degen Chain Explorer is displaying the latest status of Degen Chain as usual.