The Ethereum Prysm client experienced a mainnet incident, resulting in resource exhaustion and a large-scale loss of blocks and attestations

The Prysm team released a mainnet incident review report stating that on December 4, during the Ethereum mainnet Fusaka period, nearly all Prysm beacon nodes experienced resource exhaustion when processing specific attestations, leading to an inability to respond to validator requests in a timely manner, resulting in a significant number of missing blocks and attestations.

The incident affected epochs 411439 to 411480, totaling 42 epochs, with 248 blocks missing out of 1344 slots, resulting in a missing rate of approximately 18.5%; the network participation rate dropped to as low as 75%, and validators lost approximately 382 ETH in witness rewards. The root cause was that Prysm received attestations from nodes that may have been out of sync with the mainnet, which referenced the block root of the previous epoch.

To verify their legitimacy, Prysm repeatedly replayed old epoch states and executed high-cost epoch transitions, causing nodes to trigger resource exhaustion under high concurrency. The related defect originated from Prysm PR 15965, which had been deployed to the testnet a month earlier but did not trigger the same scenario.

The temporary solution provided by the officials is to enable the --disable-last-epoch-target parameter in version v7.0; the subsequently released v7.1 and v7.1.0 included a long-term fix by verifying attestations using the head state, avoiding the need to replay historical states.

Prysm stated that the issue gradually alleviated after December 4 at UTC 4:45, and by epoch 411480, the network participation rate had recovered to over 95%.

The Prysm team pointed out that this incident highlights the importance of client diversity; if a single client accounts for more than one-third of the network, it may lead to temporary inability to finalize; if it exceeds two-thirds, there is a risk of an invalid chain at finalization. They also reflected on the unclear communication regarding feature switches and the failure of the testing environment to simulate large-scale out-of-sync nodes, and will improve testing strategies and configuration management in the future.