prysm

The Prysm team released a mainnet incident review report stating that on December 4, during the Ethereum mainnet Fusaka period, nearly all Prysm beacon nodes experienced resource exhaustion when processing specific attestations, leading to an inability to respond to validator requests in a timely manner, resulting in a significant number of missing blocks and attestations.The incident affected epochs 411439 to 411480, totaling 42 epochs, with 248 blocks missing out of 1344 slots, resulting in a missing rate of approximately 18.5%; the network participation rate dropped to as low as 75%, and validators lost approximately 382 ETH in witness rewards. The root cause was that Prysm received attestations from nodes that may have been out of sync with the mainnet, which referenced the block root of the previous epoch.To verify their legitimacy, Prysm repeatedly replayed old epoch states and executed high-cost epoch transitions, causing nodes to trigger resource exhaustion under high concurrency. The related defect originated from Prysm PR 15965, which had been deployed to the testnet a month earlier but did not trigger the same scenario.The temporary solution provided by the officials is to enable the --disable-last-epoch-target parameter in version v7.0; the subsequently released v7.1 and v7.1.0 included a long-term fix by verifying attestations using the head state, avoiding the need to replay historical states.Prysm stated that the issue gradually alleviated after December 4 at UTC 4:45, and by epoch 411480, the network participation rate had recovered to over 95%.The Prysm team pointed out that this incident highlights the importance of client diversity; if a single client accounts for more than one-third of the network, it may lead to temporary inability to finalize; if it exceeds two-thirds, there is a risk of an invalid chain at finalization. They also reflected on the unclear communication regarding feature switches and the failure of the testing environment to simulate large-scale out-of-sync nodes, and will improve testing strategies and configuration management in the future.

Cointelegraph monitored that shortly after the Fusaka network upgrade, the participation of validators in the Ethereum network sharply declined due to a bug in the Prysm consensus client, resulting in a large number of voting nodes going offline.Prysm officially announced on Thursday that its v7.0 version client unnecessarily generated old states when processing outdated attestations, causing nodes to malfunction. Developers recommended that users temporarily start the client with the "--disable-last-epoch-targets" flag as a workaround.Data from Beaconcha.in shows that in Epoch 411,448, the network's sync participation and voting participation dropped to 75% and 74.7%, respectively. The voting participation decreased by 25%, falling just under 9 percentage points short of the two-thirds majority (66.6%) required for the network to maintain finality.The extent of the decline in voting participation roughly corresponds with the share of validators using the Prysm consensus client, indicating that the failures in attestations are likely concentrated among Prysm validators. Previously, Prysm's share had reached as high as 68.1%. As of the time of writing, the current voting participation in the Ethereum network's Epoch (411,712) is nearly 99%, with sync participation at 97%, indicating that the network has recovered.Current data from MigaLabs shows that Lighthouse still accounts for 52.55% of consensus nodes, with Prysm in second place at 18%. Ethereum educator Anthony Sassano stated that if Lighthouse encountered this bug, the network would lose finality.

The Ethereum client Prysm has indicated that it has identified the issue and provided a quick temporary solution. All nodes should disable the unnecessary generation of old states in Prysm to handle expired proofs. Simply add the flag "--disable-last-epoch-targets" in the beacon node. This parameter is applicable to version v7.0, so there is no need to further upgrade the beacon node; just add this parameter. The validator client does not require any changes.

Lido officially stated that after the successful completion of the Ethereum Fusaka hard fork, a vulnerability was found in the Prysm consensus layer client, leading to issues with network participation. The Lido protocol is operating normally, and stakers need not worry.Due to the distributed and decentralized nature of the underlying validator node set, Lido is minimally affected by this event, and the quick response from the Prysm client team and node operators has helped alleviate the issue. The Lido team worked closely with the Prysm team, monitoring the event's progress overnight and assisting affected node operators in implementing recommended temporary solutions. Currently, most of the Prysm setups running on Lido have been restored, and some nodes have temporarily switched to other client configurations to complete the fix.

The Ethereum Foundation stated on the X platform that there are currently issues with the Prysm consensus client on the mainnet. If you are using Prysm, you need to reconfigure the CL node. If you are using other clients, you are not affected and no action is required.Previously, it was reported that the Ethereum mainnet successfully activated the Fusaka upgrade, significantly enhancing data processing capabilities.

ChainCatcher news, Galaxy's Vice President of Research Christine Kim summarized the 123rd Ethereum Core Developers Consensus (ACDC) call. The Cancun/Deneb upgrade was activated on November 30 on Devnet #12, and the Teku, Lodestar, and Lighthouse client software, along with all Execution Layer (EL) client software, are currently being tested on Devnet #12. Additionally, developers identified a validator exit issue on Devnet #11, which the Nimbus client team is addressing. Devnet #11 will continue to operate normally until the issue is resolved.Given that the CL client Prysm has not yet joined Devnet #12, developers agreed to postpone the planned Goerli shadow fork until the Prysm client software is ready for testing. Tim Beiko suggested launching the Goerli shadow fork sometime before the end of this year.Furthermore, developers discussed clarifying the specifications for "BlockByRoot" and "BlobSidecarsByRoot" requests to explicitly mention whether CL nodes should respond to these requests in a specific order.

ChainCatcher news, Ethereum client Prysm announced the release of version v4.1.0, which lays the foundation for support for the Ethereum testnet Deneb, with features such as backward synchronization and file system-based blob storage planned for release in Q4 2024.It is reported that the new features of this version include: providing basic support for Deneb/Cancun, reducing memory usage through Multi-Value Slices, EIP-4881 for optimizing block processing, improved portable build performance with BLST0.3.11, and real-time previews of Multiarch Containers, etc.

ChainCatcher message, Ethereum client Prysm release reminder: "Do not upgrade the Ethereum client Geth to version V1.13.0 for now, as we have discovered a bug that affects Prysm's block production. Although most clients do not need to use mev-boost for local block production, Prysm will still validate local blocks. More updates will be released by Prysm later. Geth maintainer Péter Szilágyi has confirmed that the bug has been fixed on the master branch."

ChainCatcher news, Ethereum client Prysm announces the release of version v4.0.5, which includes many important improvements and bug fixes since v4.0.4, including significant improvements to proof aggregation. (source link)

ChainCatcher message, Ethereum client Prysm released hotfix v4.0.3, which includes necessary optimizations to prevent beacon chain nodes from using excessive resources during network instability. If users' node usage is high, an upgrade is recommended. Additionally, Prysm will release the official single-point version earlier next week. (source link)

ChainCatcher message, Ethereum core developer prestonvanloon.eth tweeted that the Prysm team is developing a new version and improving the issues with Prysm. During the brief anomaly event that occurred in the Ethereum consensus layer early this morning, Prysm performed poorly, but other clients did as well, while Lighthouse performed exceptionally well during this period.prestonvanloon.eth stated that the Prysm client encountered a high rate of cache misses when replaying the recent state, with some replay differences amounting to two full blocks, at a speed of up to 65 times per minute. (source link)

ChainCatcher message, Ethereum client Prysm has released version v4.0.3, which includes several important fixes, such as corrections to the external block builder code path and errors in the key manager. All users are advised to upgrade. (Source link)

ChainCatcher message, Ethereum client Prysm releases version V4.0.0 for the Shapella upgrade. This version is mandatory for all mainnet beacon nodes and validators and must be upgraded by April 12. (source link)

ChainCatcher news, the development team Offchain Labs of the Ethereum Layer 2 network Arbitrum has completed the acquisition of Prysmatic Labs, one of the core development teams of Ethereum.Prysmatic Labs is one of the core software development teams responsible for the maintenance, upgrades, and innovation of the Ethereum network, and it is also the development team of the Ethereum client Prysm, which is currently the largest Ethereum client by user scale, with 41.5% of validation nodes using Prysm to validate transactions. (Source link)