Understanding the Diversity of Ethereum Clients in One Article: Why Is It So Important?

Original Author: Joseph Cook

Original Compilation: Nan Feng, Unitimes

Ethereum has multiple interoperable clients developed and maintained by independent teams using different programming languages. This is a significant achievement, as it provides resilience to the network by limiting the impact of vulnerabilities or attacks to the portion of the network running the affected client. However, this advantage can only be realized if all users are roughly evenly distributed across the available clients. Currently, the vast majority of Ethereum nodes run a single client, exposing the network to unnecessary risks.

Ethereum is about to undergo the most significant upgrade to its architecture since its inception—the transition from Proof of Work (PoW) to Proof of Stake (PoS) known as "the Merge." This will fundamentally change how the network reaches consensus on the true state of the blockchain and maintains network security. This new architecture will bring benefits in terms of security, scalability, and sustainability, but it also amplifies the risks associated with a single client being dominant. This article will explore the reasons behind this…

Beacon Chain

The Beacon Chain is a PoS blockchain. It currently runs in parallel with the Ethereum mainnet, but the two will soon "merge" together. After the merge, the existing Ethereum mainnet clients ("execution clients") will continue to host the Ethereum Virtual Machine (EVM) and validate and broadcast transactions, but they will stop participating in PoW mining and relinquish the responsibility of reaching consensus on the blockchain head (the top block).

Instead, this consensus will become the responsibility of the "consensus clients," which will package transactions from the "execution clients" along with the information needed for consensus into "beacon blocks," which make up the Beacon Chain. "Miners" will be replaced by "validators," who will need to deposit ETH into a specific Ethereum smart contract (a process known as "staking").

The ETH staked by validators will serve as collateral to incentivize them to perform their validation duties correctly. Validators that fail to perform their duties (for example, due to being offline) or engage in malicious behavior will have a portion of their staked ETH destroyed. On the other hand, if validators behave properly, they will earn ETH rewards.

1. Responsibilities of Validators

For validators, good behavior means participating in the validation of beacon blocks received from other validators and voting on the blockchain head. If the block received by a validator is valid, the validator will "attest" to the block, effectively voting to support its addition to the blockchain.

A node will be periodically asked to propose a new block, which other validators will "attest" to. When there are multiple forks in the blockchain, only the chain that has accumulated the most "attestations" in its history is considered the correct blockchain.

Validators will also periodically participate in a sync committee, which is a small group of 512 randomly selected validators that will sign block headers, allowing light clients to retrieve these validated blocks without accessing the entire historical chain or the entire set of validators.

2. Justification & Finalization

The Beacon Chain sets the rhythm for the network. This rhythm is organized into two time units: slots and epochs. A slot is an opportunity to add a block to the Beacon Chain, occurring every 12 seconds. A slot may not contain a block, but when the system is operating optimally, blocks are added to each available slot. An epoch consists of 32 slots (approximately 6.4 minutes). Slots and epochs set the pace for the Ethereum blockchain.

During each epoch, the block in the first slot serves as a checkpoint. Checkpoints are crucial because they are used to make the records on the blockchain ledger permanent and irreversible—this is a two-stage process: first, if at least 2/3 (i.e., "absolute majority") of the active validators' staked ETH balances attest to the two most recent checkpoints (the current one is called the "target checkpoint," and the previous one is called the "source checkpoint"), then the segment of blocks between these two checkpoints is "justified."

"Justification" is the first step toward becoming a permanent record on the Ethereum authoritative chain. Once a "justified" checkpoint has a new "justified" checkpoint appear after it, the previous checkpoint is considered "finalized," meaning it has become permanent and irreversible (i.e., all records prior to this checkpoint become permanently immutable on the blockchain).

The process of "justification" and "finalization" requires the "attestations" made by validators to be more complex than described above. There are two types of attestations: one is LMD GHOST voting, used to attest to the blockchain head (LMD GHOST is a fork choice algorithm); the second is FFG voting for attesting to two checkpoints (FFG is the "finality gadget," which justifies and finalizes the blockchain).

All validators will cast FFG votes for each checkpoint, while only a randomly selected subset of validators will cast LMD GHOST votes in each slot.

3. Validator Rewards, Penalties, and Slashing

Rewards

As mentioned earlier, the ETH staked by validators serves as "collateral" to incentivize honest behavior. As validators earn rewards for participating in protecting the network, the staked ETH will increase over time. When validators' LMD-GHOST votes and FFG votes align with the majority of other validators, they will receive attestation rewards.

When a validator is selected as a "block proposer," if their proposed block is "finalized," that validator will also receive a reward. Block proposers can also increase their rewards by including evidence of other validators' misbehavior in their proposed blocks. These rewards are incentives for validators to act honestly.

Penalties

Validators may face "penalties" in the form of mechanisms that destroy a portion of their staked ETH. When a validator fails to submit an FFG vote, submits it late, or submits an incorrect FFG vote, they will incur attestation penalties.

However, if a validator misses the opportunity to cast an LMD-GHOST vote, they will not be penalized; they simply miss out on the rewards they could have earned by voting on the blockchain head. The amount by which a validator's balance is reduced is equivalent to the rewards they could have earned had they submitted the correct attestation.

This means that an honest but "lazy" validator, who misses attestations, will face a maximum penalty equal to 3/4 of the rewards they could have earned had they performed their attestations perfectly.

Additionally, when a validator is assigned to a sync committee, if that validator fails to sign a block, their penalty will be equivalent to the ETH value they could have earned had they successfully signed the block.

Overall, these penalties are mild, and a validator's ongoing inactivity will only result in a relatively slow reduction of their staked ETH.

Slashing

Slashing is a more severe action that results in a validator being forcibly removed from the network and losing their associated staked ETH. There are three ways a validator can be slashed, all of which equate to dishonest block proposals or attestations:

• Proposing and signing two different blocks in the same slot;

• Attesting to another block "surrounding" a specific block (effectively changing the blockchain history);

• "Double voting" on two candidate blocks for the same block.

If any of these actions are detected, the validator will be slashed. This means that 1/64 of their staked ETH (up to a maximum of 0.5 ETH) will be immediately destroyed.

A 36-day removal period then begins: during this time, the validator's stake will gradually be reduced; and at the midpoint of this period (the 18th day), the validator will incur an additional penalty proportional to the total amount of ETH slashed from all validators over the previous 36 days.

This means that as more validators are slashed, the magnitude of the slashing will increase. The largest slashing will be the total effective balance of all slashed validators (i.e., if a large number of validators are slashed, they may lose their entire stake).

On the other hand, a single, independent slashing event will only destroy a small portion of a validator's stake. This intermediate penalty, which varies with the number of slashed validators, is known as the "correlation penalty."

Inactivity Leak Mechanism

If the Beacon Chain has not finalized for more than 4 epochs, an economic mechanism called "inactivity leak" will be activated. The ultimate goal of the inactivity leak is to create conditions for the blockchain to regain finality. As explained above, "finality" requires 2/3 of the total staked ETH to reach consensus on the "source checkpoint" and "target checkpoint."

If more than 1/3 of validators are offline or fail to submit attestations, it becomes impossible for 2/3 of the absolute majority of validators to finalize checkpoints.

At this point, the inactivity leak mechanism will gradually reduce the staked ETH of these inactive validators until the staked ETH they control falls below 1/3 of the total staked ETH in the network, allowing the remaining active validators to finalize the blockchain.

Regardless of how many inactive validators there are, the remaining validators will ultimately control >2/3 of the total staked ETH. This reduction in staked ETH will serve as a strong incentive, motivating inactive validators to reactivate as soon as possible!

The rewards, penalties, and slashing mechanisms in the design of the Beacon Chain encourage validators to act correctly. However, from these design choices emerges a system that strongly incentivizes the equitable distribution of validators across multiple clients and strongly suppresses the dominance of a single client.

This is because the "absolute majority" is crucial for the Beacon Chain; a single malicious validator is relatively harmless to the network, but a large number of malicious validators could cause severe damage. Let’s look at some potential scenarios…

Risk Scenarios

The asset incentive for consensus client diversity is fraught with risks. By evenly distributing validators across multiple clients, the impact of attacks or vulnerabilities targeting a specific client can be significantly reduced, while dominance by a single client increases this risk. This risk amplification effect varies with the share of the network held by a single dominant client.

We can gain more intuitive insights through some hypothetical (but potentially real) scenarios. Let’s assume a bug is inadvertently introduced into a consensus client, which could directly lead that client to make incorrect attestations or expose a vulnerability that allows malicious attackers to force the client to make incorrect attestations. How would client diversity affect the consequences of such a bug?

Scenario 1: The Affected Client Controls Less Than 1/3 of the Staked ETH

This situation provides the Beacon Chain with maximum resilience, as 2/3 of the staked ETH is still making correct attestations, allowing the Beacon Chain to finalize normally. Therefore, from the network's perspective, the consequences of this scenario are negligible. The affected validators will incur inactivity penalties for submitting incorrect attestations. These losses are relatively small, and the affected validators can wait for the client to be fixed or switch to another client. In either case, validators can continue to make correct attestations with minimal economic consequences and without disrupting the Beacon Chain.

Scenario 2: The Affected Client Controls More Than 1/3 of the Staked ETH

The problem in this scenario is much greater, as less than 2/3 of the staked ETH is making correct attestations, meaning there is no absolute majority of validators to reach consensus correctly. This means the Beacon Chain cannot achieve finality, and the Inactivity Leak mechanism will be activated. At this point, this bug affects the entire network.

For exchanges and Dapps (decentralized applications) built on Ethereum, blockchain finality is crucial; if the blockchain cannot be finalized, it cannot guarantee that transactions are permanent and immutable.

For individual validators using the affected client, the associated penalties will be much more severe, as the activation of the Inactivity Leak mechanism means that the staked ETH of individual validators will gradually be destroyed until the affected client controls less than 1/3 of the total staked ETH, and only then will the Beacon Chain regain finality.

This destruction of ETH may actually continue for some time after the Beacon Chain recovers, providing a buffer for small changes in the number of validators. Only when an affected client controls more than 1/3 of the total staked ETH will the finality of the Beacon Chain be in jeopardy.

In this scenario, validators running other alternative clients will not receive any rewards during the activation of the Inactivity Leak mechanism. This serves as a safety mechanism to prevent attackers from deliberately triggering the Inactivity Leak mechanism to increase the total rewards earned by other validators they control that are operating correctly.

These are all minor penalties, but the key point is that no one can escape the negative consequences of a consensus bug in a client controlling more than 1/3 of the total staked ETH.

Scenario 3: The Affected Client Controls 1/2 of the Staked ETH

This situation could lead to an irrecoverable fork in the Beacon Chain. If the client with the consensus bug forks onto its own chain, then neither the original chain nor the new forked chain can achieve finality, as both chains will be missing about half of the validators and will activate the Inactivity Leak mechanism.

At this point, the staked ETH of the missing validators on both chains will gradually be destroyed until the staked ETH they control falls below 1/3 of the total staked ETH in the network, at which point validators on each chain can begin to finalize again. This process will take the same amount of time on both chains, as the amount of ETH that needs to be destroyed to regain finality is equal.

These two chains will independently finalize using different checkpoints. They may never merge into a single "authoritative chain." Resolving this will require the Ethereum community to reach a consensus on which chain is the "authoritative chain."

This process will certainly be politically difficult to manage and could lead to divisions, resulting in economic losses for half of the community due to the switch in blockchains (not to mention the potential devaluation of ETH). Perhaps worse, the community may simply continue to split (similar to how the DAO event led to the creation of Ethereum Classic).

To avoid a permanent split in the Beacon Chain, validators using the affected client will have to race against the Inactivity Leak to switch clients or fix their client before the blockchain begins to finalize. There may be a 3-4 week period during which developers will rush to save Ethereum. In this scenario, a significant number of validators will face unavoidable economic losses.

Scenario 4: The Affected Client Controls More Than 2/3 of the Staked ETH

This is a nightmarish scenario for the Beacon Chain, as the affected client controls a supermajority of validators and can finalize its own chain. This means that incorrect information could potentially be permanently fixed in Ethereum's historical record. The client team will have about 13 minutes to identify the consensus bug, fix it, and broadcast the updated client information to the affected validators before the blockchain begins to finalize illegal blocks.

The only viable mitigation for this situation is for the affected validators to withdraw their staked ETH and exit the blockchain. If, after the bug is fixed, these affected validators attempt to rejoin the correct blockchain, they will be slashed due to the "correlation penalty," as the checkpoints they are now attesting to contradict the checkpoints they previously attested to, and they are collectively engaging in such actions.

The Inactivity Leak mechanism will be activated due to the large number of validators leaving, meaning these affected validators will continuously lose their staked ETH while waiting to withdraw (exit). With a large number of validators exiting, the waiting queue will be long, slow, and costly.

The only other option is for the remaining unaffected clients to accept the bug, join the new chain, and agree that the bug will henceforth be considered expected behavior in the Ethereum consensus layer. This would contradict the core principles of the staking community and lead to significant divisions. These few clients would face inactivity penalties on the new chain, even if they behave correctly.

Neither option is good. The former is very costly for the affected validators and logically difficult to correct. The latter would severely undermine trust in Ethereum and lead us to accept a permanently tainted chain.

Other Risks

Reversal of Finality

If a single client controls more than 2/3 of the total staked ETH, then the developers of that client have the power to choose which version of the blockchain history is correct. For example, if the developers of that client become malicious, they could spend some ETH (for instance, cashing out through an exchange or bridging to another blockchain network), and then these developers could collectively vote to use another chain version that does not include this spending transaction to replace the currently finalized chain.

This is a form of "double spending," as the client controlling the vast majority of validators can reverse finality and rewrite history. Meanwhile, honest minority validators will be penalized for their inconsistent attestations.

A malicious attacker controlling an absolute majority of the total staked ETH could also threaten to engage in such behavior and control the network to demand a ransom. Even a malicious team controlling 1/3 of the total staked ETH could threaten to stop the chain's finalization and activate the Inactivity Leak mechanism.

Shared Responsibility

The first point presents a somewhat pessimistic view of client development teams, not because it is reasonable, but because malicious behavior is possible and therefore needs to be guarded against. However, these developers are most likely to remain good actors, and they need to resist the dominance of a single client, not only because they are likely Ethereum users (and ETH holders or stakers) but also because the responsibility for network security should not rest on the shoulders of a small team.

The behavior of developers has a significant impact on the overall health of Ethereum, and their stress and mental health are the real costs. Client diversity avoids this situation by distributing responsibility among multiple independent teams.

Centralization

Even if the client development team consists entirely of well-meaning developers, when they control a large portion of the staked ETH, they still retain excessive power over the operation of Ethereum. Decentralization is a core principle of Ethereum, and this must include the decentralization of developers, users, and custodians.

Decentralization of development teams across multiple clients, through the even distribution of staked ETH, limits the ability of a single client team to make critical decisions regarding forks, content, and timing, thereby limiting their influence on the philosophical direction of Ethereum. Client diversity ensures that decentralized decisions are made at the developer level.

Politics

The social recovery of an honest chain is a politically charged issue. Ethereum's consensus mechanism should be determined based on the rules encoded in its clients—this is its primary goal. Intervening in this process could lead to divisions within the Ethereum community, resulting in different users having various philosophical, ethical, and technical views on how to mitigate a major client's consensus bug/attack. Governance decisions will be clumsy, destructive, and potentially too slow to achieve maximum effectiveness.

Real Examples

The probability of the above scenarios occurring is relatively low. Developers are meticulous in researching and testing every update to their software, and there is no reason to doubt the professionalism of any client team.

However, these scenarios are not purely hypothetical. There have already been real examples showing that client diversity has saved the Ethereum mainnet from permanent damage, and some consensus bugs have disrupted Ethereum testnets. Below are some of these examples.

Shanghai Attack

In September 2016, during the Shanghai DevCon conference, hackers attacked Ethereum, exploiting several vulnerabilities in the client software, which significantly slowed down the network. The attackers persisted and quickly deployed new similar attacks, while client developers raced to reverse-engineer and patch these attacks.

Ultimately, the attackers discovered an unpatchable vulnerability in the Geth client, making a hard fork inevitable. Even after the hard fork upgrade, the attackers found a denial-of-service vulnerability that exploited the inflated state caused by the previous attack, forcing the client to perform thousands of slow disk I/O operations in each block.

Client diversity won the day, as Ethereum was able to continue using the alternative Parity client, which did not suffer from the same vulnerabilities, while developers worked to fix the vulnerabilities in Geth.

The Shanghai attack was recoverable due to the presence of multiple clients, but if a similar bug had affected the majority of consensus clients, the situation could have been drastically different.

If a "consensus client" had the same dominant position as Geth at the time of the attack, Ethereum would have been unable to achieve finality, as the majority of validators would have been unable to attest to blocks. The Inactivity Leak would have been activated, as less than 1/3 of the staked ETH would have been available for attestations.

Insecura Chain

The feasibility of a "remote attack" was recently demonstrated on the Pyrmont testnet. The idea was to establish a group of validators to attest to an alternative blockchain history. These validators were then used to lure new validators into joining this dishonest "Insecura" chain, gradually increasing the number of affected validators to the point of disrupting blockchain finalization, activating the Inactivity Leak, and draining the staked ETH of the honest majority validators.

Ultimately, this could lead to the affected client finalizing its version of the blockchain. While the time and money required make this behavior unlikely to become an attack vector, similar dynamics could lead to a bug in a dominant consensus client infecting a large portion of the network.

Medalla Testnet

Previously, the number of active validators on the Medalla testnet suddenly dropped due to a clock issue with the Prysm client. The chain was unable to finalize because too many validators exited the network, resulting in less than 2/3 of the majority of staked ETH being available for attestations.

Its recovery was gradual, as it relied on validators switching their clients from Prysm to other minority clients. Then, real time caught up with the incorrect clock time of the Prysm client, and previously invalid attestations suddenly became valid.

This caused the Prysm client to stall, while the Teku and Lighthouse clients faced significant state inflation due to suddenly processing a large volume of attestations. If Prysm had been the only client on the Medalla testnet, the entire network would have stalled; if the Prysm client controlled less than 1/3 of the total staked ETH, much of the chaos could have been avoided.

Prysm's Deposit Root Bug

In early 2021, the Prysm client encountered a bug related to Eth1 deposit root verification. At that time, the Prysm client was able to generate invalid deposit roots and pass them to other Prysm nodes. Because Prysm held such a large share of validators, this invalid deposit root quickly spread through the network, and due to Prysm's adherence to the majority voting mechanism rather than explicitly verifying the deposit root in each block, its spread was accelerated.

Although the impact of this bug was minor, not interrupting the finalization of the Beacon Chain and not imposing significant economic penalties on validators, this event demonstrated the importance of client diversity in two ways: first, if the Prysm client had a smaller share of validators, it would have limited the spread of this bug throughout the network, reducing its impact.

Second, the analysis articles following this event described how to use alternative clients as a benchmark to help developers quickly identify and fix the bug. Clearly, this would not have been possible without multiple actively maintained clients.

Current Client Diversity

The above image shows the current diversity of Ethereum clients: the left side shows the distribution of execution clients, while the right side shows the distribution of consensus clients.

The two pie charts above provide a snapshot of the current client diversity in Ethereum's execution layer and consensus layer (as of January 2022 when this article was written): the execution layer is dominated by the Geth client, with OpenEthereum a distant second, Erigon in third place, Nethermind in fourth, and other clients accounting for less than 1% of the network.

The most commonly used consensus client, Prysm, while not as dominant as the Geth client in the execution layer, still controls over 60% of the network, with Lighthouse and Teku accounting for 20% and 14%, respectively, and other clients being rarely used.

Execution layer data was obtained on January 23, 2022, via the Ethernodes website (ethernodes.org/); consensus client data comes from Michael Sproul (github.com/sigp/blockprint). Consensus client data is harder to obtain, as Beacon Chain clients do not always have clear traces that can be used to identify them.

This data was generated using a classification algorithm, which sometimes misclassifies some minority clients. However, it is clear that the majority of network nodes in the consensus layer are running Prysm. Prysm's share has sometimes exceeded 68%. Although this is just a snapshot, the proportions in the chart provide a good overall understanding of the current state of client diversity.

The client diversity in the execution layer is included in the above image, as bugs affecting execution clients could also propagate to the consensus layer, due to the coupling of the consensus layer and execution layer after the merge, with the execution payload generated by execution clients being a core component of the beacon block.

Individual Stakers & Staking Pools

To address the issue of client allocation imbalance, major exchanges and staking pools need to take action. However, individual stakers can also play a role by choosing to run a combination of non-Geth/Prysm clients. Guidelines for setting up minority clients can be found on the clientdiversity.org page: https://clientdiversity.org/

For stakers holding less than 32 ETH or who do not wish to take on the responsibility of running a validator, there are some staking service providers available. Some major centralized exchanges offer ETH staking services, but the distribution of clients in their staking pools is often opaque, and the tradability of ETH staking tokens provided by these exchanges is limited. For these and other reasons, using these centralized service providers is not recommended.

A better option is to use more decentralized liquidity staking service providers, such as Lido or Rocketpool. These providers offer ETH staking services while also providing a liquidity token (for example, users staking ETH through the Lido protocol will receive a liquid stETH token), the value of which will increase as validators accumulate rewards.

These tokens can be traded or used to earn DeFi yields. The client distribution of these liquidity staking platforms is also more transparent, with Lido publishing quarterly updates, and Rocketpool now also releasing their quarterly updates. For users who cannot or do not wish to run their own validators, these services provide a way to achieve better client diversity.

Conclusion

The reward and penalty mechanisms of the Beacon Chain directly incentivize client diversity. Dominance by a single client poses a potential threat to Ethereum; when a single dominant client performs well, this threat is intangible, but when that client encounters a consensus bug, the threat can be catastrophic. Having multiple clients is a unique advantage of Ethereum and a testament to the diligent efforts of the developer community.

However, when a client controls a large portion of the staked ETH, this effort (of the Ethereum developer community) is undermined. The ideal situation is to evenly distribute the staked ETH across at least four clients, giving each client a maximum of 1/4 of the staked ETH. This is easily achievable by using the production-ready clients currently available.