Why would a public chain stop? Understanding blockchain consensus from Cosmos's 25-hour halt

Author: imToken

On the evening of September 22, a user initiated an ATOM transfer.

After a night passed, the transaction remained in "waiting for confirmation."

The private key was not lost, and there were no signature anomalies in the wallet. Upon checking again the next day, multiple public RPCs showed that the Cosmos Hub was stuck at block height 33,086,740.

No new blocks were produced, so naturally, there was no place to package this transaction.

It wasn't until about a day later that the Cosmos Hub resumed block production, and this ATOM transfer, which had been in a waiting state, was finally successful.

For ordinary users, this might be the most intuitive lesson in understanding blockchain consensus.

We often say, "No central authority can shut down a public chain," but the reality is clearly much more complex. A sufficiently decentralized blockchain indeed usually lacks that "shutdown button" in a server room, but it can still stop.

This time, the pause of the Cosmos Hub perfectly exposed the underlying mechanisms that are usually hidden to ordinary users.

1. Why did Cosmos suddenly "stop producing blocks"?

First, it is necessary to clarify a commonly confused issue: the Cosmos Hub was not directly attacked.

The incident first occurred in Neutron.

On September 22, a governance proposal named "AIATO: AI Agent Takeover" was passed in Neutron. The attacker exploited a loophole in the chain-level governance authority, using privileged commands provided by the wasmd framework to change the contract administrators of applications like Astroport and Drop to addresses controlled by the attacker.

This is not what we usually understand as a "code vulnerability" or "protocol defect."

It can be simply understood that the application itself has its own "lock," but Neutron's chain-level governance holds a higher "master key," and when the attacker controlled the governance outcome, it was equivalent to obtaining this key, allowing them to reassign administrators, migrate contracts, and further transfer assets.

What truly brought the Cosmos Hub into the fray was the subsequent cross-chain fund transfer.

Cosmos Labs' review showed that before Neutron stopped operating, the attacker had redirected some assets to multiple networks, including approximately 1.7 million ATOM transferred to the Cosmos Hub, which began to be exchanged through cross-chain liquidity.

In other words, the Cosmos Hub itself was not directly attacked, and ordinary Hub users' funds were not directly stolen due to the Neutron vulnerability.

However, the ATOM obtained by the attacker had already entered the Hub, and to prevent the remaining ATOM from continuing to flow out, some Cosmos Hub validators began to stop their node operations.

By around 19:18 (SGT) on September 22, the validators that had stopped operating represented more than one-third of the total voting power, causing the Cosmos Hub to be unable to continue forming new blocks, ultimately stopping at 33,086,740.

This step is crucial.

It means that there is no "Pause" button that any company can directly click on the Cosmos Hub, nor was there a prior on-chain governance vote. What truly stopped the network was that enough validators ceased to participate in forming consensus.

But what is even more noteworthy is the subsequent recovery process.

About 4 hours after the chain stopped, the validators received a complete recovery plan: to execute a one-time state modification at the halted block height, transferring the remaining ATOM from the attacker's address to a multisig address jointly managed by community validators.

Subsequently, Cosmos Labs created a patch for Gaia v28.3.0 based on the plan agreed upon by the validators, tested it, and distributed it to the validators.

This version of Gaia would execute a one-time state change at the designated recovery height, transferring 1,227,121 ATOM from the attacker's address to a 4-of-6 multisig address composed of Nansen, Keplr, Enigma, Silknodes, Kiln, and Polkachu.

By the early morning of September 23, the validators confirming the installation of v28.3.0 had exceeded 67% of the total voting power. Therefore, at 12:00 UTC that day, the Cosmos Hub coordinated a restart, and about 6 minutes later, this one-time state modification was executed at block height 33,086,741, restoring normal block production.

Ultimately, the entire process from the Cosmos Hub stopping block production to resuming operation involved validators first causing the network to lose liveness, and then more than two-thirds of the voting power accepting a new set of state transition rules, ultimately making this set of rules the canonical state after recovery.

At this point, a seemingly simple question arises: Since it is a decentralized public chain, why can more than one-third of the validation power stop it, while restoring the network requires enough validators to collectively accept and run the same software?

The answer is actually hidden in the word "consensus."

2. Consensus is not "never stopping"

One of the most commonly misunderstood aspects of blockchain is equating "decentralization" with "never going down."

In reality, the consensus mechanism truly addresses the problem of how many nodes can agree on the transaction order and ledger state without a central bookkeeper.

However, different public chains implement this in different ways.

For example, Bitcoin's classic method is Proof of Work (PoW), where miners compete to produce blocks based on computational power. When the network briefly has two valid branches, nodes choose one to continue building based on accumulated work.

Thus, Bitcoin does not have a clear moment of "67% voting after which this block is forever finalized." It is closer to a probabilistic finality; the more subsequent blocks there are, the higher the computational cost required to reorganize previous transactions.

This is why it has been commonly said that a Bitcoin transaction is best waited for 6 block confirmations; after all, even with high computational power, one cannot simply bypass the consensus rules being executed by nodes.

Of course, this does not mean that Bitcoin's state is "absolutely unmodifiable" under any circumstances. Theoretically, if the entire ecosystem accepts a new client and new consensus rules, a hard fork can also make previously invalid state changes under past rules valid.

But therein lies the problem: who has the ability to get enough miners, full nodes, trading platforms, wallets, and users to accept such a new set of rules?

Almost no one.

The development team cannot decide the consensus rules for the entire Bitcoin network, and miners and trading platforms find it difficult because the consensus threshold is very high. When Binance was hacked for 7,000 BTC, some suggested that CZ contact large miners to operate, but it ultimately came to nothing.

Ethereum provides another classic example.

After transitioning to Proof of Stake (PoS), the current Ethereum uses the Gasper consensus, which combines Casper FFG and LMD-GHOST. Simply put, part of the mechanism is responsible for determining "which chain to follow," while another part ensures that blocks achieve true finality.

When validators representing at least two-thirds of staked ETH reach consensus on the corresponding checkpoint, the block can further move toward final confirmation; conversely, if more than one-third of the stake does not participate in correct voting for an extended period, the network may temporarily fail to achieve finality. However, Ethereum has also designed an inactivity leak that gradually reduces the effective weight of offline validators when finalizing is not possible for a long time, allowing the network to eventually have a chance to restore finality.

To truly change this outcome, it also requires changing the protocol rules and clients.

As seen in the 2016 DAO incident, the Ethereum community ultimately executed a special state modification, directly referred to as an irregular state change by the Ethereum Foundation, at block 1,920,000 through a hard fork, transferring the relevant ETH into a recovery contract.

However, some miners and community members who refused to upgrade and continued to maintain the original state ultimately formed Ethereum Classic (ETC), leading to the well-known ETH and ETC fork, indicating that not everyone accepted this set of rules.

Cosmos Hub is different; it uses CometBFT, which is closer to typical BFT consensus.

It can be understood as a more typical BFT consensus, meaning that for a block to be truly submitted, it needs to obtain a commit from more than two-thirds of the voting power.

The benefit is that finality is very clear; once a block is submitted with sufficient voting power, there is no need to continue waiting for more blocks like in PoW, trading probability for security.

But its other side is also very direct: if one-third or more of the voting power no longer provides the votes needed to form a commit, the remaining validators cannot gather more than two-thirds, no matter how hard they try.

At this point, the safest choice for the network is, as seen in this "pause in block production," so from the perspective of distributed systems, this brief halt of the Cosmos Hub is not mysterious at all.

In summary, after a group of validators with sufficient voting power stopped participating, the consensus protocol, according to its own rules, preferred to lose availability rather than continue confirming new blocks in the absence of sufficient consensus.

This corresponds to two concepts in distributed systems that are often confused by ordinary users:

• Safety: preventing different nodes from simultaneously confirming two conflicting final states;
• Liveness: whether the network can continue to operate and process new transactions;

For BFT systems, when there are insufficient nodes participating in consensus, pausing is sometimes precisely the cost of maintaining safety. To put it bluntly, this decentralized ledger would rather stop there than allow the remaining participants to keep their own records.

From this perspective, looking back, we find that many seemingly completely different incidents in the history of public chains actually revolve around the same issue:

What should the network do when distributed nodes cannot reach a consensus on the "correct state"?

III. From Bitcoin to Solana, where are the true risk boundaries of public chains?

This is not the first time Cosmos has brought this issue to the forefront.

As early as 2013, Bitcoin experienced a very classic chain fork incident.

At that time, Bitcoin 0.8 switched the underlying database from Berkeley DB to LevelDB, and then a block containing a large number of transaction inputs appeared. The new version of the node could process it normally, but some old version nodes, due to the limitation on the number of Berkeley DB locks, judged this block to be invalid.

Thus, a very awkward scene emerged: everyone was running Bitcoin, but the new and old clients began to produce different answers regarding "whether this block is valid."

The network split into two chains, and at one point, the new version 0.8 side had about 60% of the hash power, unable to rely on normal hash power competition to quickly converge on its own.

In the end, large mining pools coordinated to revert to the old version, regaining more hash power on the side of the old rules, and the network converged again. Bitcoin later specifically reviewed this incident with BIP 50.

By 2016, the Ethereum DAO incident pushed the issue forward another step.

As mentioned above, the Ethereum community ultimately executed a special state modification explicitly referred to by the Ethereum Foundation as an irregular state change at block 1,920,000 through a Hard Fork, transferring the relevant ETH into a recovery contract.

However, not everyone agreed with this handling; some miners and community members who refused to accept the state modification continued to uphold the original rules, leading to the long-standing existence of Ethereum Classic (ETC).

This DAO Fork is also a classic event, essentially telling everyone that when extreme events occur, there exists a social consensus beyond code consensus, and if a sufficiently consistent opinion cannot be formed, one chain can indeed split into two.

In 2021, Solana showcased a completely different failure path.

In September of that year, a large number of bot transactions flooded the network, causing validator nodes to run out of memory, leading to the crash of many nodes, and ultimately the entire network could not reach a consensus on the current state, halting the confirmation of new blocks for about 17 hours, after which validators coordinated to restore the network.

Putting these incidents together, we find that they are not the same thing:

• The issue with Bitcoin in 2013 was that different clients began to execute different validity rules;
• The issue with Solana in 2021 was that a large number of validator nodes could not continue to participate in consensus normally, causing the network to lose liveness;
• The Ethereum DAO faced a question closer to whether the community should actively modify the state through new protocol rules;
• And this time with Cosmos Hub, there is another layer of specificity, where the network proactively lost liveness through validator coordination to prevent the attack assets from continuing to move; afterward, a sufficiently high proportion of validators collectively accepted new software and recovery states, allowing the network to re-establish consensus;

Therefore, rather than simply summarizing these events as "blockchains can also shut down" or "decentralization is a lie," it is better to acknowledge a more realistic fact:

Consensus mechanisms have never been a machine that cannot break down; what they truly provide is a set of decentralized rules, such as who decides the correct chain when disagreements occur; how many participants are needed for a state to achieve finality; whether the network chooses to continue running or stop in the event of a failure; and in extreme cases, what kind of collective action can change the subsequent operating rules.

This also leaves a question from the Cosmos incident that is more worthy of ordinary users' contemplation than "should the chain be stopped."

Final Thoughts

We often say, not your keys, not your coins.

This statement still holds true; it emphasizes asset control— as long as the private key is in your hands, wallets, trading platforms, or other third parties cannot sign a transaction on your behalf.

The premise is that the blockchain you are on must have the capability to process this signature at all times.

On the day Cosmos Hub stopped producing blocks, users still held their private keys, and assets did not disappear into thin air; it’s just that even if you correctly signed a transaction, there were no new blocks to accept it.

The recovery process further illustrates that if enough consensus participants accept a new set of state rules, the on-chain state of specific accounts may change even without the original address's private key signature.

This does not invalidate "Not your keys, not your coins," but reminds us that private key autonomy and underlying consensus authority are never the same thing.

The same applies to wallets.

A wallet can ensure that private keys and signing authority are in the user's hands, can quickly identify chain-level anomalies, accurately display transaction status, establish RPC and node redundancy, and reconfirm transaction final results after the network recovers.

However, a wallet cannot restore consensus for a public chain, cannot guarantee that the underlying network will never be interrupted, and cannot ensure that the rules and states on the chain will never undergo changes at the consensus level.

Therefore, what a mature decentralized system truly needs to pursue may never be "nothing can ever change"; instead, it should clarify these imperfect boundaries as much as possible: who can pause consensus? What weight is needed? Under what circumstances is emergency intervention allowed?

Because true decentralization cannot prevent the system from encountering incidents forever; the key is that even when incidents do occur, we can still know who, based on what rules, and with what level of consensus, decides how to record the next chapter of this account.