Ethereum, as a decentralized platform, relies on a p2p (peer-to-peer) network for its core functionality. The p2p network is responsible for communication between execution layer nodes, providing underlying capabilities for other sub-protocols, propagating blocks and transactions, and supporting Ethereum's decentralization.

1. Overview of p2p Network Design

1.1 Introduction to DevP2P

The implementation of Ethereum's execution layer p2p network functionality is based on the standards defined by the DevP2P protocol stack. The p2p network layer is independent of the consensus mechanism (whether it is the previous PoW or the current PoS, which is coordinated by the consensus layer). Geth's network architecture includes two parallel working protocol stacks: a discovery protocol stack based on UDP (mainly used for discovering network nodes) and a communication protocol stack based on TCP (mainly used for data exchange and synchronization between nodes).

1.2 DevP2P Protocol Stack

DevP2P does not refer to a single protocol but is a set of network protocols customized for Ethereum's p2p network. Its design is not limited to a specific blockchain but primarily serves the needs of the Ethereum ecosystem. Its core includes:

1. Node Discovery Layer: Based on the UDP protocol, used to locate other Ethereum nodes on the p2p network. It includes protocols such as DiscoveryV4, DiscoveryV5, DNS Discovery, etc.

2. Data Transmission Layer: A TCP-based protocol that provides encrypted and authenticated communication sessions between nodes. It is mainly supported by the RLPx transport protocol, from which various application layer protocols are derived.

3. Application Layer Sub-protocols: Built on the node discovery and secure connection established by RLPx, these handle specific data interactions and application logic between nodes. These protocols enable nodes to perform operations such as blockchain synchronization, transaction propagation, and state queries. They include core protocols like ETH (Ethereum Wire Protocol), LES (Light Ethereum Subprotocol) for lightweight clients, SNAP for snapshot synchronization, as well as WIT (Witness Protocol) and PIP (Parity Light Protocol).

Below is the overall structure diagram of the DevP2P protocol stack:

Overall structure diagram of the DevP2P protocol stack

Brief description of application layer sub-protocols:

|-----------------------------------------------------------------------------------------------------|--------|-------------------------------| | Protocol | Version | Main Function | | Ethereum Wire Protocol （Ethereum Wire Protocol） | eth/68 | Main protocol for blockchain synchronization and transaction exchange | | Ethereum Snapshot Protocol （Ethereum Snapshot Protocol） | snap/1 | Efficient state snapshot exchange | | Light Ethereum Subprotocol （Light Ethereum Subprotocol） | les/4 | Lightweight client protocol (not yet implemented in Geth) | | Parity Light Protocol （Parity Light Protocol） | pip/1 | Parity's implementation for lightweight clients (not yet implemented in Geth) | | Ethereum Witness Protocol （Ethereum Witness Protocol） | wit/0 | Exchange of witness data between nodes (not yet implemented in Geth) |

1.3 Relationship between DevP2P and LibP2P

LibP2P (library peer-to-peer) is a p2p network framework that shares functional similarities with Ethereum's DevP2P, providing support for the development of p2p applications through a series of protocols, specifications, and libraries. Despite their similarities, Ethereum did not choose to adopt LibP2P as its P2P layer during its early development (around 2014-2015).

The reason is that during the critical period when the Ethereum team was developing the p2p network module (which ultimately formed DevP2P), LibP2P, although already part of the IPFS project, had not matured enough to be directly integrated into Ethereum at that time. To meet its own needs, Ethereum developers designed and implemented a set of p2p protocols independently.

Unlike DevP2P, LibP2P's initial core goal was to build a more universal and highly modular P2P network foundation aimed at serving a diverse range of decentralized applications, rather than being limited to a specific platform or application scenario.

DevP2P can be seen as a collection of protocols that clearly defines the components required by Ethereum, such as ENR, discv5, and RLPx. In contrast, LibP2P resembles a collection of programming libraries, providing composable modules to build various P2P functionalities, including transport, stream multiplexing, secure channels, node discovery, and publish/subscribe messaging.

Currently, in Ethereum's consensus layer, clients such as Lighthouse, Prysm, Teku, and Nimbus primarily use LibP2P, leveraging LibP2P's transport layer implementation (TCP, QUIC), encryption (Noise), stream multiplexing (mplex, yamux), node discovery (ENR-based discv5), and a powerful publish/subscribe system (gossipsub) to efficiently broadcast proofs and blocks.

2. Detailed Explanation of the DevP2P Protocol Stack

2.1 Discovery Layer

Discovery Protocol V4 (Discovery Protocol v4)

Discovery Protocol V4 (abbreviated as DiscV4) is one of the core components of the Ethereum DevP2P protocol stack. It is a distributed hash table (DHT) based on the Kademlia algorithm concept, used for node discovery in the network. The core mechanism of Kademlia is to organize nodes based on a specific "distance" metric between node IDs (obtained through XOR operations) and store information about other nodes (such as IP address, port, and node ID) in a routing table called "k-buckets." This information is organized into Ethereum Node Records (ENR), which contain the node's network address (IP address, TCP and UDP ports), a unique node ID generated from the public key, the public key itself, and signatures and sequence numbers used to verify the validity of the records.

Thus, the goal of DiscV4 is to help nodes efficiently discover and connect to these peer nodes in an unknown network. This efficiency is attributed to its iterative query method: when a node A wants to find a target node B, it sends query requests to a batch of nodes in its k-buckets that are closest to the target B in terms of ID "distance." These queried nodes will respond with a list of nodes that are closer to target B in their routing tables. Node A continues to query these newly discovered, closer nodes, gradually approaching the target, usually able to locate any node in the network within logarithmic time complexity (O(log N), where N is the total number of network nodes), which is far more efficient and resource-saving than random probing or broadcasting across the entire network.

Discovery Protocol V5 (Discovery Protocol v5)

Discovery Protocol V5 (abbreviated as Discv5, with the current latest specification being v5.1) is a significant upgrade to DiscV4. While retaining the core concepts of the UDP-based Kademlia DHT (such as defining network topology through XOR "distance" between node IDs, maintaining routing tables with "k-buckets," and using PING/PONG and iterative FINDNODE/NODES messages for node discovery), it introduces several key improvements to address the shortcomings of the previous protocol in address management, information carrying, and scalability. Its key update is the native implementation of Ethereum Node Records (ENR, following the EIP-778 standard).

Unlike Discv4, which later supported ENR through limited extensions, Discv5 makes ENR its foundational data unit, allowing ENR to carry richer and verifiable key-value pair metadata (such as supported protocols, specific roles, etc.), and ENR is directly included in regular messages (such as NODES responses) for exchange, eliminating the need for a specific ENRRequest like in DiscV4. Another improvement is the introduction of a "topic"-based discovery mechanism in Discv5; nodes can broadcast their provided services or interests in specific topics through TOPICREGISTER messages, while other nodes can precisely query these peer nodes with specific capabilities through TOPICQUERY, achieving a more refined service discovery capability than DiscV4.

Additionally, Discv5 defines a new, more secure discv5-wire underlying protocol, establishing encrypted communication channels through an improved handshake process (using temporary keys and authentication tags AuthTag) and providing stricter and clearer specifications for message formats.

|-------------|-------------------------------------------------|--------------------------------------------------------------------------------| | Feature | Discv4 | Discv5 | | ENR Integration and Specification Base | Support for ENR defined by EIP-778 was added later through EIP-868 as an extension. | Built natively on the EIP-778 defined ENR from the outset, making ENR its core foundation. | | ENR Exchange Mechanism | Obtaining the latest ENR typically requires sending a specific ENRRequest. | ENR is directly included in regular messages (such as NODES response of FINDNODE), without a specific request. | | ENR Content and Function Utilization | Mainly used to provide richer node metadata and improved endpoint proof. | Fully utilizes the flexibility of ENR's key-value pairs, supporting advanced features like Topic Advertisement, allowing nodes to declare specific services or protocols for more precise node discovery. | | Identity Scheme | ENR uses the "v4" identity scheme. (Here, v4 refers to the name of the identity method, not the aforementioned discv4) | Relies on the identity scheme defined by EIP-778, offering better scalability (though "v4" is still commonly used). |

DNS Discovery This mechanism serves as a complement to the UDP-based P2P discovery protocols (Discv4/Discv5), aiming to provide a stable and easily updatable list of nodes (especially bootstrap nodes) as initial entry points for clients to connect to the network. The documentation details how to encode Ethereum node information (usually Ethereum Node Records ENR) into a specific string format and store it in DNS records (typically TXT records). These DNS domain names and records are managed and published by trusted list maintainers. Clients can query these specific DNS domain names to obtain the ENR list, which can then be used to initiate the Discv5 or Discv4 p2p discovery process with the nodes in the list.

2.2 Data Transmission Layer

RLPx (Recursive Length Prefix Transport Protocol) serves as the underlying transport protocol of the DevP2P protocol stack, running on top of the TCP protocol, specifically designed to provide authenticated, encrypted, and multiplexed secure communication channels between Ethereum nodes.

Between two nodes that locate each other through the discovery protocol and decide to establish a connection, RLPx ensures the authenticity of both parties' identities and negotiates encryption parameters through an initial handshake process. This process utilizes asymmetric encryption technology, where nodes exchange temporary keys and authenticate each other using their long-term identity keys, ultimately generating a shared key securely through Elliptic Curve Diffie-Hellman Key Exchange (ECDH). After a successful handshake, this shared key is used to encrypt all subsequent data transmitted over this TCP connection, ensuring the confidentiality of the communication content.

A key feature of RLPx is its multiplexing capability, which allows multiple different upper-layer application protocols (i.e., DevP2P sub-protocols, such as the ETH protocol for blockchain synchronization, the LES protocol for lightweight clients, etc.) to be carried and distinguished simultaneously over a single TCP connection. This is typically achieved through a protocol type ID embedded in the message header for identification and routing. All application layer messages transmitted over RLPx are organized into a specific format of "frames" for transmission. These frames not only contain the encrypted message payload (usually RLP-encoded application layer data) but also include necessary metadata such as message size and protocol type, and support splitting larger messages into multiple frames for chunked transmission.

In summary, RLPx provides Ethereum nodes with a generic secure pipeline independent of specific application logic through encryption handshakes, message encapsulation, and multiplexing mechanisms, forming the basis for efficient data synchronization, transaction broadcasting, and other functionalities.

2.3 Application Layer Sub-protocols

ETH (Ethereum Wire Protocol)

ETH facilitates the exchange of Ethereum blockchain information between peer nodes through the RLPx transport protocol. The ETH protocol defines how nodes exchange blockchain-related information, such as block headers, block bodies, transaction data, and state information. It includes the protocol's version negotiation mechanism, the structure of different message types (using RLP encoding), message codes, and the interaction processes of these messages in processes like block synchronization and transaction broadcasting, ensuring that nodes in the network can effectively synchronize and verify blockchain data, thereby jointly maintaining the consistency and integrity of the Ethereum network.

SNAP (Ethereum Snapshot Protocol)

SNAP runs on top of RLPx and is used to propagate Ethereum's state snapshots between nodes. Its main purpose is to provide dynamic snapshots of recent states for semi-real-time data retrieval, but it does not participate in chain maintenance and needs to work in conjunction with the ETH protocol. SNAP supports retrieving continuous account segments from the state tree or storage slots from specific storage trees, accompanied by Merkle proofs for quick verification, and can also batch retrieve bytecode. This protocol reduces the need to download intermediate Merkle tree nodes by allowing nodes to directly obtain account and storage data and locally reconstruct the state tree, thus lowering network bandwidth consumption and latency. It mainly serves as a guiding auxiliary tool for new full nodes and relies on the ETH protocol.

LES (Light Ethereum Subprotocol)

LES is designed specifically for lightweight clients, allowing them to securely access the Ethereum blockchain without participating in the consensus process, primarily by downloading block headers and fetching other blockchain parts on demand. This protocol utilizes canonical hash trees (CHT) for fast synchronization and secure data retrieval (such as block headers, total difficulty) and employs BloomBits technology to optimize log searches. To prevent server overload and manage request rates, LES also introduces client flow control mechanisms, adjusting parameters such as buffer limits, minimum recharge rates, and maximum request costs, thus providing lightweight clients with an efficient and secure way to interact with the blockchain.

PIP (Parity Light Protocol)

PIP is a variant of the LES protocol designed by Parity Technologies for its Ethereum client. It also uses canonical hash trees (CHT), but generates one every 2048 blocks. PIP employs a token bucket-like flow control mechanism, requiring clients to mirror the server's state. Protocol messages include state, declarations, request batching, response batching, and updates to credit parameters. Notably, PIP's request and response messages are batched and cannot be sent individually, aiming to optimize the interaction rounds between clients and servers. The protocol supports various request/response pairs for retrieving block headers, transaction information, block bodies, account and storage proofs, contract code, and execution proofs.

WIT (Ethereum Witness Protocol)

WIT runs on top of RLPx and is used for exchanging Ethereum state witnesses between peer nodes. It aims to help clients synchronize to the top of the chain and ultimately support stateless client operations, but does not participate in chain maintenance and works in parallel with the ETH protocol. Version 0 of the protocol mainly provides metadata for witnesses (such as a list of tree node hashes read during block execution, including storage, bytecode, and account nodes) to assist nodes in downloading actual witnesses through the ETH protocol, thus speeding up synchronization.

3. Ethereum Node Lifecycle and Data Flow Diagram

Node lifecycle and data flow diagram

Brief explanation of the flowchart:

1. Node Startup: The Ethereum client (Geth) begins running.

2. Loading Configuration: Reads the configuration file, including network ID, genesis block information, port settings, etc.

3. Initializing p2p Server: Sets up the p2p server, preparing to listen for and initiate connections.

4. Node Discovery: The node begins searching for other peer nodes in the network.

• DNS Discovery: Obtains initial node information through the DNS seed node list.

• Kademlia (Discv4/v5): Uses node discovery protocols to discover other nodes.

1. Discovering Potential Nodes: Aggregates node information from different discovery mechanisms.

2. Connection Establishment:

• Listening for Incoming Connections: Accepts connection requests from other nodes.

• Active Dialing: Initiates connection requests to discovered nodes.

1. Accepting Connection Requests/Initiating Connections: Successfully establishes a TCP connection.

2. RLPx Encrypted Handshake: Performs an encrypted handshake over the TCP connection to ensure communication security and negotiate session keys.

3. DevP2P Protocol Handshake: After a successful RLPx handshake, a handshake at the DevP2P layer occurs. Nodes exchange their supported sub-protocols (such as eth, snap, les, etc.) and their versions. Both parties negotiate the highest version of the protocol they mutually support.

4. Sub-protocol Interaction:

   • ETH Protocol Interaction: If the eth protocol is successfully negotiated, nodes will exchange blocks, transactions, chain states, and other information through this protocol.

   • Snap Protocol Interaction: If the snap protocol is successfully negotiated, nodes can more efficiently synchronize state data, such as accounts, storage, bytecode, and Trie nodes.

   • Other Sub-protocol Interactions: Such as les (Light Ethereum Subprotocol), etc.

5. Peer Management: Nodes maintain a list of peer nodes, tracking their status (such as health, supported protocols, latest block height, etc.), and add new nodes or remove disconnected/misbehaving nodes as needed.

6. Data Exchange and Processing:

   • Block/Transaction Synchronization and Broadcasting: Through the ETH protocol, nodes download new blocks and transactions, verify them, and broadcast their new blocks and transactions to other nodes.

   • State Synchronization: Through the snap protocol (or certain messages of the eth protocol), nodes request and receive account data, storage slots, contract bytecode, and node data of the state tree (Merkle Trie) to quickly build or update local states.

   • Data Processing and Verification: All received data (blocks, transactions, state fragments) must undergo strict verification to ensure compliance with protocol rules and chain consistency.

7. Updating Local Chain State/Database: Verified data is used to update the local blockchain database and state tree.

8. Providing Services/Responding to Queries: Nodes provide services externally based on their roles (full node, light node, etc.), such as responding to RPC requests, processing transactions, etc.

The following will provide a detailed explanation of this flowchart in conjunction with the source code.

4. Geth Code Implementation

The following will illustrate the specific implementation of the devp2p protocol specification through an analysis of the relevant code in geth.

4.1 Node Initialization and Discovery

4.1.1 Node Startup and Configuration Loading

A participant in the network (i.e., a node) is typically a computer running Ethereum client software (such as Geth). Each node has its unique identity identifier, primarily including enode ID (node ID) and ENR (Ethereum Node Record).

Specifically, enode ID is a unique identifier generated from the node's own encryption key pair (usually a secp256k1 public key) through a specific hashing operation. ENR is a scalable record constructed and signed by the node itself. It encapsulates the node's network contact information (such as IP address, TCP/UDP ports for P2P communication—usually defaulting to 30303, as well as node ID and public key) and its service capabilities, acting as its business card in the network. These ENRs are subsequently used by the node discovery protocol (such as Discovery v4/v5) to help nodes locate and establish connections with each other. The execution layer nodes include the following types:

• Bootnode: The primary responsibility of these nodes is to help new nodes join the p2p network. Their ENR information is usually hardcoded in the client's configuration file or provided through the DNS discovery mechanism as entry points for new nodes to connect to the network for the first time.

• Local Node: Refers to the Ethereum node instance that the user is currently running and controlling directly.

• Remote Node: Refers to all other participating nodes in the Ethereum p2p network, excluding the local node.

The program entry point for the Geth node is located in cmd/geth/main.go. It then loads the node's configuration file, which includes network ID, port, seed nodes, boot nodes, etc.

Here, the main function parses command-line arguments, creates, and configures the node. The app.Run in main will default to executing the geth function. In the geth function, it involves makeFullNode and startNode, where makeFullNode mainly loads the configuration (makeConfigNode) and creates the core services of the node, including the Ethereum service (utils.RegisterEthService). The startNode is responsible for starting the configured node.

Configuration loading mainly occurs in makeFullNode and makeConfigNode, combining default configurations, configuration files, and command-line parameters. The cmd/geth/makeFullNode → makeConfigNode → loadBaseConfig is responsible for loading the base configuration. The structures related to node configuration and default values are defined in node/config.go, and the boot node list is hardcoded in params/bootnodes.go.

func loadBaseConfig(ctx *cli.Context) gethConfig {
// Load default configuration
cfg := gethConfig{
Eth: ethconfig.Defaults,
Node: defaultNodeConfig(),
Metrics: metrics.DefaultConfig,
}
// Load configuration file (if there is a TOML file)
if file := ctx.String(configFileFlag.Name); file != "" {
if err := loadConfig(file, \&cfg); err != nil {
utils.Fatalf("%v", err)
}
}
// Apply node
utils.SetNodeConfig(ctx, \&cfg.Node)
return cfg
}

4.1.2 Node Discovery

In a dynamic network where nodes continuously join and leave, a node needs a mechanism to find other connectable nodes. Ethereum's node discovery protocol includes the three discovery protocols introduced above, which can be quickly reviewed in the following table:

|---------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------|--------------| | Protocol Name | Features | Key Code | | DiscV4 | A distributed hash table (DHT) based on Kademlia for node discovery in the network. | p2p/discover | | DiscV5 | An upgrade to Discv4, with native support for ENR. | p2p/discover | | DNS Discovery | Used for maintaining public boot node lists and guiding nodes in private networks. | p2p/enr |

The implementation of DNS Discovery is located in the p2p/dnsdisc package. The core is the dnsdisc.Client structure (defined in p2p/dnsdisc/client.go). The Client is responsible for interacting with the configured DNS server to obtain DNS TXT records that point to an ENR tree. The client parses this tree to obtain the node list. The p2p.Server uses dnsdisc.Client to poll the DNS list and add discovered nodes to the dialing candidate pool.

The implementation of Discv4 is located in p2p/discover/v4_udp.go.

• ping is used to send PING messages to check if nodes are online and obtain their ENR sequence numbers.

• findnode sends FINDNODE requests to target nodes to obtain a set of nodes closest to a specific target in their Kademlia buckets.

• LookupPubkey looks up the nodes closest to a given public key in the DHT network.

• ListenV4 is used to start listening and processing for the v4 discovery protocol.

The implementation of Discv5 is located in p2p/discover/v5_udp.go.

• ping is the PING operation for version 5.

• Findnode is the FINDNODE operation for version 5, allowing queries for nodes at specific distances.

• Lookup is the node lookup for version 5.

• ListenV5 (the actual line number may vary by version) is responsible for running the v5 protocol.

p2p/discover/table.go implements the Kademlia routing table for storing and managing discovered node information. p2p/discover/v4wire and p2p/discover/v5wire define the network message formats for the Discovery v4/v5 protocols.

In the p2p/server.go file, the Server.setupDiscovery method initializes and starts the node discovery service based on the node configuration, which may include Discovery v4/v5. This method calls the corresponding discover.ListenV4 or discover.ListenV5 functions to start UDP listening for discovery messages from other nodes.

When a node needs to find new peer nodes or locate a specific node, it triggers the corresponding lookup operation, such as Lookup in discv5 or LookupPubkey in discv4. These operations are an iterative querying process: the node sends FINDNODE requests to a batch of nodes in its routing table that are known and closest to the target. Upon receiving the NEIGHBORS (discv4) or NODES (discv5) responses from the peer nodes, the node updates its routing table with the newly discovered, closer node information and continues querying these new nodes. This iterative process continues until the target node is found or no closer nodes than the currently known ones can be found.

Additionally, to maintain the health and timeliness of the routing table, nodes periodically send PING messages to other nodes in the table to check their online status and refresh relevant information. Receiving a PONG message from the other party confirms that the node is still active. In some cases, such as when needing to obtain the latest details of a node, a node can also send a REQUESTENR message to request the latest ENR information from the other party.

4.2 Startup and Connection Handshake of p2p Service

4.2.1 Initializing p2p Service

The p2p service is initialized according to the configuration, setting local node information, encryption keys, etc.

The initialization process of the Ethereum p2p service mainly involves the node/node.go file's node.New function and the node.Node.Start method. First, within the node.New function, the system creates and configures an instance of p2p.Server based on the incoming configuration (config.P2P).

Subsequently, in the node.Node.Start method, the openEndpoints step calls the Server.Start method located in the p2p/server.go file. The Server.Start method serves as the entry point for starting the core functionalities of the p2p service, including starting network listening, node discovery services, etc. During the execution of Server.Start, a series of critical settings and initialization operations are performed. For example, the local node's information is configured and prepared through srv.setupLocalNode, the private key used for node authentication is loaded into srv.PrivateKey, and the dialing scheduler for connections is set up through srv.setupDialScheduler.

Finally, the srv.setupListening method is called to start the listening loop. This listening loop can be seen as the main event loop of the p2p service, continuously running and responsible for handling all p2p network activities, including responding to node discovery requests, managing the handshake process for new connections, and processing message passing between nodes.

4.2.2 Connection Establishment and Communication

The establishment of connections between Ethereum nodes is a multi-stage process that ensures communication security and efficiency. At the basic connection management level, data exchange between nodes relies on standard TCP connections. Nodes can actively initiate connections to the target node's IP address and TCP port through "dialing" operations while also accepting inbound connection requests by "listening" on specific TCP ports. To effectively manage network resources and connection stability, clients typically configure parameters such as maximum connection numbers and maximum dialing attempts for nodes.

The underlying TCP connection establishment is handled by Go's net package. The conn structure under server.go encapsulates the underlying net.Conn and adds states and flags related to the p2p protocol, such as inboundConn and trustedConn.

// conn wraps a network connection with information gathered
// during the two handshakes.
type conn struct {
fd net.Conn // Underlying network connection
transport // Defines the interfaces for encryption handshake doEncHandshake and protocol handshake doProtoHandshake, as well as message read/write MsgReadWriter.
node *enode.Node // Representation of the remote node's enode.Node
flags connFlag // Connection flags, such as inboundConn, staticDialedConn, trustedConn, etc.
cont chan error // Used to pass errors between run loop and SetupConn.
caps []Cap // Capabilities supported by the remote node after protocol handshake.
name string // Name of the remote node after protocol handshake.
}

For listening and accepting inbound connections, the server's listenLoop method continuously listens on the pre-configured TCP address and port. Once its internal listener.Accept successfully receives a new TCP connection request, the system first performs some preliminary validity checks, such as verifying whether the requester's IP address is on a restricted list (blacklist) or whether its connection attempts are too frequent. Only connections that pass these preliminary checks will have a new goroutine started for them, and the Server.SetupConn method will be called to guide the connection into subsequent detailed settings and handshake stages.

Meanwhile, the p2p service will also actively initiate outbound connections based on its needs to maintain network status (such as keeping a sufficient number of peer nodes or connecting to specific static nodes). This task is primarily managed by the dialing scheduler (dialScheduler) located in the p2p/dial.go file. This scheduler selects suitable target nodes and initiates TCP connection attempts. Once the TCP connection is successfully established, similar to the subsequent steps for handling inbound connections, the system will also call the Server.SetupConn method to configure this newly established outbound connection.

Regardless of whether the connection is inbound or outbound, once the basic TCP channel is established and preliminary checks are passed, it will enter the connection setup process managed by the Server.SetupConn method. In this method, the system first creates a p2p.conn object representing the p2p layer connection state for this new connection. Subsequently, to handle the details of the RLPx transport layer protocol, an rlpx.Conn object is created, encapsulating the underlying raw network connection (net.Conn). Finally, as a key step in establishing a secure communication channel, the system calls the doEncHandshake method of this conn object (which internally triggers the execution of rlpx.Conn.Handshake) to complete the RLPx encrypted handshake with the peer node, thereby formally establishing a secure communication session.

Once the connection is successfully established, both peer nodes must proceed with the subsequent handshake process to verify identities and negotiate communication parameters. This critical process is divided into two steps:

• RLPx Handshake: This step aims to establish a secure, encrypted communication channel through the RLPx protocol. It is achieved through the following interactions: the initiator first sends an auth message containing its temporary public key and signature; the receiver verifies the signature, generates its temporary key pair, and replies with an ack message containing its temporary public key. Based on the temporary public keys exchanged by both parties and their respective temporary private keys, they compute a shared key using the Elliptic Curve Diffie-Hellman Key Exchange (ECDH) algorithm. After this, all data transmitted over this TCP connection will be symmetrically encrypted using this shared key, ensuring the confidentiality of the communication content.

• Protocol Handshake (also known as application layer handshake): After the secure establishment of the RLPx encrypted channel, the nodes need to further clarify the application layer sub-protocols they support and their versions, such as eth/66, snap/1, etc. To do this, both parties send a Hello message to each other, detailing their node IDs, client software versions, and supported protocol lists (referred to as Capabilities or Caps). By comparing this information, nodes select the highest version of the protocol they both support as the standard for subsequent specific data exchanges (such as blocks, transactions, etc.) over this connection.

4.2.3 RLPx Handshake

The handshake process of the RLPx protocol is a key step in ensuring the security of p2p communication between Ethereum nodes, and its implementation is located in p2p/rlpx/rlpx.go. The entire process revolves around the Conn object, which encapsulates the underlying TCP connection and manages session states, including encryption keys and MAC keys. The entry point for the handshake is the Conn.Handshake method, which calls either the runInitiator or runRecipient function to execute specific handshake logic based on whether the node is the connection initiator or receiver. During the entire handshake, the rlpx.handshakeState object temporarily stores necessary intermediate state information, such as temporary key pairs and Nonces.

// protoHandshake is the RLP structure of the protocol handshake.
type protoHandshake struct {
Version uint64 // DevP2P protocol version.
Name string // Client name and version (e.g., "Geth/v1.10.0/…").
Caps []Cap // List of sub-protocol capabilities supported by the node, each Cap containing protocol name (e.g., "eth") and version (e.g., 66).
ListenPort uint64 // Listening port of the node.
ID []byte // secp256k1 node's public key (Enode ID).
// Ignore additional fields (for forward compatibility).
Rest []rlp.RawValue `rlp :"tail"`
}

Handshake process:

1. Initiator: runInitiator generates a temporary Elliptic Curve Diffie-Hellman (ECDH) key pair and a random Nonce initNonce, and calculates the static shared key staticSharedSecret with the receiver. It then signs the XOR value of the static shared key and initNonce using its temporary ECDH private key. Finally, it packages this signature, its static public key InitiatorPubkey, and initNonce into an authMsgV4 message. This message is then encrypted using the receiver's static public key via the Elliptic Curve Integrated Encryption Scheme (ECIES) and sent.

2. Receiver: runRecipient receives and decrypts authMsgV4, using the InitiatorPubkey, Nonce initNonce, and the locally computed static shared key to verify the signature and recover the initiator's temporary public key remoteRandomPub. The receiver then generates its own temporary ECDH key pair and random Nonce respNonce. It constructs an authRespV4 message containing its temporary public key RandomPubkey and respNonce. Finally, it encrypts authRespV4 using the initiator's static public key via ECIES and sends it.

3. Key Derivation: Both parties first compute the ECDHE shared key ecdheSecret using their respective temporary private keys and the other party's temporary public key. Then, combining ecdheSecret, respNonce, and initNonce, they derive the main shared secret sharedSecret, AES key aesSecret, and MAC key macSecret through Keccak256 hashing. Finally, both parties use macSecret to XOR with respNonce and initNonce, and combine it with the initial authentication messages sent and received (either authMsg or authRespMsg) to initialize the outbound egressMAC and inbound ingressMAC Keccak256 hash instances for subsequent message integrity checks.

4. Handshake Completion: At this point, both parties possess symmetric AES keys and MAC keys that can be used for encrypting and verifying the integrity of subsequent RLPx frame data, marking the handshake as successful. In the implementation in p2p/server.go, Server.checkpoint pushes this connection to the checkpointPostHandshake channel after a successful RLPx handshake, awaiting a higher-level devp2p protocol layer handshake.

Once the handshake is successful, application layer messages between nodes are encapsulated into frames via the RLPx protocol. Larger messages are split into one or more frames for transmission, with each frame containing necessary metadata (such as message codes and lengths) and the encrypted message body. This entire process ensures that even over insecure networks, two nodes can securely negotiate session keys for subsequent communication encryption and integrity protection.

4.2.4 DevP2P Application Sub-protocol Handshake

After the successful encrypted handshake at the RLPx transport layer, nodes immediately proceed with the handshake at the DevP2P protocol layer. The core of this process is completed by exchanging a Hello message (with a message code of 0x00), aimed at allowing both nodes to understand each other's capabilities and identities.

The "Hello" message (the protoHandshake structure) includes the protocol version of the node, client name (e.g., "Geth/v1.10.0"), supported sub-protocols and versions (Capabilities), listening port, and node ID (public key).

In the p2p/peer.go file, each Peer object manages the main communication loop with the peer node through its run method. At the start of this loop, i.e., at the beginning of the connection establishment, the Peer.handshake method is called. This method is responsible for actively sending the local node's Hello message to the peer while simultaneously waiting to receive the Hello message from the other party.

Once both parties have completed the exchange of Hello messages, the server-side logic (in the Server.run method, through checkpoints like checkpointPostHandshake and checkpointAddPeer) further verifies the validity of the handshake results. These checks include confirming whether the connection is still alive, whether the other party is the node itself (to prevent self-connections), and whether the current connection count has reached the limit, etc. The specific check logic can be referenced in functions like Server.postHandshakeChecks and Server.addPeerChecks.

Finally, both nodes compare the lists of supported sub-protocols declared in their respective Hello messages. Through this comparison, they negotiate the common supported sub-protocols and their versions, and based on this, they will run these selected sub-protocols in subsequent communications for operations such as block synchronization and transaction broadcasting.

After the successful handshake of the basic DevP2P protocol, if both nodes support certain common sub-protocols (such as the eth protocol), they will perform specific handshakes for these sub-protocols.

In p2p/server.go, when a new peer node successfully completes the basic handshake and passes the checks (logic of checkpointAddPeer), the Server.launchPeer function is called, which will start a corresponding handling routine for each successfully negotiated sub-protocol. It will iterate through srv.Protocols and match them with the capabilities of the peer node (c.caps). During the matching, a protoRW instance is created, and the protocol's communication loop is started with go p.run() (see peer.go's Peer.startProtocols).

Specific Handshake and Message Processing of the ETH Protocol:

The ETH protocol also has a specific handshake phase that occurs immediately after the exchange of the DevP2P Hello messages. This handshake is completed by exchanging Status messages, which contain key blockchain state information such as the node's network ID, total difficulty (TD) of the chain, and the current best block hash, used for quickly assessing compatibility and synchronization needs between nodes. This Status exchange logic is mainly implemented in the eth/protocols/eth/handshake.go's Handshake function.

The structure of the Status message is as follows:

type StatusPacket struct {
ProtocolVersion uint32 // Version of the eth protocol
NetworkID uint64 // ID of the network (e.g., 1 for mainnet)
TD *big.Int // (Total Difficulty) Total difficulty of the chain
Head common.Hash // Hash of the head block
Genesis common.Hash // Hash of the genesis block
ForkID forkid.ID // ID of the fork
}

Once the ETH protocol's Status handshake is successful, subsequent communications are led by the Handle function in eth/protocols/eth/handler.go (and its internal handleMessage method). The Handle function enters a message processing loop, responsible for receiving and processing various messages defined under this ETH sub-protocol. These message types are diverse, such as NewBlockHashesMsg for broadcasting new block hashes, GetBlockHeadersMsg for requesting block headers, BlockBodiesMsg for transmitting block bodies, and TransactionsMsg for processing transactions (the specific message codes and handling functions typically correspond to protocol versions, such as eth/68). Through this loop, nodes can perform core operations like block synchronization and transaction propagation.

To support different versions of the ETH protocol (such as eth/66, eth/67, eth/68), the MakeProtocols function in handler.go will create corresponding p2p.Protocol definitions for each version. When a new peer node connects and negotiates to use a specific version of the ETH protocol, the Run method in that p2p.Protocol definition serves as the execution entry point for this sub-protocol. The Run method typically initializes a Peer object specific to the ETH protocol (which is a wrapper around the generic p2p.Peer, adding ETH protocol-related states and functionalities) and then calls backend.RunPeer(peer, Handle) to start the ETH protocol message processing main loop for that peer node, passing the Handle function as the core callback.

SNAP Protocol:

In addition to the ETH protocol, the Geth client also supports the SNAP protocol, primarily used to accelerate state snapshot synchronization for nodes. Its implementation is located in the eth/protocols/snap directory. Similar to the ETH protocol, the SNAP protocol also has its specific message processing logic, mainly handled by the Handle and HandleMessage functions in that directory, specifically dealing with messages related to state snapshot data requests, transmission, and verification.

4.3 Data Synchronization and Processing

The main structure of data processing and synchronization is illustrated in the following diagram:

4.3.1 Data Synchronization

Data synchronization is the core process for new nodes joining the network or lagging nodes catching up with the latest chain state. In Geth, this process is mainly handled by the eth/downloader/downloader.go module.

Synchronization Startup and Mode Selection

The synchronization process is primarily initiated through the Downloader.synchronise method. Geth supports various synchronization modes, such as Full Sync and Snap Sync, with the specific mode controlled by the d.mode variable, which can be obtained through the Downloader.getMode method. Notably, in Snap Sync mode, Geth first disables writing to the Trie database and pauses snapshot maintenance to ensure data consistency.

Peer Management during Synchronization

Geth manages peer nodes during the synchronization process through the following methods: the Downloader.RegisterPeer method is used to register new downloading nodes, which are stored in d.peers (an instance of peerSet). Correspondingly, the Downloader.UnregisterPeer method is used to remove nodes. Additionally, the peerDropFn callback function is triggered when node behavior is inappropriate (such as sending invalid data) to promptly remove it.

Data Retrieval and Scheduling

Data retrieval scheduling is managed by d.queue (an instance of queue), which manages the hashes of data to be downloaded. Geth defines constants such as MaxBlockFetch, MaxHeaderFetch, and MaxReceiptFetch to limit the amount of data that can be retrieved in a single request. The fetchRequest structure represents an ongoing data retrieval operation.

Specific Optimizations during Synchronization

After the Ethereum merge, Geth introduced Skeleton Sync, which efficiently synchronizes the header information of the beacon chain and uses Downloader.findBeaconAncestor to find common ancestors. Meanwhile, in Snap Sync mode, Geth selects a pivotHeader as the starting point for state synchronization.

Synchronization Control and Event Notification

Geth provides a comprehensive control mechanism to manage the lifecycle of synchronization. The d.cancelCh channel can be used to abort the synchronization process midway. The d.syncStatsChainOrigin and d.syncStatsChainHeight record the starting and target heights of synchronization, facilitating progress tracking. Additionally, d.mux (an instance of event.TypeMux) is responsible for broadcasting synchronization-related events, such as StartEvent (synchronization starts), DoneEvent (synchronization completes), and FailedEvent (synchronization fails), which are crucial for client status display and external monitoring.

4.3.2 Data Processing and Verification

In the Geth client, once data is obtained from the network, it needs to be processed, verified, and ultimately integrated into the local blockchain.

Block Header Processing

Downloaded block headers (types.Header) are passed to a dedicated header processor through the d.headerProcCh channel. The headerTask structure encapsulates a batch of downloaded block headers and their pre-computed hash values. Geth limits the number of block headers imported into the chain at once through the maxHeadersProcess constant to optimize processing efficiency.

Block and Receipt Processing

The fetchResult structure is responsible for collecting partial download results, including block bodies, uncle blocks, transactions, receipts, and withdrawals, until all parts are fully retrieved. Once the data is ready, the InsertChain method is used to insert a batch of blocks into the local chain, while the InsertReceiptChain method is used to insert a batch of blocks and their corresponding receipts into the local chain. Notably, ancientLimit defines the maximum block number considered "ancient," and data earlier than this number may be directly written into specialized ancient storage in Snap Sync mode. chainCutoffNumber and chainCutoffHash define the cutoff points for segments of the chain that can be skipped during synchronization.

Data Verification and Error Handling

Geth performs strict verification on the retrieved data and defines various error types to indicate verification failures, such as:

• errInvalidChain: The retrieved hash chain is invalid.

• errInvalidBody: The retrieved block body is invalid.

• errInvalidReceipt: The retrieved receipt is invalid.

If Geth detects invalid data or malicious behavior from a peer node, it will immediately call the dropPeer callback function to remove the corresponding node, maintaining network health. The badBlockFn callback function is used in asynchronous beacon synchronization to notify the caller that a block has been rejected due to verification failure.

State Synchronization

d.SnapSyncer (an instance of snap.Syncer) is specifically responsible for downloading and processing state data during snapshot synchronization. The d.stateSyncStart channel is used to initiate the state synchronization process, while d.stateFetcher (a goroutine started in the New method) is responsible for obtaining state data.

Data Submission and Exception Handling

The d.committed boolean value indicates whether the data has been finally submitted. In Snap Sync mode, data may not be finally submitted until synchronization is fully completed to ensure atomicity. The SnapSyncCommitHead method is used to directly submit the head block during snapshot synchronization. Possible error types during synchronization include errBusy (the downloader is busy), errTimeout (timeout), and errCanceled (synchronization is canceled), and Geth handles these exceptions appropriately.

4.3.3 Updating Local Chain State/Database

Verified data is written to the local database, thereby updating the node's chain state. The updating of local chain state and database in Geth is primarily achieved through the following mechanisms:

Block and Receipt Insertion

• The Downloader structure relies on a BlockChain interface that defines methods for inserting blocks and receipts.

• BlockChain.InsertChain inserts a batch of blocks into the local chain.

• BlockChain.InsertReceiptChain inserts a batch of blocks and their receipts into the local chain. Data earlier than ancientLimit will be directly stored in ancient storage.

• BlockChain.InsertHeadersBeforeCutoff inserts a batch of block headers into ancient storage before the configured chainCutoffNumber.

• The processFullSyncContent and processSnapSyncContent functions (in fetcher.go, but called by downloader) are responsible for processing the downloaded blocks and receipts and calling the aforementioned BlockChain interface methods to insert them into the database.

Snap Sync State Updates

• The Downloader includes a Downloader.SnapSyncer (of type snap.Syncer) specifically responsible for downloading and applying state during Snap Sync.

• During Snap Sync, state data (accounts, storage, bytecode, etc.) is directly written to Downloader.stateDB.

• In the syncToHead function, if Snap Sync mode is adopted, it calls rawdb.WriteLastPivotNumber to write the pivot block number to the database, allowing for re-enabling Snap Sync during rollbacks.

• The SnapSyncer.Progress() method is used to obtain the progress of Snap Sync, including the number of accounts, bytecode, storage, etc., that have been synchronized.

Ancient Store and Chain Cutoff

• Downloader.ancientLimit: Defines the maximum block number that can be considered ancient data. In Snap Sync mode, data earlier than this limit will be directly written to ancient storage.

• Downloader.chainCutoffNumber and Downloader.chainCutoffHash: Define cutoff points for skipping block bodies and receipts during synchronization.

Database Submission

• The Downloader.committed flag: In Snap Sync mode, if the pivot block number is not 0, initially committed is false, indicating that the state in the database may be incomplete until synchronization is complete.

4.3.4 Peer Management

Ethereum nodes maintain the stability and efficiency of their P2P network connections through peer management. A remote node that successfully establishes a connection with the local node and completes all necessary handshake processes becomes a peer node. Effective peer management is crucial for ensuring network connectivity, facilitating smooth data exchange, and optimizing the node's resource utilization. This management process primarily encompasses the following core aspects:

• Peer Identification and Connection Pool Maintenance: Once the local node successfully completes all handshake steps with a remote node and confirms it as a peer, the node maintains a list or pool of all currently active peer nodes (Connection Pool). This connection pool serves as the dynamic foundation for all network communications of the node.

• Connection Operations and Access Policies: Nodes perform a series of connection operations, such as actively "dialing" outbound or "listening" and accepting inbound requests to add new peer nodes. At the same time, nodes decide to disconnect and remove corresponding peer nodes based on communication errors, timeouts, the other party being offline, or preset access policies (e.g., prioritizing connections to "trusted" peers or limiting total connection numbers and connections per single IP).

• Response to Peer Node Status Events: Various status changes of peer nodes in the system, such as the successful establishment of new connections, unexpected disconnections of existing connections, or errors occurring during communication, typically trigger internal event notifications. These events enable other related modules in the client software (such as those responsible for blockchain synchronization logic or transaction pool management) to promptly perceive real-time dynamics of the P2P network topology and make necessary responses or adjustments accordingly.

Peer Management

• Downloader.RegisterPeer injects new peers into the download source set.

• Downloader.UnregisterPeer removes peers from the known list and attempts to return the pending fetch tasks of that peer to the queue.

• Downloader.dropPeer is called when inappropriate behavior from a peer is detected (such as sending invalid data) to remove that peer.

Error types include:

errBusy // Indicates that the downloader is busy and cannot process new synchronization requests
errBadPeer // Indicates that operations from a bad peer are ignored
errTimeout // Indicates that the operation timed out
errInvalidChain // Indicates that the retrieved hash chain is invalid
errInvalidBody // Indicates that the retrieved block body is invalid
errInvalidReceipt // Indicates that the retrieved receipt is invalid
errCancelStateFetch // Indicates that state retrieval was canceled
errCancelContentProcessing // Indicates that content execution was canceled
errCanceled // Indicates that synchronization was canceled
errNoPivotHeader // Indicates that no pivot header was found

Cancellation Mechanism

• Downloader.cancelCh is a channel used to cancel ongoing synchronization operations.

• Downloader.Cancel (called in the defer of the synchronise function) ensures that cancelCh is closed when the synchronization process exits and waits for all fetcher goroutines to exit.

• cancelLock and cancelWg are used to protect concurrent access to cancelCh and ensure that goroutines exit correctly.

Handling Invalid Data and Bad Blocks

• When data (such as block headers, block bodies, receipts) received from peers fails verification, corresponding errors (such as errInvalidChain, errInvalidBody, errInvalidReceipt) are returned.

• Downloader.badBlock is used to notify the caller in asynchronous beacon sync that the source header for synchronization has produced a chain containing bad blocks.

• Functions like queue.DeliverHeaders, queue.DeliverBodies, queue.DeliverReceipts (in queue.go) perform data validation, and if validation fails, may lead to the peer being marked as bad or removed.

Synchronization Failure Events

• When the synchronization process unexpectedly terminates due to any error, Geth publishes a FailedEvent event through d.mux.Post(FailedEvent{err}), which includes specific error information. This allows other modules of Geth or external monitoring tools to promptly perceive changes in synchronization status and respond accordingly.

5. Key Steps in Testing

Geth includes a large number of test cases, and we can directly test the key steps mentioned above.

5.1 Node Discovery Testing

In the p2p/discover directory

v4_udp_test.go contains tests for the UDP-based v4 discovery protocol.

• TestUDPv4_findnode tests the findnode request, which is core to node discovery. It checks whether it can find and return the nearest neighbor nodes.

• TestUDPv4_pingTimeout tests the timeout handling of the ping request, where ping/pong is used to check if nodes are online.

• TestUDPv4_packetErrors tests various packet error handling, such as expired packets, unsolicited replies, etc.

v5_udp_test.go contains tests for the UDP-based discv5 discovery protocol.

• TestUDPv5_lookupE2E conducts end-to-end node lookup tests to ensure nodes can successfully discover other nodes in the network.

• TestUDPv5_pingHandling tests the handling of PING requests and the generation of PONG replies.

• TestUDPv5_unknownPacket tests the handling logic when receiving packets from unknown sources, which typically triggers the handshake process.

table_test.go includes tests for operations on the routing table (Kademlia table), such as adding, deleting, and looking up nodes.

v4_lookup_test.go is used for testing the node lookup logic in the discv4 discovery protocol.

5.2 Connection Establishment

handshake_test.go is dedicated to testing the handshake process of the Ethereum protocol (eth protocol).

• It typically includes tests for encoding/decoding handshake messages, state transitions (e.g., from waiting state to handshake success/failure), version negotiation, capability announcements, etc.

v5_udp_test.go (also related to handshakes):

• TestUDPv5_handshakeRepeatChallenge tests the handling logic when receiving repeated challenges during the handshake process, ensuring the robustness of the handshake.

• TestUDPv5_unknownPacketKnownNode tests the behavior when receiving unknown packets from known nodes, which may also relate to handshakes or session re-establishment.

5.3 RLPx Handshake Testing

RLPx tests (encryption handshake, message sending and receiving, etc.) are primarily located in the p2p/rlpx_test.go file:

• TestHandshake tests the handshake, including creating peer nodes.

• createPeers creates peer nodes and attempts to perform handshakes.

• doHandshake conducts handshake tests.

5.4 Peer Management Testing

Mainly focused on p2p/server_test.go, p2p/discover/v4_udp_test.go, p2p/discover/v4_lookup_test.go, and p2p/discover/v5_udp_test.go.

• For example, server_test.go covers core logic for adding and removing nodes, trust management, and connection establishment.

• v4_udp_test.go and v5_udp_test.go primarily test the node discovery protocols (such as Ping/Pong, FindNode, Neighbors, Whoareyou, etc.), including node lookup, handshakes, and packet handling exceptions.

• v4_lookup_test.go focuses on testing the node lookup process, lookup iterators, and the correctness and ordering of lookup results.

6. Conclusion

Geth's network core is built on the DevP2P protocol stack, a set of protocols designed specifically for communication at the Ethereum network layer. Nodes first discover peer nodes through various mechanisms, primarily including the Kademlia-based distributed hash table (such as Discv4/v5 protocols) and DNS seed nodes. Once potential peer nodes are discovered, Geth attempts to establish TCP connections. After successfully establishing TCP connections, an RLPx encrypted handshake is performed to ensure the confidentiality and integrity of the communication and negotiate session keys.

After the RLPx handshake, nodes proceed with the handshake at the DevP2P protocol layer. At this stage, nodes exchange the sub-protocols they support and their versions, such as eth (Ethereum main protocol), snap (fast synchronization protocol), and les (light client protocol). Both parties negotiate to use the highest version of the sub-protocol they mutually support for subsequent communication. This process ensures that nodes with different capabilities can effectively interact.

Once the negotiation is successful, nodes exchange specific data through the selected sub-protocols. For example, through the eth protocol, nodes synchronize core data such as blocks, transactions, and chain states, which is the foundation of Ethereum's network functionality. The snap protocol allows nodes to synchronize state data more efficiently, accelerating the initial synchronization process for new nodes. Geth continuously manages its list of peer nodes, monitors connection statuses, and dynamically adjusts connections based on network conditions and node behaviors.

In summary, Geth's network module is a layered and modular system, from the underlying node discovery and secure connection establishment to the upper-layer specific functional sub-protocol negotiation and data exchange, collectively forming a robust and efficient decentralized network communication framework. This design allows Geth to flexibly adapt to network changes and support the ongoing development of the Ethereum ecosystem.

References

[1]https://github.com/ethereum/go-ethereum

[2]https://github.com/ethereum/devp2p

[3]https://www.geeksforgeeks.org/what-is-the-difference-between-libp2p-devp2p-and-rlpx/

[4]Welcome to go-ethereum | go-ethereum

[5]https://ethereum.org/zh/developers/docs/