Author: Ray, LXDAO

This article will explore the design architecture of the execution layer client Geth and the startup process of Geth nodes. The speed of Geth code updates is rapid, and the code seen later may differ, but the overall design remains consistent, and new code can be read with the same approach.

Source code version: https://github.com/ethereum/go-ethereum/commit/c8a9a9c0917dd57d077a79044e65dbbdd421458b

1. Ethereum Client

Before the The Merge upgrade, Ethereum had only one client, which was responsible for executing transactions and also for the consensus of the blockchain, ensuring that new blocks were produced in a certain order. After the The Merge upgrade, the Ethereum client was divided into the execution layer and the consensus layer. The execution layer is responsible for executing transactions, maintaining state and data, while the consensus layer is responsible for implementing consensus functions, with communication between the execution layer and consensus layer via APIs. Each layer has its own specifications, and clients can be implemented in different languages, but must conform to the corresponding specifications, with Geth being one implementation of the execution layer client. The current mainstream implementations of execution layer and consensus layer clients are as follows:

Execution Layer

• Geth: Maintained by a team directly funded by the Ethereum Foundation, developed in Go language, recognized as the most stable and time-tested client.
• Nethermind: Developed and maintained by the Nethermind team, developed in C#, initially funded by the Ethereum Foundation and Gitcoin community.
• Besu: Originally developed by ConsenSys's PegaSys team, now a Hyperledger community project, developed in Java.
• Erigon: Developed and maintained by the Erigon team, funded by the Ethereum Foundation and BNB Chain. Forked from Geth in 2017, aimed at improving synchronization speed and disk efficiency.
• Reth: Led by Paradigm, developed in Rust, emphasizing modularity and high performance, now nearing maturity and usable in production environments.

Consensus Layer

• Prysm: Maintained by Prysmatic Labs, one of the earliest consensus layer clients for Ethereum, developed in Go, focusing on usability and security, initially funded by the Ethereum Foundation.
• Lighthouse: Maintained by the Sigma Prime team, developed in Rust, emphasizing high performance and enterprise-level security, suitable for high-load scenarios.
• Teku: Initially developed by ConsenSys's PegaSys team, later became part of the Hyperledger Besu community, developed in Java.
• Nimbus: Developed and maintained by the Status Network team, developed in Nim, optimized for resource-constrained devices (such as mobile phones and IoT devices), aiming for lightweight operation in embedded systems.

2. Introduction to the Execution Layer

The Ethereum execution layer can be viewed as a transaction-driven state machine, with its most basic function being to update state data by executing transactions through the EVM. In addition to transaction execution, it also involves saving and verifying blocks and state data, running a p2p network, and maintaining a transaction pool.

Transactions are generated by users (or programs) in a format defined by the Ethereum execution layer specifications. Users need to sign the transactions, and if the transaction is valid (nonce is sequential, signature is correct, gas fee is sufficient, business logic is correct), the transaction will ultimately be executed by the EVM, thereby updating the state of the Ethereum network. Here, the state refers to the collection of data structures, data, and databases, including external account addresses, contract addresses, address balances, as well as code and data.

The execution layer is responsible for executing transactions and maintaining the state after transaction execution, while the consensus layer is responsible for selecting which transactions to execute. The EVM serves as the state transition function within this state machine, with inputs coming from various sources, potentially including the latest block information provided by the consensus layer or blocks downloaded from the p2p network.

The consensus layer and execution layer communicate via the Engine API, which is the only communication method between the execution layer and the consensus layer. If the consensus layer obtains block production rights, it will use the Engine API to allow the execution layer to produce new blocks; if it does not obtain block production rights, it will synchronize the latest blocks for the execution layer to verify and execute, thereby maintaining consensus with the entire Ethereum network.

Logically, the execution layer can be divided into six parts:

• EVM: Responsible for executing transactions; transaction execution is the only way to modify state data.
• Storage: Responsible for storing state and block data.
• Transaction Pool: Used for temporarily storing user-submitted transactions, which will be propagated between different nodes via the p2p network.
• p2p Network: Used for discovering nodes, synchronizing transactions, downloading blocks, and other functions.
• RPC Service: Provides access capabilities to nodes, such as users sending transactions to nodes and interactions between the consensus layer and execution layer.
• BlockChain: Responsible for managing Ethereum's blockchain data.

The following diagram illustrates the key processes of the execution layer and the functions of each part:

For the execution layer (here we only discuss Full Node), there are three key processes:

• If it is a newly joined Ethereum node, it needs to synchronize blocks and state data from other nodes via the p2p network. If it is a Full Sync, it will start downloading blocks one by one from the genesis block, verifying blocks, and rebuilding the state database via the EVM. If it is Snap Sync, it will skip the entire block verification process and directly download the latest checkpoint state data and subsequent block data.
• If it is a node that has already synchronized to the latest state, it will continuously obtain the current latest produced blocks from the consensus layer via the Engine API, verify the blocks, then execute all transactions in the blocks via the EVM to update the state database and write the blocks to the local chain.
• If it is a node that has already synchronized to the latest state and the consensus layer has obtained block production rights, it will drive the execution layer to produce the latest blocks via the Engine API, fetching transactions from the transaction pool and executing them, then assembling them into blocks and passing them to the consensus layer via the Engine API, which will broadcast the blocks to the consensus layer p2p network.

3. Source Code Structure

The code structure of go-ethereum is quite large, but much of it consists of auxiliary code and unit tests. When studying the Geth source code, one only needs to focus on the core implementation of the protocol. The functions of various modules are as follows. Key modules to focus on include core, eth, ethdb, node, p2p, rlp, trie & triedb:

• accounts: Manages Ethereum accounts, including the generation of public-private key pairs, signature verification, address derivation, etc.
• beacon: Handles the interaction logic with the Ethereum Beacon Chain, supporting the functionality after the merge (The Merge) with proof of stake (PoS) consensus.
• build: Build scripts and compilation configurations (such as Dockerfile, cross-platform compilation support).
• cmd: Command line tool entry, containing multiple subcommands.
• common: General utility classes, such as byte processing, address format conversion, mathematical functions.
• consensus: Defines the consensus engine, including previous proof of work (Ethash) and single-node proof of stake (Clique) as well as Beacon engine, etc.
• console: Provides an interactive JavaScript console, allowing users to interact directly with Ethereum nodes via command line (such as calling Web3 API, managing accounts, querying blockchain data).
• core: Core logic of the blockchain, handling the lifecycle management of blocks/transactions, state machine, gas calculation, etc.
• crypto: Implementation of cryptographic algorithms, including elliptic curves (secp256k1), hashing (Keccak-256), signature verification.
• docs: Documentation (such as design specifications, API descriptions).
• eth: Complete implementation of the Ethereum network protocol, including node services, block synchronization (such as fast sync, archive mode), transaction broadcasting, etc.
• ethclient: Implements the Ethereum client library, encapsulating the JSON-RPC interface for Go developers to interact with Ethereum nodes (such as querying blocks, sending transactions, deploying contracts).
• ethdb: Database abstraction layer, supporting LevelDB, Pebble, in-memory databases, etc., for storing blockchain data (blocks, states, transactions).
• ethstats: Collects and reports node operational status to statistical services for monitoring network health.
• event: Implements event subscription and publishing mechanisms, supporting asynchronous communication between internal modules of the node (such as new block arrival, transaction pool updates).
• graphql: Provides GraphQL interface, supporting complex queries (replacing some JSON-RPC functionalities).
• internal: Internal tools or code that restricts external access.
• log: Logging system, supporting hierarchical log output and contextual log recording.
• metrics: Performance metrics collection (Prometheus support).
• miner: Mining-related logic, generating new blocks and packaging transactions (in PoW scenarios).
• node: Node service management, integrating the startup and configuration of p2p, RPC, database, and other modules.
• p2p: Implementation of peer-to-peer network protocol, supporting node discovery, data transmission, and encrypted communication.
• params: Defines Ethereum network parameters (mainnet, testnet, genesis block configuration).
• rlp: Implements Ethereum's specific data serialization protocol RLP (Recursive Length Prefix), used for encoding/decoding blocks, transactions, and other data structures.
• rpc: Implements JSON-RPC and IPC interfaces for external programs to interact with nodes.
• signer: Transaction signature management (hardware wallet integration).
• tests: Integration tests and state tests, validating protocol compatibility.
• trie & triedb: Implementation of Merkle Patricia Trie for efficient storage and management of account states and contract storage.

4. Execution Layer Module Division

There are two forms of external access to Geth nodes: one is through RPC, and the other is through Console. RPC is suitable for external users, while Console is suitable for node administrators. However, whether through RPC or Console, both utilize capabilities that have already been encapsulated internally, constructed in a layered manner.

The outermost layer is the API for external access to various capabilities of the node. The Engine API is used for communication between the execution layer and the consensus layer, the Eth API is for external users or programs to send transactions and obtain block information, and the Net API is for obtaining the status of the p2p network, etc. For example, if a user sends a transaction via the API, that transaction will ultimately be submitted to the transaction pool for management. Similarly, if a user needs to obtain block data, they will need to call the database capabilities to retrieve the corresponding block.

The next layer down is the implementation of core functionalities, including the transaction pool, transaction packaging, block production, block and state synchronization, etc. These functionalities rely on lower-level capabilities, such as the synchronization of the transaction pool, blocks, and states, which depend on the capabilities of the p2p network. The generation of blocks and the synchronization of blocks from other nodes require verification before being written to the local database, which relies on the capabilities of the EVM and data storage.

Core Data Structures of the Execution Layer

Ethereum

In eth/backend.go, the Ethereum structure is an abstraction of the entire Ethereum protocol, encompassing the main components of Ethereum, with the exception of the EVM, which is instantiated each time a transaction is processed and does not need to be initialized with the entire node. The Ethereum referred to in the following text is this structure:

type Ethereum struct {
// Ethereum configuration, including chain configuration
config *ethconfig.Config
// Transaction pool, where user transactions are submitted first
txPool *txpool.TxPool
// Used to track and manage local transactions
localTxTracker *locals.TxTracker
// Blockchain structure
blockchain *core.BlockChain
// Core component of the Ethereum node's network layer, responsible for handling all communications with other nodes, including block synchronization, transaction broadcasting and receiving, and managing peer node connections
handler *handler
// Responsible for node discovery and node source management
discmix *enode.FairMix
// Responsible for the persistent storage of blockchain data
chainDb ethdb.Database
// Responsible for publishing and subscribing to various internal events
eventMux *event.TypeMux
// Consensus engine
engine consensus.Engine
// Manages user accounts and keys
accountManager *accounts.Manager
// Manages log filters and block filters
filterMaps *filtermaps.FilterMaps
// Channel for safely closing filterMaps, ensuring resources are properly cleaned up when the node shuts down
closeFilterMaps chan chan struct{}
// Provides backend support for RPC API
APIBackend *EthAPIBackend
// Under PoS, collaborates with the consensus engine to validate blocks
miner *miner.Miner
// Minimum gas price accepted by the node
gasPrice *big.Int
// Network ID
networkID uint64
// Provides network-related RPC services, allowing querying of network status via RPC
netRPCService *ethapi.NetAPI
// Manages P2P network connections, handling node discovery and connection establishment, providing underlying network transmission capabilities
p2pServer *p2p.Server
// Protects concurrent access to mutable fields
lock sync.RWMutex
// Tracks whether the node has shut down normally, aiding recovery after abnormal shutdowns
shutdownTracker *shutdowncheck.ShutdownTracker
}

Node

In node/node.go, the Node is another core data structure that acts as a container, responsible for managing and coordinating the operation of various services. In the structure below, attention should be paid to the lifecycles field, where Lifecycle is used to manage the lifecycle of internal functionalities. For example, the above Ethereum abstraction needs to rely on Node to start and must register in lifecycles. This separation of specific functionalities from the abstraction of the node enhances the scalability of the entire architecture, and this Node needs to be distinguished from the Node in devp2p.

type Node struct {
eventmux *event.TypeMux
config *Config
// Account manager, responsible for managing wallets and accounts
accman *accounts.Manager
log log.Logger
keyDir string
keyDirTemp bool
dirLock *flock.Flock
stop chan struct{}
// p2p network instance
server *p2p.Server
startStopLock sync.Mutex
// Tracks the lifecycle state of the node (initializing, running, shut down)
state int
lock sync.Mutex
// All registered backends, services, and auxiliary services
lifecycles []Lifecycle
// List of currently provided APIs
rpcAPIs []rpc.API
// Different access methods provided for RPC
http *httpServer
ws *httpServer
httpAuth *httpServer
wsAuth *httpServer
ipc *ipcServer
inprocHandler *rpc.Server
databases map[*closeTrackingDB]struct{}
}
From an abstract perspective, the Ethereum execution layer, viewed as a world computer, needs to include three parts: network, computation, and storage. The components corresponding to these three parts in the Ethereum execution layer are:

• Network: devp2p
• Computation: EVM
• Storage: ethdb

devp2p

Ethereum is essentially a distributed system, with each node connected to other nodes via a p2p network. The implementation of the p2p network protocol in Ethereum is devp2p.

devp2p has two core functions: one is node discovery, allowing nodes to establish connections with other nodes when joining the network; the other is data transmission service, enabling data exchange once connections with other nodes are established.

In p2p/enode/node.go, the Node structure represents a node in the p2p network, where the enr.Record structure stores key-value pairs of detailed node information, including identity information (signature algorithm used for node identity, public key), network information (IP address, port number), supported protocol information (such as support for eth/68 and snap protocols), and other custom information, all encoded using RLP, with specific specifications defined in eip-778:

type Node struct {
// Node record, containing various properties of the node
r enr.Record
// Unique identifier of the node, 32 bytes in length
id ID

// hostname tracks the DNS name of the node
hostname string

// Node's IP address
ip netip.Addr
// UDP port
udp uint16
// TCP port
tcp uint16
}

// enr.Record
type Record struct {
// Sequence number
seq uint64
// Signature
signature []byte
// RLP encoded record
raw []byte
// Sorted list of all key-value pairs
pairs []pair
}
In p2p/discover/table.go, the Table structure is the core data structure for implementing the node discovery protocol in devp2p. It implements a distributed hash table similar to Kademlia for maintaining and managing node information in the network.
type Table struct {
mutex sync.Mutex
// Index known nodes by distance
buckets [nBuckets]*bucket
// Bootstrap nodes
nursery []*enode.Node
rand reseedingRandom
ips netutil.DistinctNetSet
revalidation tableRevalidation
// Database of known nodes
db *enode.DB
net transport
cfg Config
log log.Logger
// Periodically processes various events in the network
refreshReq chan chan struct{}
revalResponseCh chan revalidationResponse
addNodeCh chan addNodeOp
addNodeHandled chan bool
trackRequestCh chan trackRequestOp
initDone chan struct{}
closeReq chan struct{}
closed chan struct{}
// Interfaces for adding and removing nodes
nodeAddedHook func(*bucket, *tableNode)
nodeRemovedHook func(*bucket, *tableNode)
}

ethdb

ethdb abstracts Ethereum's data storage, providing a unified storage interface. The underlying database can be leveldb, pebble, or other databases. There can be many extensions as long as the interface level remains unified.

Some data (such as block data) can be read and written directly to the underlying database through the ethdb interface, while other data storage interfaces are built on top of ethdb. For example, a large portion of the database's data is state data, which is organized into MPT structures. In Geth, the corresponding implementation is trie. During the operation of the node, trie data generates many intermediate states, which cannot be directly read and written using ethdb; instead, triedb is used to manage these data and intermediate states, which are then persisted through ethdb.

In ethdb/database.go, the interface defining the read and write capabilities of the underlying database is specified, but does not include specific implementations, which will be provided by different databases themselves, such as leveldb or pebble. The Database defines two layers of data read and write interfaces, where the KeyValueStore interface is used for storing active, frequently changing data, such as the latest blocks and states. The AncientStore is used for handling historical block data, which rarely changes once written.

// Top-level interface for the database
type Database interface {
KeyValueStore
AncientStore
}
// KV data read and write interface
type KeyValueStore interface {
KeyValueReader
KeyValueWriter
KeyValueStater
KeyValueRangeDeleter
Batcher
Iteratee
Compacter
io.Closer
}
// Interface for reading and writing old data
type AncientStore interface {
AncientReader
AncientWriter
AncientStater
io.Closer
}

EVM

The EVM is the state transition function of the Ethereum state machine, and all state data updates can only be performed through the EVM. The p2p network can receive transaction and block information, which becomes part of the state database after being processed by the EVM. The EVM abstracts away the differences in underlying hardware, allowing programs to execute on different platforms' EVMs and achieve consistent results. This is a mature design approach, similar to the JVM in the Java language.

The EVM implementation consists of three main components: the EVM structure defined in core/vm/evm.go, which outlines the overall structure and dependencies of the EVM, including execution context and state database dependencies; the EVMInterpreter structure defined in core/vm/interpreter.go, which implements the interpreter responsible for executing EVM bytecode; and the Contract structure defined in core/vm/contract.go, which encapsulates the specific parameters for contract calls, including caller, contract code, input, etc., with all current opcodes defined in core/vm/opcodes.go:

// EVM
type EVM struct {
// Block context, containing block-related information
Context BlockContext
// Transaction context, containing transaction-related information
TxContext
// State database, used to access and modify account states
StateDB StateDB
// Current call depth
depth int
// Chain configuration parameters
chainConfig *params.ChainConfig
chainRules params.Rules
// EVM configuration
Config Config
// Bytecode interpreter
interpreter *EVMInterpreter
// Flag to abort execution
abort atomic.Bool
callGasTemp uint64
// Precompiled contract mapping
precompiles map[common.Address]PrecompiledContract
jumpDests map[common.Hash]bitvec
}

type EVMInterpreter struct {
// Pointer to the owning EVM instance
evm *EVM
// Opcode jump table
table *JumpTable
// Keccak256 hasher instance, shared between opcodes
hasher crypto.KeccakState
// Keccak256 hash result buffer
hasherBuf common.Hash
// Whether in read-only mode, where state modifications are not allowed
readOnly bool
// Return data from the last CALL, for reuse in subsequent calls
returnData []byte
}

type Contract struct {
// Caller address
caller common.Address
// Contract address
address common.Address

jumpdests map[common.Hash]bitvec
analysis bitvec
// Contract bytecode
Code []byte
// Code hash
CodeHash common.Hash
// Call input
Input []byte
// Whether it is a contract deployment
IsDeployment bool
// Whether it is a system call
IsSystemCall bool
// Available gas amount
Gas uint64
// Amount of ETH accompanying the call
value *uint256.Int
}

Other Module Implementations

The functionalities of the execution layer are implemented in a layered manner, with other modules and functionalities built on top of these three core components. Here are a few core modules introduced.

In the eth/protocols directory, the current implementation of Ethereum's p2p network sub-protocols is found. There are eth/68 and snap sub-protocols, which are built on top of devp2p.

eth/68 is the core protocol of Ethereum, with the protocol name being eth and 68 being its version number. On this basis, functionalities such as transaction pool (TxPool), block synchronization (Downloader), and transaction synchronization (Fetcher) are implemented. The snap protocol is used for quickly synchronizing blocks and state data when new nodes join the network, significantly reducing the startup time for new nodes.

ethdb provides the read and write capabilities of the underlying database. Due to the complex data structures in the Ethereum protocol, direct management of these data through ethdb is not feasible, so rawdb and statedb are implemented on top of ethdb to manage block and state data, respectively.

The EVM runs through all main processes, whether in block construction or block verification, requiring the execution of transactions via the EVM.

5. Geth Node Startup Process

The startup of Geth is divided into two phases: the first phase initializes the components and resources needed to start the node, and the second phase formally starts the node and provides services externally.

Node Initialization

When starting a Geth node, the following code is involved:

The initialization of each module is as follows:

• cmd/geth/main.go: Entry point for starting the Geth node.
• cmd/geth/config.go (makeFullNode): Loads configuration and initializes the node.
• node/node.go: Initializes the core container of the Ethereum node.
  • node.rpcstack.go: Initializes the RPC module.
  • accounts.manager.go: Initializes the accountManager.
• eth/backend.go: Initializes the Ethereum instance.
  • node/node.go OpenDatabaseWithFreezer: Initializes chaindb.
  • eth/ethconfig/config.go: Initializes the consensus engine instance (the consensus engine here does not actually participate in consensus but verifies the results from the consensus layer and handles validator withdrawal requests).
  • core/blockchain.go: Initializes the blockchain.
  • core/filterMaps.go: Initializes filtermaps.
  • core/txpool/blobpool/blobpool.go: Initializes the blob transaction pool.
  • core/txpool/legacypool/legacypool.go: Initializes the regular transaction pool.
  • core/txpool/locals/tx_tracker.go: Local transaction tracking (local transaction tracking needs to be configured to be enabled, and local transactions will be processed with higher priority).
  • eth/handler.go: Initializes the protocol's Handler instance.
  • miner/miner.go: Instantiates the transaction packaging module (originally the mining module).
  • eth/api_backend.go: Instantiates the RPC service.
  • eth/gasprice/gasprice.go: Instantiates the gas price query service.
  • internal/ethapi/api.go: Instantiates the P2P network RPC API.
  • node/node.go(RegisterAPIs): Registers RPC APIs.
  • node/node.go(RegisterProtocols): Registers p2p protocols.
  • node/node.go(RegisterLifecycle): Registers the lifecycle of various components.
• cmd/utils/flags.go(RegisterFilterAPI): Registers the Filter RPC API.
• cmd/utils/flags.go(RegisterGraphQLService): Registers the GraphQL RPC API (if configured).
• cmd/utils/flags.go(RegisterEthStatsService): Registers the EthStats RPC API (if configured).
• eth/catalyst/api.go: Registers the Engine API.

Node initialization is completed in cmd/geth/config.go in makeFullNode, focusing on initializing the following three modules:

In the first step, the Node structure in node/node.go is initialized, which is the entire node container where all functionalities need to run. The second step initializes the Ethereum structure, which includes the implementation of various core functionalities of Ethereum; Ethereum also needs to be registered in Node. The third step is to register the Engine API in Node.

The initialization of Node creates a Node instance, then initializes the p2p server, account management, and HTTP protocol ports exposed to the outside.

The initialization of Ethereum is much more complex, as most core functionalities are initialized here. First, ethdb is initialized, and the chain configuration is loaded from storage, then the consensus engine is created. The consensus engine here does not perform consensus operations but only verifies the results returned from the consensus layer. If there are withdrawal requests from the consensus layer, the actual withdrawal operations will also be completed here. Then, the Block Chain structure and transaction pool are initialized.

Once all these are completed, the handler is initialized. The handler is the entry point for all p2p network requests, including transaction synchronization, block downloading, etc., and is a key component for the decentralized operation of Ethereum. After all these are completed, some sub-protocols implemented on top of devp2p, such as eth/68 and snap, will be registered in the Node container, and finally, Ethereum will be registered as a lifecycle in the Node container, completing the initialization of Ethereum.

Finally, the initialization of the Engine API is relatively simple, only registering the Engine API in Node. At this point, the node initialization is fully completed.

Node Startup

After completing the node initialization, the node needs to be started. The startup process of the node is relatively simple, requiring only the registered RPC services and Lifecycle to be started, and then the entire node can provide services externally.

6. Conclusion

Before deeply understanding the implementation of the Ethereum execution layer, it is necessary to have an overall understanding of Ethereum. Ethereum can be viewed as a transaction-driven state machine, where the execution layer is responsible for executing transactions and changing states, while the consensus layer drives the execution layer's operation, including producing blocks, determining the order of transactions, voting for blocks, and achieving finality for blocks. Since this state machine is decentralized, it requires communication with other nodes through the p2p network to jointly maintain the consistency of state data.

The execution layer does not decide the order of transactions; it is only responsible for executing transactions and recording the state changes after transaction execution. There are two forms of recording: one is to record all state changes in the form of blocks, and the other is to record the current state in the database. At the same time, the execution layer is also the entry point for transactions, storing transactions that have not yet been packaged into blocks in the transaction pool. If other nodes need to obtain block, state, and transaction data, the execution layer will send this information out via the p2p network.

For the execution layer, there are three core modules: computation, storage, and network. Computation corresponds to the implementation of the EVM, storage corresponds to the implementation of ethdb, and the network corresponds to the implementation of devp2p. With this overall understanding, one can delve into the understanding of each sub-module without getting lost in the specific details.

Ref

[1]https://ethereum.org/zh/what-is-ethereum/

[2]https://epf.wiki/#/wiki/protocol/architecture

[3]https://clientdiversity.org/#distribution

[4]https://github.com/ethereum/devp2p

[5]https://github.com/ethereum/execution-specs

[6]https://github.com/ethereum/consensus-specs