Sharding Technology: Another Possibility for Blockchain Scalability

On September 15, 2022, Ethereum completed the Merge(. This is a historic moment that Ethereum prepared for over 5 years and delayed 6 times. Due to long-term development and widespread attention, many people mistakenly believed that the Merge would naturally bring higher scalability, security, and sustainability, but that is not the case. The transition from Proof of Work)PoW( to Proof of Stake)PoS( is merely a "change of tracks and wheels", and does not directly lead to faster speeds, higher capacity, or lower fees. What can truly achieve these goals is a complete set of solutions: a mainnet with Sharding capability paired with enhanced scalability Layer 2 solutions.

As pointed out by Ethereum founder Vitalik Buterin, Sharding is a scalability solution to the scalability trilemma. It works by dividing the nodes in the network into smaller groups to handle different sets of transactions and achieve parallel processing. This is similar to opening multiple checkout lanes at Walmart to intuitively reduce waiting time and improve checkout efficiency.

This is the logic of Sharding, direct and simple, yet the devil is in the details. The principles and direction are correct, but many problems always arise during implementation. This article aims to clarify the direction and dilemmas on the "Sharding" path, sketching a map for the Sharding explorers who gaze at the stars and keep their feet on the ground. At the same time, by comparing existing Sharding solutions, we find some common issues and propose a feasible exploration direction: Shardeum and dynamic Sharding.

![A Comprehensive Explanation of the New Public Chain Shardeum: Another Possibility of Sharding])https://img-cdn.gateio.im/webp-social/moments-69c7de2bfe4ae7b233bec1f706fad9ad.webp(

1. About "Sharding"

In simple terms, considering the constraints of the impossible triangle, starting from Ethereum as the origin of the coordinate system )0,0(, we can categorize the current scalability methods of the Blockchain into two major types based on "vertical" and "horizontal" approaches:

Vertical Scaling ): Achieved by improving the performance of the existing hardware in the system. Establish a decentralized network where each node in the network has supercomputing power, meaning each node requires "better" hardware. This method is simple and effective, allowing for preliminary improvements in throughput, especially suitable for high-frequency trading, gaming, and other latency-sensitive application scenarios. However, this scaling method limits the level of decentralization of the network, as the cost of running validating nodes or full nodes increases. Maintaining the level of decentralization is limited by the approximate growth rate of computing hardware performance (, which is known as "Moore's Law": the number of transistors on a chip doubles every two years, and the cost of computation halves ).

Horizontal Scaling(: Horizontal scaling generally has several approaches. One approach is to distribute the transaction computation within a certain ecosystem across multiple independent blockchains in the context of blockchain, where each chain has its own block producer and execution capability. This method allows for full customization of each chain's execution layer, such as node hardware requirements, privacy features, gas fees, virtual machines, and permission settings, etc. Another horizontal scaling solution is modular blockchain, which divides the blockchain's infrastructure into execution layer, data availability layer)DA(, and consensus layer. The most mainstream modular mechanism in blockchain is rollup. Another approach is to split a blockchain into many shards and execute in parallel. Each shard can be seen as a blockchain, meaning that many blockchains can execute in parallel. Additionally, there is usually a main chain whose sole task is to keep all shards in sync.

![A Comprehensive Explanation of the New Public Chain Shardeum: Another Possibility of Sharding])https://img-cdn.gateio.im/webp-social/moments-7aa1677db6b8128b68accfe04fcda738.webp(

It should be noted that the above scalability ideas do not exist in isolation; each solution finds a trade-off point within the impossible triangle, designed in conjunction with the incentive mechanisms created by the economic forces within the system to achieve an effective balance on both macro and micro levels.

To discuss "Sharding", we need to start from scratch.

Assuming such a scenario, during checkout at Walmart, in order to improve checkout efficiency and reduce customer waiting time, we expand from a single checkout lane to 10 checkout windows. To avoid ledger errors, we need to establish unified rules at this time:

First, if we have 10 cashiers, how should we assign them to work at which counter?

Second, if we have 1000 customers lined up waiting, how should we decide which window each customer goes to for checkout?

Third, how should the 10 individual ledgers corresponding to these 10 windows be summarized?

Fourth, how can we prevent cashiers from making mistakes in order to avoid discrepancies in accounts?

These questions actually correspond to several key issues in Sharding, which are:

How to determine which shard the nodes/validators on the entire network belong to? That is: how to perform network sharding )Network Sharding(;

How to determine which shard each transaction is allocated to? That is: how to perform transaction sharding )Transaction Sharding(;

How is Blockchain data stored across different Shardings? That is: how to perform state sharding ) State Sharding (;

Complexity implies risk. Based on all of the above, how can we avoid the fragmentation of the entire system's security?

) 01 网络分片(Network Sharding)

If we simplify the understanding of blockchain as a decentralized ledger, whether it is a PoS or PoW consensus mechanism, it is all about allowing each node to compete for the right to keep accounts according to certain established rules, while ensuring the correctness of the ledger in the process. Network Sharding refers to the need for another established rule to shard the blockchain network, allowing each shard to handle on-chain transactions and compete for the right to keep accounts, with the premise of minimizing communication between them - that is, the grouping rules of the nodes.

The problem encountered in this process is that as the internal nodes of the blockchain are divided into different shards, the difficulty and cost for attackers will decrease linearly. We can infer that if the rules and results of this grouping process are fixed and predictable, then an attacker who wants to control the entire blockchain network only needs to selectively control one shard and buy off some of the nodes within that shard.

The founder of Near, Alexander Skidanov, described this issue as follows: If a single chain with X validators decides to hard fork into a sharded chain and splits the X validators into 10 shards, each shard now has only X/10 validators. To compromise a shard, it only requires compromising 5.1% of the total number of validators ###51% / 10(. This leads to the second point: Who selects the validators for each shard? It is only harmful to control 5.1% of the validators when all of those 5.1% are in the same shard. If validators cannot choose which shard to validate in, it is highly unlikely that participants controlling 5.1% of the validators will place all validators in the same shard, thus greatly reducing their ability to compromise the system.

The sharding system must develop a mechanism to trust that the network will not reverse these transactions from external shards. So far, the best answer may be to ensure that the number of validators within a shard is above a certain minimum threshold, so that the likelihood of dishonest validators overwhelming a single shard is very low. The most common approach is to build a certain degree of unbiased randomness, relying on mathematical methods to minimize the attackers' chances of success. For example, Ethereum's solution is to randomly select validators for a shard from all validators, and every 6.4 minutes, the length of an epoch is ), the validators are changed.

Simply put, it means randomly grouping the nodes and then assigning work for independent verification to each group of nodes.

However, it is important to note that the randomness in the Blockchain is a very challenging topic. Logically, the process of generating this random number should not depend on the computation of any specific Sharding. For this computation, many existing design ideas involve developing a separate Block to maintain the entire network. Such chains are referred to as Beacon chains in Ethereum and Near, Relay chains in PolkaDot, and Cosmos Hub in Cosmos.

( 02 Transaction Sharding )

Transaction sharding refers to the formulation of rules regarding "which transactions are to be allocated to which shards," which can achieve the goal of parallel processing while avoiding the occurrence of double spending issues. The differences in the ledger model of the Blockchain will affect the development of transaction sharding.

Currently, there are two types of accounting methods in blockchain networks, namely UTXO( Unspent Transaction Outputs model and account/balance model. The typical representative of the former is BTC, while the latter includes ETH.

UTXO Model: In BTC transactions, each transaction will have one or more outputs. UTXO refers to the unspent outputs of blockchain transactions, which can be used as inputs for new transactions, while spent transaction outputs cannot be reused. This is similar to the case of paper currency transactions where a customer pays with one or more banknotes, and the merchant gives change with one or more banknotes. Under the UTXO model, transaction sharding requires cross-shard communication. A transaction may include multiple inputs and multiple outputs, with no concept of accounts and no record of balances. One possible method is to process it through a hash function according to a certain input value of its transaction to determine which shard the data should go to. As follows:

In order to ensure that entries are placed in the correct Sharding consistently, the values input to the hash function should come from the same column. This column is called the Shard Key. Then, transactions that produce a value of 1 will be allocated to Shard 1, and transactions that produce a value of 2 will be allocated to Shard 2. The downside of this approach is that communication between shards must occur to avoid double-spending attacks. Limiting cross-shard transactions will restrict the platform's usability, while allowing cross-shard transactions requires weighing the costs of cross-shard communication against the performance gains.

Account/Balances Model: The system records the balance of each account. When a transaction is made, the system checks whether the account has sufficient balance for payment, similar to how banks record the balance of each account during a bank transfer. A transaction can only proceed if the account balance is greater than the required transfer amount. Under the account/balances model, since a transaction has only one input, as long as transactions are sharded by the sender's address, it can ensure that multiple transactions from the same account are processed within the same shard, effectively preventing double spending. Therefore, most blockchains that adopt sharding technology are account ledger systems like Ethereum.

![Detailed Explanation of the New Public Chain Shardeum: Another Possibility of Sharding])https://img-cdn.gateio.im/webp-social/moments-4227a2e49f76cd01b23d7b5398e51a3c.webp###

( 03 状态Sharding)State Sharding(

Status sharding refers to how data in the blockchain is allocated and stored in different shards.

Using our Walmart queue example, each window has a ledger. How is their ledger recorded? If a customer queues at a certain line, then that ledger is recorded. For example, if Customer A goes to Window A, and the next day that customer goes to another checkout window, say Window B, and Window B does not have that customer's past account information ), such as involving a stored value card or other payment methods (, what should be done? Should we call Customer A's account information from Window A?

State sharding is the biggest challenge of sharding, and it is more complicated than the aforementioned network sharding and transaction sharding. Because under the sharding mechanism, transactions are processed in different shards based on address allocation, which means that the state will only be stored in the shard where its address is located. At this time, one issue to face is that transactions do not occur solely within one shard, and often involve cross-sharding ) Cross-Sharding ###.

Consider a transfer scenario where Account A transfers 10U to Account B, and A's address is allocated in Sharding 1, the transaction record will also be stored in Sharding 1. B's address is allocated in Sharding 2, and the transaction record will be stored in Sharding 2.

Once A wants to transfer funds to B, a cross-sharding transaction will be formed. Sharding 2 will call the past transaction records from Sharding 1 to confirm the validity of the transaction. If A frequently sends coins to B, Sharding 2 must continuously interact with Sharding 1, which will lower the efficiency of transaction processing. However, if participants do not download and verify the entire history of a specific sharding, they are not necessarily required to.