A quick overview of Ethereum’s latest developments: mergers, Shanghai upgrade, and data sharding

Written by: Trent Van Epps, Ethereum Foundation

Editor: South Wind

Ethereum is a protocol that is undergoing significant change. In the medium term, we are driving upgrades to the Ethereum protocol that will allow it to scale and meet growing global demand, while also improving the security and decentralization of the Ethereum network. It’s a long and winding road, but researchers and developers in the Ethereum “Agency” are more active than ever.

Before reading on, please keep in mind that this is not an “official” roadmap, but only a limited and subjective view of the current state of Ethereum.

01. The naming has changed

Timeline → In Progress

Let's first talk about the naming we use in the Ethereum space. While it may seem a bit strange to start with this aspect, it is important to understand that Ethereum's naming framework is determined by its development roadmap. Below are two examples of terminology changes that have become popular recently, and the reasons behind such naming changes.

Execution & Consensus

For all intents and purposes, the terms “Eth1” and “Eth2” are no longer used in Ethereum core development, see the Great Renaming document by Tim Beiko:

https://notes.ethereum.org/@timbeiko/great-renaming

The two old naming methods convey two meanings, namely "Eth1 comes first, Eth2 comes later" and "Eth1 ceases to exist after Eth2 is born". Ethereum core researcher Danny Ryan has pointed out this problem since October 2020. Although the PoS beacon chain has been running in parallel with the current PoW Ethereum mainnet since its launch in early December 2020, using the terms Eth1 and Eth2 would imply that the current PoW Ethereum has disappeared to some extent. In fact, the chain state on the current Ethereum mainnet will be seamlessly combined with the beacon chain after the merge occurs.

No data will be lost and no migration is required.

We have moved to using “Execution” and “Consensus” instead of the terms “Eth1” and “Eth2”. I recommend reading this in-depth article by Dany Ryan:

https://blog.ethereum.org/2021/01/20/the-state-of-eth2-january-2021/

In short, "execution" refers to everything at the user level, including applications, account balances, tokens, etc., which can also be called "state." "Consensus" refers to the PoS consensus mechanism that ties all these things together, including the finality of transactions, fork selection rules, validators, and incentive mechanisms.

In the merged environment, the execution layer and the consensus layer will coexist together.

“Features”, not “phases”

Another term we are moving away from is “Phases.” In the past, this referred to a specific protocol change, such as “Phase 0 is the beacon chain.”

Late last year, we began informally and incrementally refactoring phases to features. First, using specific features is more flexible: when the protocol design is updated or the scope is expanded or reduced, it is more difficult to communicate these changes using the shorthand "Phase X" than using the name of the feature proposal.

Second, the word “phase” implies sequentiality: “Phase X” must be followed by “Phase X+1.” A good example of how this naming doesn’t work is when the currently ongoing “merge” (formerly known as “Phase 1.5”) takes precedence over “sharding” (formerly known as “Phase 1”).

The preference for using specific “feature” names means that these names can be easily reordered when needed and more clearly communicate their ultimate impact.

Cognitive flexibility

More abstractly, I tend to think that both of the above changes are based on cognitive flexibility. In other words, this is a flexible form of knowledge formation that allows our community to better understand the unstructured creativity that emerges in the Ethereum "market".

We are able to make adaptive decisions based on our roadmap because we are not tied to phases or sequences. There is a level of humility in admitting and internalizing that we don’t have a complete roadmap. This is an important part of Ethereum’s philosophy, and I’m glad to see it take root.

It may take a while for the wider Ethereum community to adjust to the new naming convention, but we will! Thanks in advance for helping us with this important endeavor :)

02. Merge

Timeline → 5-8 months

Now that we’ve discussed the naming change, let’s discuss the next exciting Ethereum feature: the merge. The merge refers to the upcoming shift in Ethereum’s consensus mechanism from Proof of Work (PoW) to Proof of Stake (PoS).

This is one of the most anticipated protocol changes for Ethereum and the wider crypto space. For most of the blockchain industry’s existence, negative perceptions about PoW and its energy consumption have dominated the popular media. Ethereum will be the largest protocol to ever shift its consensus mechanism and is expected to change that narrative.

Benefits of the merger to the agreement

The merge includes many significant improvements to the Ethereum protocol:

When the merger is complete, the Ethereum blockchain will become more secure. Blocks will be "finalized" after a certain point in time, and a penalty mechanism will be introduced for validators who reorganize the blockchain, which will inhibit validators from reorganizing blocks or internal transactions.

Secondly, PoS eliminates a lot of the energy consumption and hardware waste associated with PoW. Researchers expect Ethereum's energy consumption to drop by up to 99.95% after the merger. Regular consumer-grade hardware will replace the ASICs and GPUs hardware that currently run Ethereum consensus. These two major impacts will bring Ethereum more energy-efficient, more diverse, more geographically dispersed, and more resistant consensus participants.

Third, Ethereum PoS will build a platform for sharding, which is also a major protocol change that will split the Ethereum blockchain into many shard chains running in parallel. Sharding will bring additional benefits to L2 expansion efforts by increasing available block space for data availability and settlement.

Finally, the merger will reduce ETH’s annual issuance rate from the current net 3.5% to approximately net 0%.

The road to merger

The first major event to lay the groundwork for the “merge” was the month-long Rayonism hackathon earlier this year. This work simulated what the Ethereum blockchain would look like after the merger, and how consensus/execution clients would communicate with each other.

Recently, we held the Amphora Interoperability Workshop, a week-long event that continued to build on the success of Rayonism and added the key moments of the transition from PoW to PoS. 10 client teams contributed, and by the end of the week, a developer testnet modeling this transition from PoW to PoS was successfully running!

If you want to learn more about this event, you can check out Tim Beiko’s article Amphora: A Major Merge Milestone:

https://blog.ethereum.org/2021/10/15/amphora-merge-milestone/

The experience gained from this event has been incorporated into the latest version of the merged specification, "Kintsugi". In parallel, there is a long-term development testnet called Pithos. This will be restarted multiple times between Q4 2021 and Q1 2022 to retest the transition from PoW to PoS with updated specifications. Once this transition becomes reasonably stable, existing testnets such as Goerli can be upgraded to match the merged specification.

Interested community members can follow this “The Merge Mainnet Readiness Checklist”, which provides a comprehensive overview of the remaining unfinished items:

https://github.com/ethereum/pm/blob/master/Merge/mainnet-readiness.md

03. Shanghai Upgrade

Timeline → 10-12 months

One interesting thing about the merged Ethereum is that, although the old and new chains have been merged, the clients are still independent of each other: how they are architected, and the teams working on them are also independent. For validators, this means a lot of optionality: every execution client can be combined with every consensus client, and vice versa. For fun, I listed the new names that these client combinations might use:

Twitter source: https://twitter.com/trent_vanepps/status/1445008969756467203?s=20

Independent execution and consensus layers also allow for a decoupled upgrade process if necessary. This fits in well with Ethereum’s philosophy of “separation of concerns”. In other words, smaller changes will be easier to manage than in a monolithic blockchain.

However, the Shanghai upgrade will upgrade both the consensus layer and the execution layer in a coupled manner, making withdrawals possible for validators, that is, allowing validators to withdraw their ETH from the consensus layer to the execution layer, thus binding the two more closely together. When ETH is withdrawn from the beacon chain (consensus layer) to the execution layer, ETH can be used just like we do on the current Ethereum chain: ETH can be used as a store of value, or used to pay for NFTs, or to pay transaction fees, etc. There are currently some other proposals for the execution layer that are being considered for inclusion in the Shanghai upgrade, but these proposals have not yet been formally accepted.

We won’t know what this acceptable range is until the merger actually goes live.

04. Ethereum Research

Timeline → In Progress

While the above work is being specified, implemented, and tested, there are other parallel research efforts moving Ethereum forward.

Data Sharding

Sharding may be the most important change that will happen to Ethereum when it switches to PoS. It is important to note that the current proposals focus on data sharding, not shard execution. Data sharding will provide L2s networks with more block space available for storing data, but these shards do not yet support local user transaction execution functions like we are familiar with on the current Ethereum mainnet. Currently, the Rollups network uses the Ethereum mainnet for such settlement operations. The basic research for this type of sharding is not complicated, which means that data sharding will be able to go online on the mainnet faster and bring benefits to the L2s network!

Above: Original diagram from Hsiao-wei Wang, designed by Quantstamp

This prioritization of data availability is in line with the dynamic progress in scalability research and dApps over the past 18 months. Vitalik’s October 2020 post “Rollup-centric Ethereum Roadmap” does a great job of articulating this possible future. This is a great example of the flexibility of the Ethereum community’s perception!

At some point in the future, the Ethereum community may decide to add sharding capabilities. But this remains an open research question.

State Expiration & Weak Statelessness

Research in this area will reform the way the Ethereum protocol handles state. State refers to all user records, including contracts, tokens, NFTs, and addresses. In the current Ethereum, users incur a one-time cost for each transaction, which causes the transaction record to be stored indefinitely in the Ethereum state. In the long run, this is not sustainable.

In the past few years, several proposals with different trade-offs have been explored, including state rent and ReGenesis.

One of the main proposals is called "Weak Statelessness", which will change the way Ethereum nodes store and handle state: specifically, only block proposers will be required to store state, while all other nodes can verify blocks without storing state. Here are the impacts of this approach on different participants:

User: can discard the state, but also need to submit a "witness" when submitting the transaction. A witness is a proof sent along with the transaction to verify the validity of the transaction;

Non-validator nodes: can discard the state;

Validators/block proposers: can discard state if they rely on a third party to produce blocks;

Block Producers: Still need to store all states. Responsible for using witnesses submitted by users to produce blocks containing valid state changes.

Another proposal is called "State Expiry". According to this proposal, if a state has not been accessed for a specified period of time, then the state can become inactive, or change from active to "expired". This could be ETH in cold storage, or ERC20s tokens abandoned after a community left, etc. If a user wants to reactivate their state, any transaction sent by the user will need to be accompanied by a witness (weitness). A big benefit of limiting the size of Ethereum's active state is that it will become easier for nodes to sync the network and continue maintenance.

Both concepts are being actively researched, benchmarked, and implemented via proof of concept (PoC). For a more in-depth look at the current progress, see:

Summary with links by Guillame Ballet:

https://notes.ethereum.org/@gballet/Sy-a6T5St

These links on the EthR&R Discord Guidebook:

https://github.com/tvanepps/EthereumDiscordGuidebook#statelessness--state-expiry

… and more

There is much more I could write about, such as EVM (Ethereum Virtual Machine) improvements and ways to enhance consensus, but I will save that for the next article.

If you are interested in studying these important problems, please send me a direct message on Twitter (@trent_vanepps) to invite you to join the Ethereum R&D Discord chat channel. Or you can go to the EthResearch forum for in-depth research:

https://ethresear.ch/top