Ethereum "Shanghai" upgrade latest progress

This article introduces the important events that have happened in Ethereum from January to now, including the operation status of the Kiln test network, the latest plan for the "Shanghai" upgrade, the executable specification work of the Ethereum execution layer, relevant information about the protocol union, and some detailed updates. Users who are closely following the latest developments of Ethereum can read it carefully.

TL;DR

A lot has happened since January, and I’ve been struggling to find the time to write about it all. Here are the highlights:

• The latest merged testnet Klin has been launched. The PoS transition on it exposed some implementation issues, and now everything is ready for merge testing.

• The next Ethereum upgrade, called “Shanghai”, is being planned. It includes EVM upgrades, beacon chain withdrawals, lower layer 2 fees, and more.

• The work on the executable specification of the Ethereum execution layer is progressing smoothly. The next step is to coordinate the upgrade process of EL+CL (execution layer + consensus layer).

• The Protocol Guild now has over 100 members and is about to launch a pilot, an initiative to provide token compensation to client development members and research members.

Kiln

Following Kintsugi, the Klin testnet was recently launched. It has changes to The Merge spec based on edge cases found on Kintsugi and some renaming. While it looks like The Merge is almost in its final form, some implementation issues have arisen in several clients during the transition to running on Klin. The team is doubling down on testing to ensure all implementations are secure and stable. Danny covered this in his recent Finalized post.

In addition to this, we are asking the wider developer community to use Klin and make sure their products work as expected. Thanks to Kurtosis, Tenderly, Lido , Uniswap , EthStaker, Infura and Blockdaemon for trying out Klin.

Klin is expected to be the last public testnet to be released, assuming no critical issues are found. Once we are satisfied with the client implementation and infrastructure/tooling readiness, the next step will be to run The Merge on existing testnets (Ropsten, Goerli, Sepolia, etc.).

As with every upgrade, we will continue to test the testnet after the upgrade to ensure the stable operation of the testnet. When we are satisfied that the testnet is running as expected, we will arrange a transition for the Ethereum mainnet.

While we are very close to this exciting moment for the entire community, a safe transition is the top priority of The Merge, even more important than running on the target date. This is the most complex upgrade Ethereum has ever undertaken. We hope to complete it smoothly.

Once the timeline for testnet and mainnet updates is determined, it will be disclosed through community publications such as Week in Ethereum, What's New in Eth2, the EF blog, etc. Any current advertised target dates are incorrect as no specific dates have been set. Please be extra vigilant about potential scams and false announcements in the coming months.

A note on the difficulty bomb

The difficulty bomb that was delayed in the Arrow Glacier upgrade last year is expected to have an impact on the block speed around June (click here to view the latest progress). Although it is ideal to transition to POS before delaying the difficulty bomb upgrade, the following three points are still worth noting:

• The impact of the difficulty bomb on block time is gradual. This means that once users feel the impact of the bomb on the network, the block speed will continue to decrease within 4-8 weeks, but it will not be too slow (probably 14-17 seconds).

• Historically, when the bomb is delayed, we choose to delay it for at least 6 months, as we usually plan to do another network upgrade at that time. That said, there is no hard and fast rule on how long to delay the bomb, and it is entirely possible to delay the bomb for a month or two, as long as doing so is more appropriate than delaying the bomb for more than 6 months.

• Once again, merging safely is more important than merging quickly. We hope the transition will be smooth, and the stability and security of Ethereum is our primary concern.

Shanghai Upgrade

As mentioned in the last update, with the specs for The Merge mostly frozen, planning work has begun around the Shanghai upgrade. For the specs for this upgrade, click here. Tentatively, three major changes are planned for this upgrade, as well as some minor refinements. Read below to learn more about this update with us!

EVM (Ethereum Virtual Machine) Object Format

Over the years, researchers and client developers have worked hard to improve the EVM without breaking current contracts. Last year, the Ipsilon team came up with a very clever idea: provide new functionality to contracts deployed with specific identifiers, but keep existing contracts as they are. This is now known as the EVM Object Format, or EOF for short.

In the London upgrade, we preserved part of that identifier by disabling deployment of new contracts starting with the 0xEF byte. Before the London upgrade, a small number of contracts were deployed that started with 0xEF, but now that this is no longer possible, we can add a second byte (which we call the "magic" byte) to the 0xEF prefix and get a sequence that we can guarantee will not be used by any contract.

EIP-3540 elaborates on this and highlights the first tangible benefit of this approach: separation of code and data, which facilitates verification of on-chain code. This also lays the foundation for introducing new types of contract code parts that can help enable current complex features, such as abstract accounts, control flow in the EVM, and EIP-3074.

As a supporting protocol for EIP-3540, EIP-3670 will enable code verification when the EOF contract is deployed.

Beacon Chain Withdrawals

Another major feature of the "Shanghai" upgrade is the activation of beacon chain withdrawals. After many proposals, the EIP-4895 protocol we designed has been able to satisfy the client team: the beacon chain can withdraw as a push operation.

The meta-spec outlines how the entire process works. At a high level, in each slot, the beacon chain can process a certain number of full or partial withdrawals. Each withdrawal is tracked by a receipt that contains the withdrawal amount, the destination address, and a unique index. These withdrawals are then credited to the execution layer as part of the block creation and validation process, similar to how proof-of-work issuance is credited to miners today. Tracking issues for the various changes required to the consensus layer can be found in the consensus specification repository. The option for partial withdrawals will allow validators to withdraw their accumulated rewards, but will still need to ensure a stake of 32 ETH to maintain validation and continue to earn rewards.

Reduction of second-tier fees

The last major update we want to complete in the "Shanghai" upgrade is to reduce the second layer fees. Because from the perspective of transaction data or proofs published by the second layer on the first layer, a large part of the transaction fees of end users comes from the amortized gas cost of storing data on the first layer. Sharding provides a cheaper option for publishing data on the second layer, however, the specification seems to have been determined, but the full sharding implementation is not yet ready.

In the meantime, there are two options for reducing these costs: lowering the cost of CALLDATA on the mainnet, or implementing “raw sharding,” perhaps by introducing a new transaction type on Ethereum called Shard Blob Transactions.

Reduce costs with CALLDATA

The simplest way to reduce transaction fees on the second layer is to reduce the storage cost of the first layer. EIP-4488 proposes to do this by reducing the cost of CALLDATA from 16 gas per byte to 3 gas per byte. The reduced storage cost of the first layer will in turn reduce the fees of the second layer [1].

While reducing gas costs is a simple change, it does have some second-order effects. First, adding CALLDATA to blocks makes them larger. To balance this, the EIP proposes to cap the amount of CALLDATA in each block. Second, even with an upper limit, this EIP will increase the rate at which historical on-chain data grows at the execution layer. To address the above, EIP-4444 proposes the need to develop out-of-band historical data retrieval and changes to the guarantees of historical data on the Ethereum peer-to-peer network.

Although the historical on-chain data is gradually increasing, running this EIP means that we need to deal with this problem more urgently after it goes live. In addition, a small part of this EIP will be reused in the full sharding implementation. It is mainly a temporary solution. That said, EIP-4488 is a relatively simple update, but it does have a significant effect in reducing the fees of the second layer.

Shard Blob Transactions

Another proposal, EIP-4844[2], introduces Shard Blob Transactions, which brings us closer to a complete sharding deployment. Like beacon chain withdrawals, this proposal has a meta-spec that links to consensus layer specifications and other resources.

At a deeper level, this new transaction type will include a commitment to a block of data to be propagated on the beacon chain. This proposal can be thought of as "mini-sharding", but instead of relying on data availability sampling, every node in the network will need to verify all the data in the blob. Just like in the case of full sharding, the data in these blobs is only guaranteed to be available on the network for a period of time, rather than being stored forever. To keep node requirements manageable, blob data is limited to 1MB/slot, compared to 16mb/slot for full sharding.

EIP-4844 will lay the foundation for full sharding. It is worth noting that all future changes are limited to the consensus layer. From the perspective of the execution layer, sharding will be up and running.

The Optimism team, which has been working on the EIP, has launched a website that provides an overview of the EIP, aggregates various specification links and highlights the community that has actively contributed to the work.

Note:

[1] Due to the other components involved in Layer 2 transaction pricing, the cost reduction will not be a full 5x. Optimism has a good explanation of the Layer 2 fee components here.

[2] EIP-4488 and EIP4844 have very similar numbers in the competing proposals, which is frustrating.

Details Improvements

In addition to the three major improvements mentioned above, some minor improvements were also considered in the "Shanghai" upgrade, namely:

• EIP-3651, which reduces the gas cost of accessing Coinbase addresses, fixes a vulnerability in EIP-2929.

• EIP-3860, which limits the size of inicode and introduces gas fee measurement for fields.

• EIP-3855 adds a new opcode, PUSH0, which, as you might expect, pushes “0” onto the EVM stack.

In addition, there are several other EIP proposals in this upgrade. EOF, beacon chain withdrawals and reducing second-layer fees have made the "Shanghai" upgrade one of the largest updates so far, so we now need to work very hard to prioritize the upgrade content.

Once we start implementing and testing the various EIPs, we will have a better idea of ​​whether we have any additional capacity. Of course, before then, we still need to complete the merge.

Ethereum Execution Layer Specification (EELS)

As you may have noticed reading above, several proposals for the Shanghai upgrade span across the execution and consensus layers. As we have learned in the past, we use different processes to introduce changes at each layer.

At the execution layer, the core EIP contains the specifications for the improvements. The Ethereum Yellow Paper is the reference specification for the network, but it is usually updated after the upgrade goes live, sometimes with significant delays. This means that for the execution layer, the Yellow Paper plus the corresponding EIP protocol is the effective specification for the execution layer.

At the consensus layer, an executable specification is used as a reference in which changes are specified directly, which can then be used to generate tests for the changes.

So, while the execution layer process is well understood by the community (and we provide easy-to-reference data descriptions), it is still not ideal from a technical perspective. Conversely, while the consensus layer process is technically more environmentally friendly, it is more difficult for the large community to follow. Fortunately, work has already begun on EELS (Ethereum Execution Layer Executable Specification).

Having executable specs at both the execution and consensus layers allows us to coordinate the process of change on both layers. While there are still many issues to be resolved, we have already started discussions on how best to accomplish the migration. The Ethereum Magician thread is now dedicated to this topic. While EELS is still under development, we may be able to use it in parallel with the current process during the Shanghai upgrade.

Hopefully the process of merging the execution and consensus layers is simpler than merging the actual execution and consensus layers?

Agreement Union

I want to close by talking about the also-important Protocol Guild, which now has a full explainer site. Compensation for protocol maintainers has become a hot topic lately, and the Protocol Guild wants to be part of the solution to this problem. Disclosure: I am a member of the Protocol Guild and receive funding from it.

You can break down compensation into three areas: base salary, aligned incentives, and potential upside. Currently, client developers and researchers receive base salaries from their respective employers. While some of this may be paid in the form of equity incentives, the Ethereum Foundation announced a client incentive program last year with 39,000 ETH, ensuring that all client teams have a significant stake in Ethereum.

The Protocol Union is different from the basic compensation and incentive plan in that it aims to provide its members with tokens of various Ethereum-based projects rather than Ethereum itself. The Protocol Union is composed of protocol engineers, researchers, and a small number of people like me who work on protocol coordination. It currently has about 100 members.

In short, the union allows sponsors to donate tokens, which are then transferred to recipients over time. The list of recipients can be updated, thus allowing new contributors to be added and previous contributors to be removed regularly.

Protocol unions are still in the early experimental stages, but if successful, they could complement grassroots-focused initiatives such as Gitcoin and Retroactive Public Goods Funding.

After successfully obtaining Gitcoin funding, the next step for the Protocol Union is to test the smart contract architecture. In parallel, outreach activities for initial donors will begin. The plan is to run the Protocol Union based on limited donations for about 1 year to ensure that both the technical and governance components are running smoothly. Hopefully, this pilot will prove that we can create new mechanisms to coordinate public goods funding on Ethereum.

Next steps

Our main priorities remain merging and refocusing on testing. Over the next month, we hope to finalize the implementation, run multiple short-lived development networks, and gather feedback from application, infrastructure, and tool providers. Everything else (Shanghai upgrade, implementation spec, protocol unions) will also continue.

We expect to have other updates in a month or two. In the meantime, we can discuss all this face to face at Devconnect, hope to see you in Amsterdam ??!

Translator’s Notes:

The Ethereum Klin test network encountered some problems during operation. The relevant team is currently solving them. When the team feels that everything is ready, it will be merged into the existing test network. The project party attaches great importance to the security of the Ethereum merger, so the specific merger time cannot be determined; the difficulty bomb will be carried out at an appropriate time, all of which is based on the premise of a safe merger process. The "Shanghai" upgrade update mainly includes the EVM (Ethereum Virtual Machine) object format, beacon chain withdrawals and the reduction of second-layer fees. This update is mainly aimed at improving user experience, enhancing the functions of smart contracts, allowing users to withdraw part or all of the beacon chain, and proposing two ways to reduce the transaction fees of the second layer. The protocol union has begun a pilot, and everything is going well. The main purpose is to provide compensation for protocol maintainers.

Original author: AllCoreDevs Updates

Original translation: H. Forest Ventures, 0x123

Original title: AllCoreDevs Update 010