Technical analysis of the OpenEthereum client error after the Berlin upgrade

Original title: "Popular Science | The whole story of the error after the OpenEthereum client "Berlin" upgrade"
By Alex Stokes
Translation: Ajian

As you may have heard, a bug in the @OpenEthereum client caused downtime for some important services supporting the Ethereum network.

Let's think about the transaction that caused the accident.

First of all, I would like to thank all the engineers who responded quickly to the incident and resolved the problem.

Also, I didn't keep track of all the details myself, the important facts below were brought up by user eb in the Eth R&D; discord server.

Let’s start with the transaction that triggered the error.

This is a contract call transaction, sent from KuCoin exchange, distributing ETH to many addresses. The ABI encoding of the call data of this transaction was wrong, which eventually led to the chain split. You can see the "Input Data" of this transaction on Etherscan.

1/ When calling sendEths in a contract, two parameters need to be provided: one is a dynamically sized array of the target address; the other is a dynamically sized array of the transfer amount (in wei). The two are combined to know how much money to transfer to which address.

2/ We can parse the call data to see what went wrong: the first line (marked as "[0]" on Etherscan) indicates that the address list starts at byte 64 (line "[2]"). The second line indicates that the list of transfer amounts starts at byte 416 (line "[13]").

3/ So, in general, we want to send a certain amount of ETH to a certain address in pairs, from top to bottom - it seems straightforward.

4/ However, when we start walking this list, we jump to the correct byte of call data first, and the Solidity ABI states that the first word of data is the length of the entire indefinite-length array.

5/ This is the source of the final bug: because the value in call dada is "0x10" (note, this is hexadecimal!), but the call data only gives 10 address-value pairs. The correct ABI encoding for this call data (filled in lines [2] and [13]) should be "0xA" - not "0x10"!

6/ You may have guessed what happens at that time, we can see it through the execution trace.

7/ The contract successfully traverses the first 10 addresses. The contract should stop executing at this point, but according to the call data statement, there are still many addresses! So let's continue executing.

However, according to the structure of call data, the "11th address" is 0x10, which is used to encode the length of the list, so the contract tries to send 0 ETH to address 0x10.

8/ Additionally, it seems that when the contract tries to read call data that doesn't exist, 0 ETH is returned - you can imagine that the contract ran into an error here, but it continued to send 0 ETH to the other 6 "addresses" it read from the call data.

At this point, you might notice that 0x10 is likely one of our so-called “special addresses”, which is well within the scope of EVM precompiled contracts (the so-called “precompiled contracts” are a special type of contract that has an optimal implementation outside of the EVM, but compiles like most contracts).

And we don’t expect the precompiled contract 0x10 to return ETH. In this way, it becomes an ETH black hole. However, this does not necessarily cause any problems. What exactly caused the entire client to crash?

The reason is that 0x10 is actually a precompiled contract asserted by EIP-2537, which is designed for BLS pairing cryptography, but this EIP has not yet been deployed to the mainnet. So although you can interact with this address, there is no contract at this address on the mainnet, and no further actions will be taken.

In addition, we need another fact to explain this split, which you may have guessed is the "Berlin" hard fork (which was also the hard fork that brought this issue to the surface): it changed the way gas consumption is measured in the EVM.

After EIP-2929 is implemented, if you perform state storage operations on the same storage slot multiple times in a transaction, the first execution will consume more Gas, and subsequent executions will consume less. This repricing can theoretically more accurately reflect the current cost of client access to storage items...

Also, remember that in all client-side executions, this data is often stored on cheaper hardware tiers.

Now we have finally found the bug that occurred in OpenEthereum at block #12244294: the client included all implemented precompiles as part of the EIP-2929 access list. (Translator's note: This should be "EIP-2930")

Because EIP-2537 is already implemented in most clients (and was proposed to be included in the Berlin upgrade!), OpenEthereum gives a gas discount to all transactions that access 0x10.

But the vast majority of active clients on the network do not implement EIP-2929 this way, they only provide gas discounts for transactions that access activated precompiled contracts - and EIP-2537 belongs to precompiled contracts that have not yet been activated! Therefore, the OpenEthereum client's calculation of how much Gas the transaction consumed disagrees with other clients on the network.

Fortunately, @mhswende quickly found the bug, and @sorpaas worked on fixing it.

There is much more to say, and I anticipate that someone with a better view of the whole picture than I will write a better report.

All I can say is that this bug highlights the inherent risks of hard forks, and the importance of continuing to work on building more resilient infrastructure.

Single-client systems that rely on the OpenEthereum client were down for a while today as the client was unable to keep up with the network after a problematic block appeared. Etherscan itself was also down as a result.

Fortunately, this bug was not severe enough to cause a major chain fork, but it is possible. We can improve resistance by using multi-client implementations - multi-client itself is a major strength of our Ethereum ecosystem - and push your infrastructure providers to do the same.

We have seen that the speed of adoption in 2021 is unprecedentedly fast, and the prospects are very bright. We must learn lessons from this incident and build a better Ethereum together.