Opinion | Five questions about Ethereum: Where does the client diversity problem come from? How to solve it?

The recent 90th core developer meeting was almost entirely devoted to one question. I highly recommend that you listen to the meeting in person.

In this meeting, Alexey raised the issue of client developer overload. Although I think this discussion is an important start, we are too eager to find a solution. It is urgent to fully understand the problem. It is important to take some time to analyze the problem. The "Five Whys" method is one of the simplest and most effective methods to analyze the connotation of the problem.

Without further ado, let’s take a look at what the first question is.

Question 1: Why is the pressure on the Geth development team so great that it is even overloaded?

Through etherscan, we can see the statistics of the installed capacity of each client as follows:

• Geth: 75%

• Parity & OpenEthereum: 20%

• Nethermind: 1%

The remaining 4% consists of some clients with less than 1% market share and are therefore ignored.

Importantly, more than 51% of the hashrate is concentrated on the Geth client. Suppose in the upcoming Berlin hard fork, Geth has a bug in the implementation of one of the EIPs (Ethereum Improvement Proposals). Even if there are no other bugs in the client, as long as a block encounters this bug, it will cause the Ethereum network to fork. In theory, this block is invalid and other clients will also regard it as an invalid block. However, more than 51% of the mining nodes are running the Geth client, so the entire network will be led to the wrong fork chain.

This requires that the Geth client and development team must not make any mistakes.

Therefore, the answer to the first question is:

Because the Ethereum network lacks sufficient diversity in clients.

It is worth mentioning that client diversity will not suddenly turn client development into an easy task. But client diversity itself is still an area worth exploring, which will help us find ways to improve the efficiency of client development while reducing the burden on development teams. It is undeniable that only working on the Geth team is unlikely to solve this problem.

Question 2: Why does the Ethereum network lack client diversity?

When the Ethereum mainnet was launched, we had multiple clients. The two most important ones were Geth and CPPEthereum. Later, Parity appeared and CPPEthereum was eliminated.

Since then, no client has gained a significant market share except Parity. Nethermind emerged as a rising star last year, but currently only has 1% of the market share. Recently, Parity's market share has dropped significantly due to some setbacks and bleak future. We believe that in an ideal situation, the Ethereum network needs to have 3 or more clients, each client's share of the client is not too high, and no one client can occupy much more than 51% of the market share. Although ideally, we should have diversified clients, but we have become accustomed to the situation of client hegemony.

So, why do we need multiple clients?

From my personal experience, building an Ethereum client is extremely difficult. The reason why Geth can run stably on the Ethereum network is that it introduces many complex optimizations. The Geth team spent several years to achieve such a high level of complexity, and is still continuing to optimize.

One might immediately suggest that we find ways to support and help clients that are lagging behind. I'm wary of this "mythical man-month" solution - throwing more engineers at a hard problem rarely works in software development, and I don't expect it to.

Instead, I think the focus should be on complexity.

Note: The mythical man-month states that using a large number of people in a short time does not shorten the software development process. A herd-like approach does not help software production and will create trouble and produce worse software. Adding manpower to a project that is behind schedule will only make the progress further behind schedule.

Question 3: Why is it so difficult to build an Ethereum client?

Now, we are getting closer to the root of the problem.

It turns out that most of the difficulty comes from the networking protocol, which is the set of tools that Ethereum client software uses to connect to each other and share blockchain information. Ethereum's networking rules (defined in the devp2p codebase) ultimately influence and even determine the design and requirements of Ethereum clients.

Some networking tools specify non-optimized architectures and even require Ethereum clients to run unnecessary functions. Client developers need to work within these limitations.

Question 4: Why does the Internet protocol increase the difficulty of client implementation?

I believe the answer to this question can be broken down into basically two parts.

• State Management

• Overall network interconnection requirements

In terms of state management, Ethereum clients must be able to sync the full state on the network, and maintain a local copy of that state. Both of these are difficult to do. Synchronizing state requires millions of requests and saturates disk I/O, both for the client and for the server that reads and processes state requests. The newly synced state needs to be maintained and pruned so that the database can execute new blocks quickly enough. This is a serious engineering challenge!

GetNodeData is the only networking tool we use to sync state, optimized for a specific state database format, commonly known as "the native layout". The "flat" database layout popularized by Turbo Geth has great performance advantages for state maintenance, but using this layout makes GetNodeData requests more difficult to process.

Once we turn our attention to the network technology, and in particular the DevP2P ETH protocol, we see that there are other factors that increase the complexity of the client. To join this network, the client needs to have the following capabilities:

• Handles GetNodeData requests to access the state generated by the most recent block;

• Processes requests to access the on-chain data history, including block headers, block bodies, and receipts.

Fundamentally, for many client operations, the underlying data required to process these requests is not necessary, but they are now forced to support these features. This requires all clients to build a lot of functionality in addition to meeting their own needs. For example, a client that mainly acts as a transaction sending gateway does not need historical on-chain data, and may only need a small subset of the state. However, with the current version of Ethereum, clients still need to save a full copy.

Question 5: Why...

It seems that I found the root cause by asking only four "whys". The Ethereum protocol is not yet fully mature. When designing the Ethereum protocol, we were not aware of most of the problems found today, or because the state size was smaller and the development history was shorter, these problems were not yet a problem.

Solution

I’ve been focusing on this issue for the past year, and it’s been striking to me that many of the problems on Ethereum can be traced back to the networking layer.

Perhaps the most obvious example is that disk I/O has historically been a bottleneck for clients. This bottleneck exists because clients tend to use a naive representation of a tree structure (Trie) to implement their state database. The state database is constructed in a way that is determined by the GetNodeData network element.

To solve this problem, we need to overhaul various parts of the Ethereum consensus layer and the networking layer. Currently, the overhaul has begun. A lot of work has been carried out under the name of "Stateless Ethereum", which Alexey and I have co-led for 8 months. Some of the work we have done at least reduces the burden on the Geth team, because they use their own SNAP synchronization protocol that has been developed for many years. Some of the work needs to be undertaken by talented people who deeply understand the problem and have a feasible solution.

Currently, such a large DevP2P ETH protocol has not been fully deconstructed. We have a basic understanding of how to split this network into three independent dedicated networks, but no one has yet directly started working on this.

Alternatively, there are ideas like re-genesis that offer mechanisms to sidestep these problems entirely. This is a radical approach that, if successful, could give us a huge advantage.

First of all, we should be clear that the Ethereum network still has many difficult tasks to accomplish, and only a few people are capable of these tasks. Although more and more developers are participating every day, they need to invest time and energy to learn the necessary skills. Client developers must also take time to develop new EVM features while focusing on solving low-level problems that everyday users cannot see.

If we want the Ethereum network to be successful in the long term, I think the entire community needs to work together to address these issues, give enough attention and discussion to their root causes, and most importantly, work together to create effective technical solutions.

(over)

Original link: https://snakecharmers.ethereum.org/applying-the-five-whys-to-the-client-diversity-problem/ Author: Piper Merriam Translation & Proofreading: Min Min & A Jian

Link to this article: https://www.8btc.com/article/623686
Please indicate the source of the article when reprinting