The truth behind the Infura outage: a hard fork whose details will not be released yet

Note: Any consensus failure is a serious problem, and an unannounced hard fork shows that these vulnerabilities can be very harmful to Ethereum if not controlled. Fortunately, thanks to the keen eyes of bug bounty hunters and the hard work of Ethereum developers, this incident did not cause any damage. The original text comes from rekt, the author is anonymous.

Consensus is not just about reaching agreement, it’s also about changing things around.

The Ethereum network experienced some issues today due to a consensus bug that hit the mainnet.

Service providers running old versions of geth nodes found themselves stuck on a minority-participated fork of the blockchain, which subsequently had a chain reaction on all applications that depended on them.

Most users became aware of the problem after the Infura API went down. Infura is the largest node provider on the Ethereum network, providing tools and infrastructure for some of the most commonly used Web3.0 applications (such as Metamask and Uniswap) and centralized exchanges (such as Binance).

The degradation of Infura’s service revealed that Binance was either dependent on Infura or running an outdated node, which is either unsuitable for such a large exchange.

While it may seem strange that Infura is not running the latest geth client version, it actually makes some sense for such a large-scale operator not to directly adopt the latest client version, because without a hard fork, there is no urgent reason to switch from a stable code infrastructure to an unproven code infrastructure.

Search and analytics engine Blockchair also experienced issues, with its operator tweeting the following:

“We are also experiencing issues with our Ethereum explorer, which we are currently working on fixing. It appears that a chain split has occurred, with some nodes (including ours and some miners’ nodes) getting stuck on a minority fork of the chain.”

Blockchair lead developer Nikita Zhavonronkov reported the following error:

 ########## BAD BLOCK #########
Error: invalid merkle root (remote: 57cc91ee8b91b956592a27b14386abc2aba723b5f4f9e5d3181ace6b5d3cd433 local: 1f9ee59bfa683a25c7a15b626995a3ad7c58c571b40df96eea31e5c5eed9732d)

Two serious vulnerabilities were discovered in the geth network, both discovered by John Youngseok Yang (Software Platform Labs), which earned him 20,000 Ethereum Bounty Program points.

To prevent the network from being attacked, developers have not discussed these consensus errors on Github, so the specific details of the vulnerability are not yet known . For those eager to know more technical details, Mhswende said:

"There may be a summary of these vulnerabilities in the future, or a devcon talk."

To minimize disruption, Ethereum developers decided to hard fork.

As Péter Szilagyi wrote on Twitter:

“This was an ‘unannounced hard fork’ (from a bad blockchain to a good one), meaning that quietly fixing a bug that had been dormant for more than 2 years would reduce the chances of causing chaos rather than announcing it in a high-profile manner. We are trying to minimize the potential damage.”

This incident should serve as a timely reminder to anyone who has complained about Infura’s service outages, because once you delegate your node to another party, they get to decide how you conduct business.

Infura has been very transparent about the whole incident and it is clear that they are working hard to resolve the issue. As of now, Infura has been restored to normalcy and status updates can be found here.

The Infura outage made many of us realize how much we rely on such a single entity.

This is a centralized service that acts as a gatekeeper to our decentralized system.

Both individuals and institutions need to consider their future practices.

The perceived power of large, trusted exchanges like Binance and Bithumb has fallen as they were forced to disable ETH and ERC-20 withdrawals during the Infura outage, despite their responsibility to their users not to be impacted by such incidents.

We cannot rely so much on Infura. Because MetaMask uses this centralized node provider by default, the entire Ethereum network temporarily became desolate and the gas price once dropped to 12 gwei.

This demonstrates an unhealthy dependence and clearly shows the potential harm that such dependence can cause.

Do we want our digital society to replicate the same mistakes that have occurred elsewhere, relying on centralized single points of failure?

We have built a free internet, but handed it over to a small group of centralized entities — Chrome, Safari, Brave. We have built an alternative internet, where the original values ​​of anonymity and decentralization still exist, but we call it the dark web, and it can only be accessed with alternative, often blocked software.

We cannot let the same thing happen to cryptocurrencies.

David Mihal wrote:

“Today’s Infura outage has left users scrambling to find another RPC provider. I’ve listed the public, free RPC endpoints and their current status at http://ethereumnodes.com.”

Michael O'Rourke pointed out:

“If your Metamask wallet doesn’t work, you can change the RPC provider to Pocket at the following URL: https://eth-mainnet.gateway.pokt.network/v1/5f3453978e354ab992c4da79”

Any consensus failure is a serious problem, and an unannounced hard fork shows that these vulnerabilities could be very harmful to Ethereum if left unchecked.

Fortunately, thanks to the keen eyes of bug bounty hunters, and the diligent work of Ethereum developers, no damage was done this time, and as Nikita Zhavonronkohov wrote on Twitter, the fix looks to be pretty simple…