What is a hard fork, what is a soft fork, and what is consensus?

Chapter 0 Introduction

Bitcoin is a set of software. Modifying and upgrading the software code often involves two concepts: hard fork and soft fork. What do these mean? The most common definition in the community is that it is a modification of consensus. But it is difficult to understand what "consensus" specifically refers to. Let's look at their definitions at the level of actual code modification.

Chapter 1 Bitcoin Transaction Format and Transaction History

Before understanding what hard forks and soft forks are, we must first know what they are going to modify.

Bitcoin transactions have a dual meaning in Chinese, which is a key point that hard forks and soft forks are often misunderstood.

One of the meanings of Bitcoin transaction refers to the data structure that we use to send Bitcoin . This is a set of rules that all of us must follow when sending Bitcoin, no matter what wallet software you use. It is defined as a data structure, and each field in it represents a meaning. All wallet software can understand this meaning.

Another meaning of Bitcoin transaction is the event of sending Bitcoin between people who use Bitcoin. For example, if I send you 0.12345 BTC, this is a transaction. All these events of sending Bitcoin constitute the Bitcoin transaction history, which is the main content of the more than 80G data that currently constitutes the complete Bitcoin node. All these records can be checked on the blockchain browser. You can check all your transaction records by simply typing your Bitcoin address on the blockchain browser.

Let me explain it with an analogy. The transaction format is similar to the remittance slip we fill out when we go to the bank to remit money to someone. The remittance slip clearly defines the remitter's name, ID card, mobile phone number, account number, amount, recipient's name, account number, etc. The Bitcoin transaction format also specifies the remitter's information and recipient's information, but this information has nothing to do with a person's identity information. Please see the next chapter for the specific information required by Bitcoin transactions.

Chapter 2 Bitcoin Transaction Format and Block Format

The concept of Bitcoin transaction history is easy for everyone to understand, so I won’t elaborate on it. However, the Bitcoin transaction format, that is, the data structure of the transaction, needs to be explained in detail, otherwise it is difficult to understand what hard forks and soft forks are.

A Bitcoin transaction is a data structure containing input and output values, which contains code information for transferring funds from the initial point (input value) to the target address (output value). This data structure contains some fields, let's take a closer look at these fields, excerpted from Chapter 5 of "Mastering Bitcoin".

All Bitcoin wallets must send Bitcoins in the manner specified in this form, otherwise they will not be recognized on the Bitcoin network, which means the transaction is invalid. This is like going to a bank to transfer money, but if you don't fill out the remittance form according to the bank's instructions, the bank will not accept it.

The Bitcoin network collects transactions from various nodes within an average of 10 minutes, packages them into a package called a block, stamps them with a timestamp, and then adds them to the blockchain. What is a block?

A block is a container data structure that aggregates transaction information (that is, the average Bitcoin transactions sent by each node on the network within ten minutes). It consists of a block header containing metadata and a long list of transactions that form the main body of the block. The data structure of a block is defined as follows.

All mining node wallets must package blocks in the manner specified in this table, otherwise they will not be recognized by other complete nodes on the Bitcoin network (SPV nodes do not count), which means they will be considered invalid transactions.

Chapter 3. “Official” Definitions of Hard Forks and Soft Forks

Both hard forks and soft forks are defined on bitcoin.org, but the definitions are actually very vague. To prevent me from talking nonsense (questioning the definition on bitcoin.org really makes me feel very nervous), I quote the original text:

A hard fork is defined as follows:

A permanent divergence in the the block chain, commonly occurs when non-upgraded nodes can't validate blocks created by upgraded nodes that follow newer consensus rules.

https://bitcoin.org/en/glossary/hard-fork

A permanent divergence occurs in the blockchain. After the new consensus rules are released, some nodes that have not been upgraded cannot verify the blocks produced by the upgraded nodes, and a hard fork usually occurs.

The definition of a soft fork is:

A temporary fork in the block chain which commonly occurs when miners using non-upgraded nodes violate a new consensus rule their nodes don't know about.

https://bitcoin.org/en/glossary/soft-fork

When the new consensus rules are released, nodes that have not been upgraded will produce illegal blocks because they are unaware of the new consensus rules, resulting in a temporary fork.

I guess the two vague definitions on bitcoin.org are mainly responsible for the community’s dispute over whether to implement a hard fork or a soft fork for block size expansion. The biggest responsibility is that the definition of “consensus rules” in the original text is not clear. If we go to bitcoin.org to find out what “consensus rules” means, the original text is as follows:

The block validation rules that full nodes follow to stay in consensus with other nodes.

https://bitcoin.org/en/glossary/consensus-rules

The block validation rules that full nodes follow in order to maintain consensus with other nodes.

This definition of consensus logically commits the fallacy of circular authentication, which means that "consensus is the rules for maintaining consensus." This is the same as what Xu Sanduo said: "To live meaningfully is to live well." "Then what is to live well?" "To live well is to do meaningful things." In fact, he did not answer what is to live well and what is meaningful.

In addition, the two definitions define hard fork as "permanent divergence" and soft fork as "temporary fork". This definition method, which is almost a play on words, makes many people in the community afraid of hard forks. They think hard forks are a very scary thing.

To be honest, I have had the outline of this article in my mind for a long time, but I have never dared to write it, because I doubt the definition on the holy land bitcoin.org. I read the definition of Bitcoin protocol upgrade on bitcoin.org, but I couldn't find a description that would clear my doubts. I asked many big Vs in China, but I didn't get a complete answer. In the past few days, soft fork enthusiasts in the community have posted a large number of articles to promote soft forks and call for resistance to hard forks. After reading and reasoning, I am sure that they are wrong within my knowledge system. The root of their mistakes comes from the understanding of "consensus" as an illusory description, and they don't know what it specifically refers to. It seems that as long as "consensus" is mentioned, it is sacred and inviolable, and all logic and reasoning stop at "consensus".

Since there is no abstract definition of soft fork and hard fork, let’s first analyze what they actually do.

Chapter 4: Specific Examples of Soft Fork Upgrades on the Bitcoin Blockchain

A soft fork upgrade was recently implemented on the Bitcoin blockchain a few months ago, which made BIP68/112/113 officially effective in the Bitcoin protocol. The community called it the CSV soft fork. The upgrade was completed around June 2016. Where did this soft fork change the Bitcoin code? To answer this question, we need to look for the answer in the Bitcoin transaction data structure.

What changes have occurred to the Bitcoin transaction data structure before and after the CSV soft fork?

The field in bold red in the picture is the main modification of the CSV soft fork. Before the CSV soft fork, this field was "Serial Number (currently unused transaction replacement function)" as shown below:

It is precisely because this field in the original Bitcoin transaction data structure was not used, or was vaguely defined at the time, that it can be redefined. This unclearly defined field will not be carefully verified on the old version of the Bitcoin full node. The new version of the node produces a new block according to the defined rules, and it can also be verified and accepted by the old version of the node. This is the specific process of soft fork.

But this is obviously not a long-term solution, because there is only one such field. Once you use it, it will be gone in the future.

Bitcoin also had an important upgrade in 2012, called P2SH, which was called a multi-signature soft fork by the community. This was a modification of the Bitcoin transaction signature script, making it very convenient for Bitcoin to send transactions through multi-signatures. What was modified that time?

The field in bold red in the picture is the main modification of the multi-signature soft fork. Satoshi Nakamoto initially defined this field as P2PKH (Pay-to-Public-Key-Hash). Currently, this field can have 5 scripts.

Because after the multi-signature soft fork, when the nodes that have not been upgraded verify the blocks generated by the upgraded nodes, they can also verify this new P2SH locking script, so this is also a soft fork.

The above two soft forks are modifications to the Bitcoin transaction data structure. Soft forks can also modify Bitcoin's transaction history in disguise.

On August 15, 2010, a hacker exploited a loophole in the Bitcoin code and created a transaction containing 184.4 billion Bitcoins on the 74638th block. The developer discovered the issue within half a day and released a patch to invalidate the output of the transaction. However, this does not simply invalidate the transaction itself, but invalidates a category of transactions called "negative outputs". The transaction just now exploited the loophole that Bitcoin did not prohibit negative output values ​​before, and only defined that the total output amount could not be higher than the total input amount. The patch (probably released by Satoshi Nakamoto) fixed this loophole. What exactly was modified?

The field in bold red in the picture is the main modification of this vulnerability patch. The rule before the modification was that the "total amount" could not be higher than the "transaction", and the rule after the modification added that the "total amount" could not be a negative value. This modification allows non-upgraded nodes to verify the blocks generated by upgraded nodes, so it is also a soft fork.

Now let’s abstract these three cases and give a definition of soft fork: soft fork means that when the data structure of Bitcoin transactions (this is the widely circulated “consensus”) changes, non-upgraded nodes can verify blocks produced by upgraded nodes, and upgraded nodes can also verify blocks produced by non-upgraded nodes.

The changes to the "consensus" brought about by soft forks certainly also include changes to the block format, but the three cases here are all changes to the transaction format.

Let’s look at the hard fork case below.

Chapter 5: A specific example of a hard fork on the Bitcoin blockchain

On March 12, 2013, Bitcoin Qt 0.8.0 was released. Version 0.8 adopted a new database, level db. Some miners upgraded their nodes to Bitcoin Qt 0.8, while others continued to use Bitcoin Qt 0.7. Both parties produced blocks, but the blocks produced by the new database adopted by Bitcoin Qt 0.8 were rejected by the Qt 0.7 nodes. The specific reason was that the old database sometimes did not accept blocks larger than 800Kb. Therefore, at block height 225430, the Bitcoin blockchain was split into two chains, resulting in two chains in the Bitcoin blockchain, one containing blocks larger than 800kb, and the other refusing to recognize these chains containing larger blocks, which led to a hard fork.

At that time, miners who used bitcoin qt version 0.8 abandoned the chain they were mining and returned to bitcoin qt version 0.7 to continue mining.

This hard fork was an accident. It was caused by a bug in the bitcoin qt 0.8 version of the software, which caused the nodes using the old software to refuse to verify the blocks produced by the nodes using the new software. But the cause of the hard fork was that the nodes using the old software version refused to verify the blocks produced by the nodes using the new software version, and then both parties mined separately.

On July 4, 2015, the Bitcoin blockchain had a hard fork at block height 363731. At that time, Bitcoin Core developers added BIP 66 to the new version of Bitcoin Core 0.10.0. This was originally a soft fork modification. When the major mining pools on the Bitcoin network used version 0.10 of the software, one mining pool, BTC Nuggets, did not upgrade, resulting in two blocks mined by BTC Nuggets being rejected by other miners. Then both parties mined and continued to mine what they believed to be the correct blockchain, resulting in a hard fork and splitting into two chains.

Subsequently, bitcoin.org issued an announcement calling on miners to upgrade to bitcoin core version 0.10.2 to eliminate the fork.

This was also an accident. The cause of the hard fork was that the nodes using the new software version refused to verify the blocks produced by the nodes using the old software version, and then both sides mined separately.

So far, both hard forks were accidental, but the cause of the hard fork was the disagreement on the format of the newly generated blocks on different nodes, and the disagreement on the transaction format on different nodes. However, because it was an accident, the community did not discuss enough information, and I could not figure out which field of the block or transaction was modified to cause the fork.

At this point, we can abstract these two cases and give a definition of hard fork: a hard fork refers to when the Bitcoin block format or transaction format (this is the widely circulated "consensus") changes, the non-upgraded nodes refuse to verify the blocks produced by the upgraded nodes, but the upgraded nodes can verify the blocks produced by the non-upgraded nodes, and then everyone continues the chain they think is correct, so it is divided into two chains.

Next, let’s look at the soft forks and hard forks that the community is planning but has not yet activated.

Chapter 6: The planned block size increase hard fork

The Bitcoin community is currently planning a hard fork because the current blocks are filled with transactions. In order to accommodate more transactions in a block every ten minutes, the data structure of the block needs to be modified. Let's take a look at the data structure of the block first.

Currently, the main complete node software of the Bitcoin network is Bitcoin Core 0.12. This software stipulates that the maximum value of the field "block size" in Table 6 is 1M. This results in the last field "transaction" being limited in the amount of Bitcoin transaction data that can be accommodated. A transaction is at least 250 bytes, and 1Mb can only hold more than 4,000 transactions, and can only process an average of 7 transactions per second. Since the actual transaction size often reaches 500 bytes, in fact, it can only accommodate an average of 3 transactions per second.

So some people proposed to increase the maximum value of this field. For example, the Bitcoin Classic software has set the maximum value of this field to 2M, and plans to take the median of the previous 2016 block sizes and multiply it by an agreed multiple to determine the upper limit of the size of the next batch of blocks. Bitcoin XT has changed this value to 20M, and doubled it every two years until the upper limit reaches 8.3G. Bitcoin Unlimited directly changed this field so that the mining pool can decide how big to pack.

The problem is that the blocks produced by the nodes using the software that modified this field will be incompatible with the nodes that have not been upgraded, and the nodes that have not been upgraded will refuse to verify the blocks produced by these new nodes. This will lead to a hard fork.

At this point, we basically know that the so-called "consensus" that needs to be modified in a hard fork refers to modifying the block data structure format or the transaction data structure format.

Chapter 7. Segregated Witness Soft Fork

Another disguised expansion plan for blocks is Segregated Witness. This plan modifies the Bitcoin transaction data structure. Let’s take a look at the picture.

The red field in the figure is where the segregated witness mainly works. This is to move this part of the data out of the data structure defined by the transaction and also out of the data structure defined by the block. Therefore, the transaction has less data, so the size of each transaction will be smaller, and the entire block 1M space can accommodate more transactions.

But removing this field will also cause non-upgraded nodes to refuse to verify blocks produced by these upgraded nodes. However, there are ways to prevent them from refusing, but the code is very complicated to write. Anyway, this code has been written for more than a year, and it has not been completed yet. The more complicated the code, the more potential vulnerabilities it will have. Think about Bip 66, which was a soft fork that led to a hard fork, and this segregated witness is even more complicated.

At this point, it has been verified once again that the "consensus" that needs to be modified in the so-called soft fork refers to modifying the block data structure format or modifying the transaction data structure format.

Chapter 8 The "consensus" to be modified by soft forks and hard forks is a modification of the data structure

Now we can summarize what soft forks and hard forks actually do, especially what the difference is between soft forks and hard forks, so as not to be confused by the inaccurately defined "consensus".

The "consensus" modified by the soft fork specifically refers to the modification of the Bitcoin transaction data structure or the modification of the Bitcoin block data structure.

The "consensus" modified by the hard fork specifically refers to the modification of the Bitcoin transaction data structure or the modification of the Bitcoin block data structure.

At the level of specific modified objects, there is absolutely no difference between soft forks and hard forks. So far, all the forks that have occurred and are planned to occur are like this. They all modify or attempt to modify the transaction data structure or block data structure.

The hard forks and soft forks that have occurred or are planned to occur will never modify the transaction history. Even the loophole that caused a huge amount of coins to be swiped on August 15, 2010, modified the transaction data structure, but the incidental effect was to invalidate the transactions in a block because the transaction was illegal under the definition of the new transaction data structure.

Therefore, there is essentially no difference between soft forks and hard forks in terms of modifying the "consensus" level.

So what is the difference between a soft fork and a hard fork?

Chapter 9 The main difference between soft forks and hard forks is the compatibility between new and old nodes

The difference lies in the compatibility between new and old nodes. After the soft fork modifies the data structure, the transactions and blocks produced by the new node can be verified and accepted by the old node, but the hard fork cannot.

Because the hard fork modifies the data structure, the blocks produced by the new nodes will be rejected by the old nodes. If the old nodes refuse to upgrade the software and insist on continuing to mine according to the old data structure, then Bitcoin will produce two chains.

In a soft fork, there will be no two chains because no upgraded nodes can accept the new data produced by the new nodes. However, it is very difficult to make the data produced by the new nodes and the old nodes fully compatible with each other because they are essentially different data structures and everyone must recognize each other. Once a node refuses to verify different transactions or blocks, it will become a hard fork. This is why the Bip 66 soft fork eventually became a hard fork.

From Chapter 4, we can see that the specific method of modifying soft forks is to redefine the originally defined fields, such as multi-signature soft forks, or to define the fields that were originally retained, such as CSV soft forks.

But now from the data structure of the transaction, all the fields have been occupied and defined accurately and in detail. If you want to do a soft fork again, you can only erase the original definition of some fields and redefine them. This will cause the original function to be lost. If this function is indispensable, it will lead to a hard fork, and the new and old nodes will reject each other. So soft forks must be done with extreme caution. This is what segregated witness does. Segregated witness directly deletes a certain field, but in order to ensure the compatibility between the new and old nodes, that is a waste of effort.

A hard fork does not consider whether the non-upgraded nodes will reject the data and code produced by the upgraded nodes, which is much simpler.

Let’s take a look at the difference between the compatibility required to achieve a soft fork and a hard fork.

There are six major changes in the new and old data produced by the new and old nodes to achieve soft forks and hard forks:

1. Old nodes that have not been upgraded;

2. Upgraded new nodes;

3. Transactions in old transaction formats that occurred on old nodes;

4. Transactions in new transaction formats occurring on new nodes;

5. Old blocks produced by old nodes only contain old block formats;

6. New blocks containing new block formats produced by new nodes.

In fact, there are more complex factors to consider, including full nodes; SPV nodes; the ratio of new and old node computing power; SPV mining nodes; unconfirmed transactions and multiple confirmed transactions; CSV transactions; RBF transactions... All of these are factors to consider for compatibility. But for simplicity, I will only consider the above 6 major changes here, let's make a table first.

To ensure perfect compatibility, soft forks require the following two conditions:

1. Upgraded nodes accept transactions and blocks produced by non-upgraded nodes (backward compatibility);

2. Nodes that have not been upgraded accept transactions and blocks produced by upgraded nodes (forward compatibility).

The compatibility that the hard fork needs to ensure only needs to meet the first situation above, that is, the new node needs to verify the entire blockchain starting from the 0-height block.

Let's use an analogy to illustrate these two compatibilities:

Backward compatibility is actually the easiest to understand. It means that we can use word 2013 to open word 2010 files. Forward compatibility means that the old version of the software must unconditionally accept the data produced by the new version of the software that it cannot understand. It's like you use word 2010 to open a word 2013 file. I don't know how difficult it is. How can the old version of the software know the data that has not been defined yet? The only thing it can do is ignore these new functions.

Hard forks do not consider such complex compatibility. First of all, they refuse to be forward compatible. Hard forks will not care about the fact that non-upgraded nodes refuse to verify blocks produced by upgraded nodes. If you don’t want to be verified, I won’t play with you if you don’t upgrade.

Chapter 10 Comparing the pros and cons of soft forks and hard forks

Soft forks can ensure that people who do not want to upgrade do not upgrade. This demand for not wanting to upgrade is actually very common in real life.

A hard fork must require all old nodes to be upgraded, otherwise the old nodes will not be able to recognize the transactions and blocks produced by the new nodes, causing the blockchain to be split into two chains.

The upgrade space of soft forks is limited, because all fields of the current Bitcoin transaction data structure and block data structure have been defined in detail. If you want to ensure forward compatibility, you cannot add new fields, otherwise the old nodes will reject you. Therefore, the upgrade space of soft forks is limited to redefining existing fields. For example, soft forks cannot redefine the "block size" field in the block data structure, which means that soft forks can never achieve a breakthrough in 1M blocks. Moreover, if there is a slight error in this extremely complex compatibility, the new and old nodes will be incompatible, which will lead to a hard fork. This has already happened once.

The upgrade space of hard fork is much larger, because hard fork only needs to consider whether the transactions and blocks produced by the old nodes can be accepted, and hard fork does not need to consider whether the old nodes will accept the transactions and blocks produced by the new nodes. Then hard fork can modify the transaction data structure and block data structure more boldly.

Finally, both the Bitcoin transaction data structure and the block data structure have a field called "version number", which means "clarifying the rules that this transaction or block refers to". This means that Satoshi Nakamoto wanted to use a hard fork to modify these rules, that is, if we want to modify the rules, we have to redefine the version number. However, the soft fork modified the rules without modifying the "version number".

Chapter 11 Conclusion

The word "consensus" has fooled too many people. In fact, soft forks, which are packaged as "not destroying consensus", are essentially the same as hard forks in that they modify the same objects, and the more terrible cost is forward compatibility, which requires software that is not upgraded to ignore the data and code produced by upgraded software. The benefit of taking such a risk is just being too lazy to upgrade the software. I think any rational person is unwilling to accept this risk-benefit ratio.

(I have always wanted to write about this topic, but I was afraid that I would not be able to write it well. Today I mustered up the courage to write it. It took me a whole day to write it. Please take a look. If there are any mistakes, please point them out. If you think it is useful, you are welcome to reward me with some coins.)