Mike Hearn: The controversy over Bitcoin hard forks and soft forks

The long, hot summer of 2015 has revealed something surprising about the bitcoin community: many people are unfamiliar with the technical principles of how the blockchain algorithm actually works.

Eventually, Michael Marquardt, the administrator of bitcoin.org and various forums, banned all discussion of Bitcoin XT and threatened to remove services using Bitcoin XT from the official Bitcoin website. His reasoning was that Bitcoin XT was an altcoin because it triggered a hard fork. On the other hand, he believed that "soft forks" were completely acceptable.

This position is fundamentally untenable from a technical point of view, and in this post I will explain why.

Soft forks and hard forks

You may not see these two words on Bitcoin forums, and in my random chats with Bitcoin enthusiasts, I find that many people throw this terminology behind. Time and again, I encounter people who don't understand what these words mean, but I think they should understand.

This is not surprising, the concept of a “soft fork” is relatively new and seems to have emerged from IRC discussions rather than BIPs or other types of standards documents. What is strange is that even developers can define the term incorrectly.

A hard fork is when the rules of the Bitcoin protocol change and old nodes refuse to accept blocks created by new nodes. Blocks that violate the rules will be ignored and miners will follow their rule set and create blocks after the last block they witnessed.

A soft fork is when the rules of the Bitcoin protocol change, and old nodes will not realize that the rules are different, they will follow the changed rule set and continue to accept blocks created by new nodes. Miners may work on blocks that they do not understand or verify at all.

This is often described by some Bitcoin Core developers as "向后兼容", but this is incorrect and can be confusing to others. Backward compatibility is defined as new software accepting data or code generated by older software, such as Windows 10 being able to run Windows XP applications. Both types of forks need to be backwards compatible, otherwise new nodes will not be able to verify the blockchain from scratch.

The correct term is actually向前兼容: it means that the old software continues to accept the data and code generated by the new software. When you save a Word 2013 document, you can still open it with Word 2011. This is forward compatibility.

It is worth mentioning that Satoshi Nakamoto himself did not use the phrase "hard fork". Presumably, he had never encountered a fork before, and the concept of soft fork had not yet come around. That's right, because the concept itself has a big flaw: in a normal Bitcoin network, soft fork should never happen.

This may seem like a bold claim, but let’s analyze it a little further.

Why are soft forks bad?

There may be situations where old software continues to process data without the need for an upgrade: why wouldn't you want that? The reality is that software engineers love both forward and backward compatibility, and if your new application can interoperate with older versions, it's usually a win-win.

Forward compatibility is rarer than backward compatibility, because if you add some new functionality to your files or protocols, older versions of software won't understand it. Sometimes you can make the new functionality optional: meaning that old software can ignore it. This is called feature regression, and it's a popular technique, especially when some users want to use old browsers to access a site that uses new features.

Forward and backward compatibility is already very common in software engineering, so is Bitcoin any different? The reason is simple:

向前兼容，会使运行一个节点的全部目标失效。

The reason is easy to explain. The download of both XT/Core versions and the processing of each block start from the genesis block. The purpose of this process is to audit the Bitcoin blockchain. Every time a number is added, they check each signature. When the audit is completed, you can ensure that the ledger you use applies the rules written into the software.

This is a key part of Bitcoin’s decentralization: by using your own copy of the ledger, running calculations through your computer, and then accepting only valid bitcoin payments based on that ledger, you are enforcing Bitcoin’s key economic policies.

The situation is simple: no one, not even miners with a majority of hash power, can trick you into accepting money that violates these rules.

Or...can they?

Here’s the problem with soft forks.

In a soft fork, a protocol change is carefully crafted to essentially trick old nodes into believing something is valid, when it may not actually be.

Here’s an analogy. Imagine a large firm that has a team of auditors and a trading team. The traders want to do a new type of trade, but the firm currently doesn’t allow it: the auditors enforce the firm’s policy and check what the traders are doing. And changing the strategy can be a slow process. Suddenly one day, a trader has an idea. “Hey, guys,” he says, “I have an idea. I’m going to submit some derivatives trades. I’m going to write it down on paper, and when you see it, it will look like a normal trade, but it’s not, and the auditors won’t be able to tell!”

The auditors here are the people and services that run Bitcoin full nodes, and the traders are the ones who want to change the rules. Whether such rule changes are a good idea or not is not relevant here: what matters is how they are doing it. The auditors are cross-checking every transaction, but their calculations may come up with wrong answers because they don’t understand the true nature of the transactions they are verifying.

How does this work technically? We can look at a Bitcoin pattern called P2SH , which is a very useful pattern that makes multi-signatures easier to use, but it looks weird on the blockchain:

https://blockchain.info/tx/f837ca5c1a15fa6c2e5c7380386bacba10e936fe8625e12d9fa6c177a8f605c1

 OP_HASH160 6af7caf9b09224af8a171318f69d254c1756e54e OP_EQUAL

This is a Bitcoin script, which means in English: "Anyone who knows the password can spend this money."

The problem with it is that it is insecure: because there is no signature, when you broadcast a transaction and provide the "password" to anyone, others can also make an output transaction because the password is now public, so in the end it becomes purely a race to see whose transaction is passed to the miner first.

P2SH is possible because the Bitcoin protocol was modified to include a new rule: when you see an output script of the form above, don't actually treat it as a script, but as a special treatment: the password is actually the "real" script that is run. So P2SH is actually safe, don't worry. But why use such a roundabout and strange way?

You guessed it, soft forks are the problem. The structure was always designed so that old nodes that don’t understand the P2SH rules will consider transactions valid. If a transaction comes along that spends coins that were made under the old Bitcoin rules but don’t meet the new P2SH rules, censors won’t be able to check it correctly and will calculate an incorrect ledger.

But preventing this from happening is the whole reason you want to run a full node in the first place! Oops!

What exactly happens during a hard fork?

Let’s imagine P2SH is introduced in a different way: via a hard fork, then the script would look like this:

 OP_HASH160 6af7caf9b09224af8a171318f69d2... OP_EQUAL OP_RUN

Note OP_RUN at the end: this is an opcode that does not exist in the current Bitcoin instruction set. Correctly defined, the script would look like this:

 OP_HASH160 6af7caf9b09224af8a171318f69d2... OP_EQUAL OP_???

When they get to OP _ ??? point, they will stop and reject the transaction and the block because they will realize that they have no idea what this transaction means.

I should point out that there is actually a counterpart to P2SH called OP_EVAL , and there is some very exciting discussion about it. I don’t remember the details of that debate and don’t want to rehash it here, this is just an example.

OK, so your node has rejected a block because it doesn't understand it. What to do? In our imaginary company, the auditor summons the CEO (that's you) to make a decision, and you are the decision maker. In Bitcoin's case: you are alerted in some way, such as a text message or email, and you make the decision on what to do. You can...

1. Read the rules change, then if you're ok with it, upgrade and move on.

2. Read about the rule changes, and then decide that this is not a good idea and needs more consideration.

3. Explicitly trust any script output you don't understand, which you might do if you feel your node's uptime is more important than correct audit results.

The last option is risky, but you will find that the soft fork happens! The difference is that you explicitly requested it, and your choice does not affect others. Only you risk having an incorrect ledger. Bitcoin Core and Bitcoin XT do not support the third option today, but if anyone wanted, it would be easy to add a switch to enable it.

My point? Knowledge is power. When you know the rules have changed, you can use that information to make better decisions. With a soft fork, you won’t know the rules have changed, and it’s like flying blind.

choose

Recently, bitcoin.org adopted a new policy which states:

Contentious hard forks are a bad thing for Bitcoin. At best, a contentious hard fork will make people who choose the losing side chain feel disenfranchised. At worst, it will cause Bitcoin to lose its value forever.

This is a very strange statement, because it would mean that it is possible to modify Bitcoin without causing so-called “option deprivation”.

Any change to Bitcoin is made by the majority accepting the rule change, and it is these people who "disempower" those who do not want the change. Hard fork or soft fork, no matter which technology is used, if the transaction uses the new rules, it will enter the wide broadcast, and sooner or later you will receive the payment and trace it back to a transaction with the new rules. Then you have two choices, you can verify the new rules, or not verify. The problem with not verifying is that you reduce your security: it is like cutting the nose off your face. Therefore, in practice, everyone will always choose to upgrade. The majority disagreeing with the soft fork is no different than the majority disagreeing with the hard fork. If you don't like the new rules... then just choose firmly.

We can see this demonstrated by P2SH itself, a competing proposal that had some support from miners and developers, but ultimately everyone accepted the version we have now. They had no choice in the matter, or rather, they chose to accept the change and continue to use Bitcoin, and a soft fork would have made no difference.

Therefore, it is technically impossible to describe the issue of hard forks vs soft forks as a great struggle for individual rights. They are completely unrelated.

What does the fork mean for miners and merchants?

This article is a bit long, so I will try to keep it simple. If you are a miner or a businessman, you may prefer the hard fork change method for the following reasons.

Merchant: For the money paid to you, you want the transaction to be verified as much as possible, and you want it to be verified quickly. This means that your node needs to be running the main ruleset, and if you are left behind, you can be cheated. If you are on a chain that is not the main one, you need to know as soon as possible, and then you need to ignore the transactions you cannot verify.

Note that after I raised some issues with Bitcoin Core developers, they adjusted the code so that some soft forks are actually closer to hard forks: Core will ignore transactions that appear to use unknown opcodes (which is what we want!)... until they appear in a block. So whether this adjustment helps you depends a lot on timing.

Miners: If you are a miner and fall behind the majority chosen rule, you will lose financial rewards for being on a forked chain. However, a hard fork is better. First, it is detectable, so a properly configured node can let you know via email/sms/call that your node is out of date. It can also automatically shut down mining operations, so you don't waste resources mining invalid blocks. Second, if you think the other chain is wrong, and you think they will agree with you, you can continue to try to mine the weak chain. Then at some point, everyone returns to your chain, and you get the lovely coinbase that all other miners don't have. Of course, this strategy is very dangerous and I would not recommend it. But it is technically possible.

With a soft fork, you mine what you think is a valid block, but it is actually a block that other miners ignored. Then, when they orphan your block, you think it is just plain bad luck and try to build another invalid block… and you will get orphaned again. If you are just a small miner who does not produce blocks very often, it may take you quite a while to figure this out. And you still have a hole in your pocket.

Will a Hard Fork Destroy Bitcoin?

No, if there was such a thing, then literally anyone could kill Bitcoin by mining a block or two with different rules. The fundamental purpose of the blockchain algorithm is to ensure that nothing destructive can happen to it.

The idea that a "contentious" hard fork will cause Bitcoin to permanently lose value comes from the policy of bitcoin.org. Requiring 100% (or 99%) approval of any change to Bitcoin, or the approval of at least one Bitcoin Core developer, means that anyone can hold the entire community in place, and they can redeem it by refusing to agree unless they get what they want, and no infrastructure works this way. If Bitcoin were like this, it would never develop and would eventually become worthless.

－－－－