BiTor Open Class: Segregated Verification

Hello everyone, I am Wenhao from Bitai. Before starting this "Bitai Open Class", I would like to wish you all a happy new year!

It’s been a long time since the last “Bitai Open Class”, and the reason is still the same word: “busy”.

During this period, topics such as "Hard fork? Soft fork?" and "Core? Classic?" have caused great controversy in the Bitcoin community at home and abroad. Many people have asked our Bitcoin team for their opinions:

* Some foreign industry media have sent us questionnaires;
* Some users asked which side Bitai stood on (of course, I personally hate "standing on the side");
* Many people have consulted us about some relevant technical details. For example, some netizens have consulted us about the technical details of SegWit through QQ and WeChat.
In addition, I also participated in two related discussion meetings during this period:

* Conducted technical exchanges with core developer Jeff Garzik in Beijing with multiple companies;
* Participate in technical discussions related to forks at Haobit’s annual meeting;

While participating in the discussion, I found that everyone was arguing with each other and having a great time.

This open class will not discuss controversial issues. Since many users are concerned about the implementation of technologies such as "isolated verification", Babbitt also hopes that we can communicate with everyone about the relevant technical content. Therefore, I have prepared this "Bitcoin Open Class", hoping to explain the relevant content of "isolated verification" to everyone in the clearest and simplest way possible.

The method used in this open class is slightly different from the previous ones:

1. It is mainly conducted in the Bitai WeChat group through voice;

2. The BiTai QQ group and Babbitt Forum synchronize the main content in text format;

3. Interested students can choose their own convenient way to participate in this open class;

4. You can ask questions in several places, and we will choose to answer them according to the situation. However, please do not use voice when asking questions, as I may not have time to hear them.

Before we start talking about “isolated verification”, let me emphasize a few points:

1. This discussion will not discuss topics such as forks, block sizes, splits, who is right and who is wrong, but only the technical principles of isolated verification;

2. I will assume that the audience of this discussion has a basic understanding of things like hashes and signatures (you don’t need to really understand them, just roughly understand their meaning). I will not tell you “what is a hash?”, “what is a signature?” and other such things. If you really don’t understand, please Google it yourself.

3. I will explain the technical principles of segregated verification in the simplest way possible, so that even those who are not very technically savvy can understand the principles of segregated verification;

The main content of this open class can be found on the Babbitt Forum for those who missed it:

http://8btc.com/thread-28719-1-1.html

Segregated verification itself has nothing to do with the topic of block size. "Segregated verification has increased the block size to a certain extent" is actually a byproduct, not its main purpose. However, since the topic of block size and forks is too intense, and segregated verification is precisely the technical implementation that the conflicting party is working hard to promote, it is regarded as a technical implementation of increasing the block size corresponding to 2MB. For example, we often see people say "Classic is directly upgraded to 2MB, and Core uses another complex technology called segregated verification to upgrade to a maximum of 1.8MB." In fact, it is not the case at all.

Here, I must repeatedly emphasize this sentence: "Isolated verification is isolated verification. Its main purpose is the same as its literal meaning, which is to perform verification in isolation, rather than to increase the block size. Ultimately, isolated verification can increase the block size only as a by-product." Please remember this sentence.

Okay, before we officially get into the discussion of isolated verification, let me ask you a question: What is the structure of Bitcoin transactions?

The audiences of today’s lecture are either industry insiders or Bitcoin enthusiasts. Everyone has some understanding of Bitcoin transactions to some extent. Many people may have read similar articles about transaction structure. Coders may have also read the Bitcoin wiki and developer documentation, and even know some terms such as: TxIn, TxOut, OP_CHECKSIG, OP_EQUAL, ScriptSig, ScriptPubKey, etc. But do you really "understand" Bitcoin transactions?

For example, why is Bitcoin trading designed this way? What is good about it? What is bad about it?

Okay, before we officially get into the discussion of isolated verification, let me ask you a question: What is the structure of Bitcoin transactions?

The audiences of today’s lecture are either industry insiders or Bitcoin enthusiasts. Everyone has some understanding of Bitcoin transactions to some extent. Many people may have read similar articles about transaction structure. Coders may have also read the Bitcoin wiki and developer documentation, and even know some terms such as: TxIn, TxOut, OP_CHECKSIG, OP_EQUAL, ScriptSig, ScriptPubKey, etc. But do you really "understand" Bitcoin transactions?

For example, why is Bitcoin trading designed this way? What is good about it? What is bad about it?

Let's look at a specific Bitcoin transaction:

 bitcoin-cli getrawtransaction c0a0d0f4de3be3b57f516f127a6fa5d5895f09bfacbb001853593320fb0a4164
0100000001473e2a366b081a0e6645199692ccf2f6e25e4556a18265785a88b46d50fd68c2000000006a4730440220296dd1587b8290a04 a9253de431b5dbb698030527df5ca75592ab29eea12e85b0220737cafd85b79a7dea1b29c0a5753432d1dacb50f5225221974e38134617ff c85012103dd38b9b13b24498c382045afb08a4256c3c8bf86e15122df741bd2c2a0bd120fffffffff02a0d21e000000000017a914d16402 64c522d9a31aaf69207e57d0a224eb7e668770110100000000001976a914dfd6b01e5ec1ecb1568da79931ddd8327dafd9b588ac00000000

tx_id = tx_hash = c0a0d0f4de3be3b57f516f127a6fa5d5895f09bfacbb001853593320fb0a4164
raw_tx = 0100000001473e2a366b081a0e6645199692ccf2f6e25e4556a18265785a88b46d50fd68c2000000006a4730440220296dd1587b8290a04 a9253de431b5dbb698030527df5ca75592ab29eea12e85b0220737cafd85b79a7dea1b29c0a5753432d1dacb50f5225221974e38134617ff c85012103dd38b9b13b24498c382045afb08a4256c3c8bf86e15122df741bd2c2a0bd120fffffffff02a0d21e000000000017a914d16402 64c522d9a31aaf69207e57d0a224eb7e668770110100000000001976a914dfd6b01e5ec1ecb1568da79931ddd8327dafd9b588ac00000000

tx_hash = double_sha256(raw_tx)

This is actually a Bitcoin transaction in raw format (hexadecimal): rawtx. This is the kind of transaction that is broadcast on the Bitcoin network, and the Bitcoin transactions you send and receive actually look like this.

Some people may say that they don’t understand.
Don’t understand? It’s okay. You don’t need to understand the transaction format to understand today’s content.
In the previous transaction, the signature looked like this:

 30440220296dd1587b8290a04a9253de431b5dbb698030527df5ca75592ab29eea12e85 b0220737cafd85b79a7dea1b29c0a5753432d1dacb50f5225221974e38134617ffc8501

OK, now you can find a text editor (such as Notepad), and then search for the signature in the original transaction rawtx. Let me ask you a question: where does the signature appear approximately in the transaction?

Those who have searched should be able to answer this question now. It is in the front middle. In other words, the signature is basically in the first half of the transaction (actually the TxIn position).

Now the question is, why is the signature here and what is the benefit of having it here?

The answer is: Satoshi Nakamoto designed it this way, so the signature is here. As for the benefits, there are actually not many benefits. Not only are there no benefits, but there are also many disadvantages.

1. Due to this design, a lot of "cheats" are hidden in the transaction structure of Bitcoin. For example, unsigned transactions actually use another content called ScriptPubKey, which first occupies the position required for the signature, then signs the unsigned transaction, and then replaces the signature result. This is actually a very troublesome thing. It is not easy to explain clearly, and it is difficult to implement with code. When there are multiple inputs (TxIn) in a transaction, there is a set of "cheats" to prepare unsigned transactions in sequence and replace the signatures in sequence; and when a transaction involves multiple signatures, another set of agreed rules is needed to prepare unsigned transactions and then replace the data after signing; what if a transaction involves multiple inputs and multiple signatures? That is a cheat code within a cheat code, which is extremely troublesome. Those who are interested and capable may wish to look through the implementation of the Bitcoin protocol in various languages. They will find that no matter whether it is a Bitcoin protocol library written in C++, Java, Python and other languages, it usually requires writing a lot of messy codes to do the above-mentioned transaction data construction work, which is very troublesome. Those who have written it will understand.

2. Put the signature in the transaction structure. Due to the malleability of the signature, the transaction also has malleability. For example, I told you a long time ago that due to the symmetry of the elliptic curve, s and s' are both verifiable signatures. In this case, any node that receives the transaction you broadcast has the ability to replace s with s' and then rebroadcast it. The rebroadcast transaction actually has the same content (input TxIn and output TxOut), but has a different tx_id (tx_hash), which can achieve the purpose of confusing people and even affect the services of some Bitcoin companies (such as exchanges). Transaction ductility will not directly cause you to lose coins, but it will affect individuals and companies' judgment of the status of Bitcoin transactions, which may lead to possible asset management risks. Previously, bitcoin-core has made some attempts to solve the transaction malleability problem in BIP62, but in a sense, transaction malleability cannot be completely cured. At least the private key owner can regenerate a signature. The transaction remains unchanged but the signature changes, and the final tx_id also changes. This situation is applied to multi-signature occasions and may even affect the judgment of other key holders on the transaction, leading to some potential asset risks. Under the current transaction structure, we cannot completely prevent this situation from happening;

3. When we need to sign a transaction, the hash for signing each TxIn is different (see the "cheat" above), so the implementation of offline signatures such as "cold wallets" actually increases the difficulty, both in terms of data structure and code;

In other words, putting the signature in the transaction content does not actually have any benefit. Instead, it increases the complexity of the structure and the risk of transaction scalability, which is obviously not worth the cost.
So, what should the most reasonable transaction structure look like? In fact, it is very simple:

TxIns + TxOuts + [签名]

In other words, taking out the signature separately and putting it outside can solve all the above problems.

This brings us back to the title of what we are going to talk about today – “Isolated Witness”.

Segregated verification actually takes out the signature part of the original Bitcoin transaction and puts it into another structure called witness to achieve the purpose I mentioned.

After reading this, everyone should understand that the basic principle of "isolated verification", a topic that seems complicated, is actually not complicated. The goal to be achieved is actually very simple, which is to take out the signature and put it in another place, which is the so-called "isolation". It's that simple.

The signature is the data used for "verification", and putting it in another place is to "isolate" it, so it is reasonable to name this thing "isolated verification".

Through the above description, everyone should be able to understand the benefits of isolated verification, and its benefits are indeed huge:

Transactions are transactions, signatures are signatures. For transactions, you only need to care about TxIn and TxOut. We all know that a Bitcoin transaction is actually spending a previously received sum of money, which means that the previously received TxOut is used as the TxIn of this transaction and then sent to the subsequent TxOut. The isolated transaction structure means that when you decide which sum of money to spend and to whom to spend it, the content of the transaction is already determined, and the transaction data will not change, and the final tx_id is unchanged. Want to verify the signature? Then go to the witness to find the corresponding signature data to verify the legitimacy. It is simple and there is no problem of transaction ductility. In the language of programmers, it is really "too cool"!

As we all know, our BitTorrent wallet provides an easy-to-use and secure cold wallet solution. For us, of course, we hope that during the offline signing process, the hash that the cold private key needs to sign is accurate and not changing. This is actually one of the benefits of isolated verification.

Just now, I saw someone on the Babbitt forum asking whether isolated verification is good for the Lightning Network?

My answer is that isolated verification is beneficial to all extended applications based on the Bitcoin blockchain (even including exchanges and wallets, etc.). Accurate tx_id will certainly make many applications based on tx_id much easier and more reliable.

The above benefits are actually the essence of isolated verification.

The so-called isolated verification is not a change to solve the problem of block size, nor is it a conspiracy by several core developers of BlockStream in the Core team to obstruct expansion as some people say. Instead, Bitcoin transactions should be like that. The structural improvement of isolated verification, like the compressed format of the public key (for this part, please refer to our previous open class - Gossip about Satoshi Nakamoto, http://8btc.com/thread-14057-3-1.html), is a content that needs to be improved due to the unreasonable initial structural design.

In other words, Bitcoin transactions should have originally been "isolated verification", but because Satoshi Nakamoto originally designed it into its current form, it has been retained to this day. If it had been resolved to change it back then, perhaps there wouldn't be so many controversies today. Bitcoin programmers would have been able to save a lot of brain cells to do more meaningful things instead of struggling with the structure of transactions, and various Bitcoin companies would have suffered less from the problems of transaction scalability, and so on.

Okay, after discussing so much, should we do isolated verification?

My answer is: Of course it should be done. Why not make reasonable improvements? For example, should the improvement of "compressed public key" be made?

How should improvements to isolated verification be made?

Due to the need to consider backward compatibility and compatibility with older versions of various clients, developers hope to use a "soft fork" approach as much as possible. In order to achieve this, a total of 4 related BIPs (141-144) have been proposed.

We all know that it is easy to design a new structure for any computer system, but it is much more difficult to achieve backward compatibility. It is even more difficult to make the new and old structures coexist and the old version run normally, especially when we consider that the data of these new and old structures have to be transmitted on the same blockchain, more efforts are needed. These parts are described in detail in several BIPs. Those who are interested can read the relevant technical documents:

For example:
bip141 describes in detail a new structure called "witness" to which data such as scripts and signatures are moved, and also introduces how Segwit can achieve backward compatibility.
bip142 describes the new address format for isolated verification;
bip143 describes the verification method of isolated verification transaction signature with examples;
bip144 describes how to notify nodes of their support for isolated verification and how to propagate and request verification data while maintaining backward compatibility;

Well, after explaining all this, everyone should have a general understanding of what isolated verification is all about. Isolated verification itself has nothing to do with block size. It is just that the verification data is placed in another space for the purpose of compatibility. At the same time, the calculation method of the block upper limit is modified so that isolated verification itself can play a certain role in capacity expansion. This is why everyone regards it as another capacity expansion solution and as the opposite of the 2MB solution.

Finally, to sum up, isolated verification is actually an optimization of the previously unreasonable Bitcoin transaction structure. Developers tried to implement it in a "soft fork" way with as little impact as possible and backward compatibility as possible. This improvement itself (just like "compressed public key") is very reasonable and has nothing to do with the block size dispute. If the soft fork of isolated verification is launched, we may see the old and new versions of transactions coexisting on the Bitcoin network for quite a long time, just like some people still use private keys starting with 5 (instead of compressed versions of private keys starting with K and L) until today.

The increase in block size brought by isolated verification is just a byproduct. Whether the block size should be increased and how it should be increased is actually another topic. This requires full discussion by all parties and listening to each other's opinions as objectively and impartially as possible. I will not make any comments on the block size debate. As for the question of how to manage the Bitcoin code, this is a governance issue, which is a political issue and is not within the scope of discussion of this open class.

Segregated verification is an improvement that should be made. Because it is mixed with the controversy over block size, many people have a lot of emotional color when looking at "segregated verification". Personally, I suggest that segregated verification is segregated verification. If you want to achieve backward compatibility, soft fork and no problems, it requires the joint efforts of core developers, related companies in the upstream and downstream of the industry chain, and Bitcoin wallet developers to ensure the safety of this improvement. If you must ask me whether the segregated verification solution is risky, I think the biggest risk is that everyone is forcing you to say "When will your segregated verification solution be available? 2MB can be forked next month", forcing core developers to rush to launch it in order to achieve the purpose of increasing the block size, which is a bit of a loss. The upgrade of segregated verification still needs to be fully tested, and related companies and software solutions should also test compatibility as much as possible before going online.