Flexible Transactions

I’ve been asked a question more and more frequently lately about the Segregated Witness solution, specifically the hard fork-based version.

Segregated Witness (or "SegWit" for short) is incredibly complex. It attempts to solve quite a few, disparate and unrelated problems, and attempts to do so in a backwards-compatible way. This is no small feat!

So, what problem does SegWit seek to solve? We can find this information in the benefits document:

1. Ductility Repair

2. Linear scaling of sighash operations (i.e., twice as many hash operations as needed for signatures, 10 times as many operations as needed for signatures )

3. Input value signature

4. Increased multisig security through the Pay to Script Hash (P2SH) transaction standard

5. Script version

6. Reduce UTXO growth

7. Compact fraud proofs

As mentioned above, SegWit attempts to solve these problems in a backwards compatible way. This requirement is only set because the SegWit authors set it. It is set this way because they want to promote this protocol upgrade with a soft fork. This article will try to answer the question: Is this the best way to solve these problems?

Starting with malleability, the problem is that a transaction starts with the owner of the funds and the miner who mines the block. Because of the malleability issue, malleability changes will be considered a valid transaction, but the transaction identifier (TX-id) has changed. Before we go deeper, let's take a look at the transaction data structure.

If we look at the transaction data structure now, we will notice a few problems.

The original transaction format designed by Satoshi Nakamoto was to include a 4-byte version number. This design approach is common in the industry, and the approach is that when any field in the data structure needs to be changed, a new version is defined. In the case of Bitcoin, we have not yet achieved this, and we are still at version 1.

Instead, Bitcoin makes smaller, semi-backwards-compatible changes. For example, the CHECKSEQUENCEVERIFY feature repurposes the sequence field as a way to add data without breaking data for old clients. Incidentally, this particular change (detailed in BIP68) is not backwards-compatible in major clients because it relies on a transaction version number greater than 1, and they all check for standard transactions and say that only version 1 is standard.

A design with a version number implies that the designers intended to use a hard fork to handle the change. It should be obvious that a new client needs to know how to parse a newly designed data structure. Therefore, an idea is to change the version number so old clients will know that they cannot parse this new transaction version. In order to keep operating, everyone must upgrade to a client that supports this new transaction version.

Let's look at why we might want to change versions, I've highlighted some items in red that are confusing. Most notably, the numbers in the transaction are stored in 3 different, incompatible formats. This isn't necessarily a big, definite bug.

The transaction is cryptographically signed by the owner of the bitcoins so that others can verify that he can actually transfer the bitcoins from that address. The signature is stored in the TX-in-script.

Crypto geeks may have noticed something odd that goes against any textbook knowledge. Namely, that a digital signature must be placed outside the thing it signs. This is because a digital signature protects itself from being altered. But a signature itself can cause such an alteration. So you must store the signature outside the thing you sign. (Translator's note: That is, a digital signature must be placed outside the data it signs. This is because a digital signature protects the signed data from being altered, but a signature itself can cause such an alteration. So you must store the signature outside the signed data)

The creators of Bitcoin did some "clever" design on how to actually sign transactions, so the signature doesn't actually have to be outside the transaction. Most of the time, it does work. But we want it to work flawlessly, because currently these designs are too "clever" and cause terrible scalability issues that people have lost money on.

What about SegWit ?

SegWit actually only solves one problem in these projects. It moves signatures out of transactions. SegWit doesn’t solve any other problems with Bitcoin transactions, and it doesn’t change versions after a transaction is made, meaning that older clients are largely unable to understand its meaning.

Old clients will no longer be able to check the SegWit type of a transaction, because the authors of SegWit simply made SegWit transactions labelled "everything is OK" when the real data was moved to the "trailer", knowing that old clients would ignore the "trailer". (Translator's note: The authors of SegWit designed it similar to putting an "everything is OK" label on a car, but moving the real data to the trailer behind the car, knowing that old clients would not check the trailer)

SegWit wants to keep the data structure of the transaction unchanged and tries to fix the data structure of the transaction. This will lead to inconsistencies because you can't do both at the same time, so you have a non-ideal situation and hacks are expected.

The problem then is that SegWit introduces more "technical debt," a software term for when a development team chooses a design or architectural choice that is easy to implement in the short term, but has a more negative impact in the long term, which is the debt owed by the development team. The word "debt" is very accurate here, as over time, everyone using the transaction must understand the flaw and operate correctly. This is very similar to paying interest.

With a soft fork, old clients will not be able to validate transactions, or even fully parse them, but they themselves believe they are doing sufficient validation.

Can we improve on this?

I wanted to suggest a way to change the data structure of transactions once to make it more future-proof and fix the problems it encounters over time, including the malleability issue. It turns out that this new data structure enables SegWit to fix and resolve all of the other, fairly trivial, issues.

I'm going to propose an upgrade which I'm going to call:

Flexible Transactions

Last weekend I wrote a small application (source code here) that reads a transaction and writes it out in a new format I devised for Bitcoin. It's based on ideas I've been using for a while on other projects, but this is the first open source version.

The basic idea is to change transactions to be more like modern systems like JSON, HTML and XML. It is a "tag-based" format that has various advantages over the closed "binary-blob" format.

For example, if you add a new field, just like a tag in HTML, your old browsers will simply ignore it, making it backwards compatible and suitable for future upgrades.

More advantages;

• Solve the ductility issue and it will become trivial.

• A "tag" based system allows you to skip writing unused or default values.

• Because we are making this change, we can just use var-int (variable integer) to encode data instead of 3 different types in transactions.

• After adding new tags (such as ScriptVersion), no further modification of the transaction data structure is required. All existing customers can still parse all known data.

• The actual transaction time is about 3% shorter than the average (calculated for transactions over 200K)

• SegWit added a lot of technical debt, while my "Flexible Transactions" solution will not cause a lot of technical debt to be amortized.

A “Flexible Transactions” solution generally looks like this:

Flexible Transaction suggests using a list of tags, like JSON; "Name:" "Value". This makes the content very flexible and extensible. Instead of just text, Flexible Transaction uses a binary format. Note how unused tags are skipped. NLockTime and Sequence are not used, so they are skipped in the transaction.

The biggest change is that the TX-in-script (aka witness data) has been moved to the end of the transaction. When a wallet generates this new transaction type, they will append the witness data at the end, but the transaction identifier is obtained by hashing the data that ends before the witness data.

Witness data usually contains a public key and a signature. In the Flexible Transactions scheme, the signature is signed by the exact same data set, and the TX-input is generated using a hash algorithm. This solves the problem of scalability. If someone changes the transaction, this will invalidate the signature.

I tested this using a test application where I ran 187,000 recent transactions and checked what effect this change would have on the size of transactions.

• Transaction size dropped from an average of 1,712 bytes to 1,660 bytes, and the average size dropped from 333 bytes to 318 bytes.

• After they have been confirmed, transactions can be further pruned (by removing signatures). After pruned, the size drops by an average of 450 bytes, or a median of 101 bytes.

• In contrast to SegWit (Segregated Witness), all clients will see smaller transaction sizes after this upgrade.

• After signatures are removed, transactions and blocks can expect up to 75% size reduction.
  

Broken OP_CHECKSIG script command

Actually, to solve the scalability problem from the source, we need to fix this instruction. However, we can't change the original script until we decide to make a version 2 scripting language.

This change was not a good time to do version 2, which would have been crazy since at the same time we were also rolling out a new format for transactions themselves (too many changes at the same time is bound to cause problems).

This means that in order for the Flexible Transaction scheme to actually work, we now need to use a NOP code in our script that is never used before, and have it act essentially the same as OP_CHECKSIG, but, instead of being overly complicated in how it is signed, we simply define it to sign the exact same transaction fields that we also use to create the TX-ID. (See the far right column of the table above).

This new opcode should be relatively easy to code and will make it very easy for us to fix issues introduced into the script in future versions of the script.

So, how does this compare to SegWit?

First of all, introducing a new version of transactions does not mean that we stop supporting the current version. So all of this is perfectly backwards compatible, and customers can continue to trade on older versions using the old transaction format. Of course there will be some issues, but no one will end up being stuck.

For a transaction, using a marked format is a one-time hard fork upgrade protocol, which allows more changes and has much less impact on future systems. It is also similar to SegWit, after all, it strives for the same goal. However, SegWit attempts to adjust the static storage format by reusing existing fields, while the Flexible transactions scheme proposes a coherent and simple design that eliminates a large number of conflicting concepts.

Most importantly, years after the Flexible transactions solution was introduced, we can continue to benefit from the tagging system to extend and fix problems we haven’t thought of or discovered yet, using the same and consistent concepts.

Similar to SegWit, we can fit more transactions in the same (block) space, signatures (witness part) can be streamlined by full nodes without causing any security issues, these two solutions are the same. What SegWit does not implement is that unused functions do not take up space. So if a transaction does not use NLockTime (nearly 100% of them do), in SegWit they will still take up space, while this proposed solution will not. Expect your transaction size to be smaller, so fees will be lower!

In terms of size, SegWit aims to increase space by 60%. This is done by removing signatures to reduce overhead. In my tests, Flexible transactions showed a 75% increase in space.

SegWit also describes how to change the data stored in the block. It creates an additional branch on the Merkle process tree. Flexible transactions essentially solve the same block application problem as SegWit using the same method. This means we can also use the Merkle process tree solution. No changes have been made.

In the list of advantages mentioned above in this article, including the advantages stated by the author of SegWit (Segregated Witness). It turns out that these advantages are completely unrelated to each other, and they each have a very independent solution to their own problems. The tricky part comes from the requirements of old clients, who are forced to push them all to one "fix" problem.

Let’s review each of these:

Ductility Repair

All known scalability issues are resolved using this new version of the transaction data structure.

Linear scaling of sighash operations

The problem this attempts to solve is described in this bitcointalk thread (https://bitcointalk.org/?topic=140078).

As a result of this discussion, the SigOps variable was introduced, which soon proved to be suboptimal. A better solution was introduced in BIP109 called sig-hash-bytes (total number of nodes). It is based on the fact that hashing is the most expensive part of the verification process (note that hashing is also part of signature verification).

In this solution, a clear conclusion is given that a 1M block is only allowed to calculate a maximum of 650MB of hashes, and accordingly only 5 seconds of verification time is required. This is a protection for the block verification time so that it will not be particularly long.

This is a good balance, with 650MB of hashing per MB block to ensure block verification when the most extreme transactions occur.

This seems like a problem that doesn't need to be solved. Modifying the protocol, which could have unintended consequences, is overkill.

Signature of the input value

included in this proposal.

Increased multi-signature security through the pay-to-script-hash (P2SH) transaction standard

This change is one of many additional changes to SegWit that could be easily added later, as part of the Flexible transactions proposal listed in this article.

The bottom line is that with SegWit, using a larger hash to make SegWit-safe changes only works with SegWit, because SegWit does not solve the problem of transaction versioning, making processing it alone trivial and of little value.

Through the Flexible transactions solution, such changes can be made at any time in the future with minimal impact.

Script version

Note that this only introduces the versioning bytes. It does not actually introduce a new version of the script.

This is a good example where the tag format of the Flexible transactions scheme has advantages over the static storage format used by SegWit, because adding version tags like this is cleaner, more convenient, and less intrusive. Just add a new tag, which defaults to version 1, so it can be consistent with older versions of transactions without tags.

Imagine having to include "body background=white" on every HTML page, because it's not allowed to be dropped. This is what SegWit does now. Although there is currently no support for actually changing it.

Reduce UTXO growth

I recommend you read this carefully for yourself, it is quite interesting technically and I am sure many people will not fully grasp and understand the idea. The bottom line is that they claim that the UTXO database will avoid growth because SegWit will not allow more clients to be served.

I don't even know how to respond to a solution like this. This is classic throwing the baby out with the bath water.

Database technology has matured greatly over the past 20 years, and this database is trivial compared to what free and open source databases can do today. Sure, a UTXO database is slightly incompatible with a normal SQL database, but in the long run, no long-term project is going to do something like telling clients to go somewhere else.

Compact fraud proofs

Again, this is not included in SegWit, but only provides a basis for it. Flexible transactions provide the same basis, and here the two solutions are the same.

What do we offer that SegWit doesn’t?

1. Transactions become scalable. Future improvements are facilitated.

2. Smaller transaction size. Uses fewer features and requires less space.

3. We only use one integer type instead of 3 to encode it.

4. We removed technical debt and simplified. SegWit did the opposite.

Summarize

SegWit has some good ideas and some necessary amendments. It is possible to take the good and discard the bad, but it would require a hard fork. This post shows that the advantages are significant and worthwhile.

We introduced a tagged data structure. Conceptually, it is as flexible as JSON and XML, but the proposed solution is a compact and fast binary format. Adopting a Flexible transactions data format allows many future improvements and innovations to be consistent and clean, in a more backwards-compatible way than SegWit at later stages, even for longer periods of time. We can see that the fundamental things that SegWit tried to fix separately can be done all at once, and for Bitcoin as a system, this is less risky.

A year after SegWit was designed, we are still seeing issues and delays in releasing the software, and I think it’s time to discuss abandoning the requirement to maintain backward compatibility.

Introducing the Flexible transactions upgrade has a huge benefit because the transaction design becomes scalable. Doing a one-time hard fork allows us to do soft upgrades in the future.

Flexible transactions reduce the amount of changes required across the system, not just for full nodes, but especially given that many tools and wallets rely on shared libraries to create and parse transactions.

Anyone concerned about protocol stability should consider the Flexible transactions upgrade proposed in this article, as the risk of failure after the upgrade is several orders of magnitude lower than SegWit. It has no technical debt, which will allow us to better innovate in the future.

Flexible transactions have smaller transaction sizes and can provide more significant space savings than SegWit.

(Thanks to @vatten for proofreading)