Building a better Bitcoin fee market

Over the past year, we have seen the Bitcoin fee market develop at a rapid pace. As transaction volume continues to grow, the demand for block space for Bitcoin transactions is also growing. Currently, its transaction supply limit is still an average of 1 megabyte (MB) every 10 minutes. This article will focus on the history and current status of Bitcoin transaction fees.

In fact, the pressure of high competition for block space has degraded the user experience, forcing Bitcoin wallets to make adjustments to ensure that their users can confirm their transactions instantly.

However, we still have a long way to go to operate an optimal fee market.

History of Transaction Fees

In the first few years of Bitcoin, transaction fees were optional and they were considered a gift to miners.

Figure: Bitcoin Core fee settings

For every transaction, the wallet pays the same fee, with a default fee set by the wallet developer.

Bitcoin Core's default fee has changed several times over the past few years as Bitcoin's exchange rate has risen, from 0.01 BTC to 0.0005 BTC to 0.0001 BTC . In addition, there are some "priority transactions" that allow users to send transactions without paying fees, such as if their inputs are old enough and the amount is high enough, although miners have gradually abolished this rule.

Over the past few years, we’ve learned that hard-coded static transaction fees are terrible for several reasons:

1. From the miners’ perspective, they only have 1 MB of block space, but they want to insert as many transactions as possible so they can collect more fees. For example, a 200-byte transaction with a fee of 0.0001 BTC will be prioritized over a 1,000-byte transaction with a fee of 0.0001 BTC because the former can collect five times as much fees as the latter for the same space.

2. From a user perspective, if you always set a static fee, you might end up creating a large volume of transactions (due to spending too many low-value inputs) and transactions with very low fee rates that may never get confirmed.

3. Static fee wallets are unable to adapt to rapidly changing market conditions, causing users to either overpay or underpay when broadcasting transactions. The former does not make transactions confirm faster, while the latter can lead to long confirmation times as miners favor more profitable transactions.

In November 2010, Bitcoin 0.3.15 included a change to calculate fees based on the amount of transaction data, but not every wallet followed suit, and many users continued to blindly set the same fee for every transaction. This was not a problem at first, but when Bitcoin block space began to become congested, the situation changed because miners would confirm any valid transaction that was successfully communicated to them.

As Bitcoin blocks have become increasingly congested since 2015, it became clear that a dynamic fee algorithm would be the best approach as it would respond to ongoing changes on the network.

Bitcoin Core started using dynamic fees after the 0.10版本client was released (February 2015), and Alex Morcos has been steadily improving the dynamic fee algorithm since then. The fee estimation algorithm in the Core client is quite complex, and you can view its code and its English description here.

The rise of fee markets

Antoine Le Calvez, developer of p2sh.info, provides a dynamic and static analysis of Bitcoin transaction fee history.

Here are the results of the analysis over the past two years:

Last fall, as the network was stress-tested and attacked, we saw a jump in the use of dynamic fees.

However, I suspect this was not caused by ordinary users switching to dynamic fee wallets, but rather the attackers themselves deliberately setting higher than most static transaction fees.

During the attack, some clever users adjusted their hard-coded fees, but these transactions only accounted for a small portion of the total transactions. We can also see that the use of dynamic fees has seen another jump in March 2016, most likely due to the deployment of dynamic fees in Blockchain's new wallet a month ago.

Rusty Russell did a great analysis of the dynamic fee market in October 2015, showing that more transactions are using dynamically calculated fees, and that the average value of each transaction is increasing because smaller payments are going off-chain.

AJ Towns followed up Rusty’s post with a deeper analysis, identifying eight distinct fee market phases in Bitcoin’s history:

Towns went on to publish his findings in a second blog post, drawing several conclusions about the impacts of the emerging fee market on users.

They are:

1. A large number of wallets use dynamically calculated fees, which are calculated on a per-byte basis.

2. Many wallets still don't calculate fees dynamically, or even on a per-byte basis.

3. As adoption of dynamic fee wallets grows, market-driven fees will grow further.

4. Paying excessively high market rates will not make your transaction confirmed faster.

5. There have been two wallet incidents due to dynamic fees, and a third fee incident is coming soon.

6. Wallets that automatically calculate their fees will pay significantly lower rates than those that do not.

I’ve been tracking Bitcoin Core’s fee estimates via Statoshi, and you can see below that transaction fees have tripled over the past six months as competition for block space continues to grow.

Calvez also provides a dashboard of all publicly available fee estimation APIs, as well as their historical estimation data:

Interestingly, it appears that fee estimates from 21, BitGo, and Blocktrail are the most responsive to changing market conditions, while fee estimates from BitPay, Blockchain, and BlockCypher are less volatile.

However, I would be remiss if I recommended widespread use of dynamically calculated fees without pointing out the dangers involved.

As the old saying goes, every solution brings new problems, and dynamic transaction fees are no exception.

Dynamic Difficulty

Dynamic cost estimation will never be perfect because it is an attempt to predict the future.

Danish physicist Niels Bohr once quipped that “prediction is very difficult, especially about the future.” If dynamic fee estimation methods fail to correctly predict future market fees, users will fall into a “fee trap,” as Dr. Washington Sanchez of OpenBazaar points out.

The fee trap phenomenon described above occurs if you broadcast a transaction with a fee that is perfectly in line with current market conditions, after which many other high-fee transactions are broadcast by other users, which pushes your transaction to the back of the miners’ priority queue. The problem is that without Replace By Fee, you cannot update your “bid” to meet the new market conditions.

Bugs in fee estimation algorithms can also cause disruption to the fee market. For example, recently someone mistakenly created a transaction with a transaction fee of 300 BTC. This is bad for the unfortunate user, who may have confused "fees" with "transaction value" due to some bad coding software, but it can also have knock-on effects:

This is just a guess, but it now appears that the fee estimation algorithm used by BlockCypher's "1 to 2 block target" is based on a weighted moving average of the fees of blocks over the past two days. As a result, when the 300 BTC in fees were processed, the recent average fees, as well as their estimated peak, soared 800%.

This doesn’t mean I’m attacking BlockCypher, we’ve seen a lot of wallets have issues with transaction fees. Even Bitcoin Core is not immune, I recently observed an unexpected spike in fee estimates for the Bitcoin Core 0.12.0 client:

I noticed that on February 27th, Statoshi.info's "two block target" fee estimate spiked from 44 satoshis per byte to 112 satoshis per byte for no apparent reason, and my other bitcoin nodes did not report the same spike.

Some concerns

While there are numerous challenges facing Bitcoin developers, we should also be aware of the impact of the fee market as a whole.

I worry about several potential scenarios that could drive expense ratios up faster than necessary:

1. Poorly coded dynamic fee algorithms can create a feedback loop where transactions continue to flood the network without a response to the mempool backlog, causing fee rates to rise indefinitely. If most transactions are manually initiated by humans who are aware of the fee changes and decide whether to respond to them, then this will not be a big problem. However, if many transactions are automatically generated without any human decision-making, and these transactions are created by algorithms without any sanity checks on fee payments, the market may get out of control until the engineers who wrote the algorithms realize the problem and make changes to fix it.

2. Frustrated users (or lazy developers) still relying on hardcoded fees may keep manually updating their wallet settings until their transactions start confirming, which could result in significantly higher fee payments, but ultimately forces other static fee users to do the same so they can stay competitive. Dynamic fee users are thus forced upwards. I have seen this happen, but not to the point where it significantly affects dynamic fees.

One recent development that has not received much attention despite its potential to impact the fee market is the secondary miner fee market.

This could take the form of special customer rewards programs, such as BTCC’s BlockPriority service, or it could take the form of private prepaid block space purchases.

Quoting the BTCC press release:

“BlockPriority prioritizes all BTCC customers’ transactions, including “0” fee transactions.”

This shouldn’t be attributed to a problem with Bitcoin Core’s fee estimation algorithm, but from an economics perspective it probably makes sense for mining pools to sell prepaid block space contracts as it gives them a new predictable revenue stream.

However, my warning to mining pools is this: if it becomes a prominent part of the transactions you mine, you will not be able to hide this activity. There are many wallet engineers who detect this type of behavior (like me) and if something goes wrong we will make these transactions public. I guess a mining pool that is caught engaging in this behavior will not be so lucky if individual miners discover that the pool has mined lower fee transactions without sharing the profits of the private block space contract.

This could cause hashrate holders to transfer their hashrate to what they perceive as more profitable mining pools.

Segregated Witness may also have an impact on the fee market. It will provide a 75% fee discount in an attempt to rebalance the cost of creating and consuming unspent transaction outputs ( UTXO ).

Moving forward

There are still many wallets and Bitcoin services that have not implemented dynamic fees. You know which category you fall into, and your inaction may result in a poor experience for some of your users. If you have not implemented dynamic fees, then every transaction you broadcast will fall into one of two categories:

1. You're wasting money but not getting your transaction confirmed any faster.

2. You pay too little in fees and your transaction gets stuck, resulting in a poor user experience.

It is extremely unlikely that a transaction with a hard-coded fee would be broadcasted and happen to match current market conditions.

Wallet developers should add safety mechanisms, such as micro-level check thresholds and macro-level circuit breakers, to prevent users from shooting themselves in the foot.

The minimum fee threshold set by your wallet should be higher than the minimum relay fee. By looking at the lowest fee transactions entering blocks today, we can clearly see that if you pay less than 5 satoshis per byte, you will wait a long time for it.

I would encourage all Bitcoin wallets to make their fee estimates accessible via a public API. I think the fee estimation algorithms themselves will become a point of competition, they are usually kept closed source, but if they were public it would be easier to observe unusual activity. Maybe someone would even create a "Bitcoin average" aggregator of fee estimates.

While we're all discussing transaction fees, Bitcoin users should stop saying they paid "X cents" or "Y bits" in fees on their transactions, because this type of statement doesn't speak to the scale of transaction sizes and is absurd. We should be discussing using a standardized fee rate, preferably in satoshis per byte. Obviously, "20 satoshis per byte" is easier to write and track than "20,000 satoshis per kilobyte."

The Bitcoin fee market has had a bumpy ride so far.

Initially, fees were set manually by developers, then they became manually set by users, and now we’re at a more confusing and potentially dangerous point where developers are taking a more active role in the economic transformation of this emerging market.