The Evolution and Future of Bitcoin Mining: Are Mining Pools a Problem?

Bitcoin advocate and educator Bitcoin Mechanic delves into the various problems and challenges in Bitcoin mining, with a particular focus on the dynamic relationship between mining pools and miners. He proposes solutions to existing problems, focusing on improving miners' autonomy and the overall decentralization trend. Among them, the discussion of the StratumV2 protocol and the future development of the mining ecosystem are full of unique insights. In addition, the article points out that the latest developments in ASIC manufacturing and mining pool infrastructure provide new possibilities for decentralization in the mining field. Through a critical analysis of the existing mining model and a look at possible future improvements.

Bitcoin miners provide a vital service to the entire ecosystem. They receive rewards from the network as part of ensuring its security, which is one of the most solid and elegant designs designed by Satoshi Nakamoto and one of the most compelling aspects of Bitcoin.

However, more and more people seem to overlook that mining is not just about hashing.

Someone who participates in the whole process must run a node to get the latest state of the blockchain and then start building a new block. This includes verifying the validity of the previous block, discovering unconfirmed transactions and generally choosing the most profitable of them. Paying yourself in the generation transaction, building multiple Merkle trees of these transactions, and finally hashing to actually solve the block. The transactions in the block template will constantly change as new transactions are broadcast to the network. When someone else finds a new block, the miner must switch to building on it and dump all the transactions that are already in the blockchain to fill a new template.

Fork Activation

As you can see, doing the hashing to actually solve the block is only part of the process. Bitcoin mining ASICs can also only do hashing. In the current environment, all other aspects of mining are usually delegated to mining pools. This leads to some confusion. For example, in any discussion of soft forks activated by version bit flips within block templates, people will always mention that the process is a MASF, or "miner activated soft fork", and someone will always stand up and say that it is the sole responsibility of the mining pool, and the mining pool is not the miner. They may also point out that miners are still ultimately responsible because if they want to upgrade but their mining pool does not, they can simply switch pools. For the sake of clarity, for the rest of this article, I will only refer to those who only participate in hashing, leaving all other aspects of mining to the mining pool, as "hackers."

Back to soft forks, in the current environment where more than 99% of blocks are built by the same organization, it might be more accurate to call these "pool-activated soft forks", although no one is currently calling them that, which leads to a dangerous illusion: mining can be considered decentralized simply because of the distribution of hashrate. This claim becomes unbelievable when all hashrate is limited to a few mining pools, so the content of the Bitcoin blockchain will eventually not include anything that these few entities deem unacceptable, as well as a host of other problems.

By only performing hashing of blocks built by mining pools, Bitcoin miners are largely giving up a critical component of their role. The fact that this is not only possible, but the smoothest path, suggests that we have a systemic problem.

Mining Pools and Blockspace Markets

The impact of just doing the hashing and letting the pools do everything else goes far beyond the soft fork activation. For example, currently miners have absolutely no idea what the solved block will look like, which means miners are working under the blind trust that the block contains only the transactions they expect. But you see a blatant violation of that trust in blocks like this one, the famous block that kicked off the “Ordinal” craze. Note that the miners working on this block actually only received about $200 in Bitcoin transaction fees, while the average transaction fee for blocks on both sides of it reached about $5,000 in Bitcoin.

Block space has value, it’s part of the reason Bitcoin works in the long run. But in a world where only a handful of players can make the templates they build end up on the blockchain, these entities have a near-exclusive right to sell this space, and get outsized compensation for doing so. Do they have a duty to, and perhaps even be upfront with their miners about, the fact that they are doing this? Of course, in this case, the intention is to surprise everyone. In the future, will they forward the payments they receive for selling out-of-band block space to their hashers?

In short, while the incentives of a mining pool and its hashers are generally aligned to maximize profits, it is possible for a mining pool to sell block space for something other than regular Bitcoin transactions, and miners’ income is more limited, unless the pool chooses to be transparent and agrees to share revenue. Even if they do, validation requires permission from the pool, rather than validating funds earned from subsidies and transaction fees (also tricky when using FPPS pools, more on this later).

Further influence of mining pools as builders of Bitcoin’s centralized block template stems from a more fundamental fact — at a more basic level, there are twelve “supernodes” with their own “super memory pool”.

This has led to people dealing directly with mining pools and ignoring mempools altogether. Some argue that mempools are doomed to fail regardless. The current centralized state of template building simply speeds up the process, but is not desirable under any circumstances, and in a world where truly decentralized template building is considered feasible, it may be too pessimistic to make such an assumption. Then, if the person who purchased the block space wants it to be included in the blockchain in the same time frame, the out-of-band payment must be passed to a larger group of people. This may be more transparent and similar to how it works currently. Instead, "supernodes" are expected to be broken down into smaller parts and therefore no longer be able to provide the same guarantees.

To stray from the mining aspect of this, let’s shift our focus to how payments are currently processed.

Mining Pool Payment Model

Almost all mining pools pay their hashers via FPPS (Full Payment Per Share) or something similar. The only exception is ViaBTC which offers PPLNS (Payment Per Last N Shares) in addition to FPPS. Antpool also offers PPLNS, but hashers must give up all transaction fee income, which illustrates the point I am about to try to make. Basically, in a world that focuses on transaction fee income rather than subsidies, FPPS is not a model that works well. It is worth mentioning that the Braiins mining pool (formerly Slushpool) uses a system called "scores" that is actually very similar to PPLNS.

Why is there such a strong preference for FPPS? From the hasher's perspective, they get paid regardless of what happens on the blockchain. This aligns with the purpose of pool mining, with more consistent income. FPPS provides more consistent payments because pools pay based on projected income and settle independently of the blockchain.

This makes life very easy for miners who wish to minimize problems due to interruptions to their cash flow, but of course there are disadvantages - important ones that I wish to highlight here.

FPPS first requires mining pools to become custodians of all newly mined Bitcoins. These Bitcoins cannot be forwarded to miners for a minimum of 100 blocks after mining, so the newly mined Bitcoins cannot be spent before then, and in fact, the mined Bitcoins have nothing to do with the Bitcoins that miners ultimately receive when withdrawing from the mining pool. The risks of third-party custody should be obvious to almost everyone reading this article, so I will skip this and move on to the other issues with FPPS.

The next concern comes from a more general problem, which is that FPPS pools are an important intermediary between hashers and the network itself. We have already established that hashers have no way of knowing in advance what the block they are working on will ultimately look like, not until after it is solved. FPPS means that they now don’t even care if the block is found or not, that’s the pool’s problem. Ignoring the increased predictability of payouts (which will never happen if the pool decides to cheat the hashers), we must acknowledge the tradeoffs of doing so.

Miners who are paid directly by Bitcoin itself - which is possible in alternatives like PPLNS or of course solo mining - can expect to receive their full reward, including transaction fees. An FPPS pool can only calculate this after the fact, as there is simply no way to predict what the fees will be when determining the fees a hasher actually receives per share. Pools cannot simply assume that the fees will be some value greater than 0 and include that in the miner's income as they mine, because if the fees are lower than this, they will simply pay the miner out of their own pocket. They must periodically allocate fees and attribute them to the miners who actually have custody of the pool.

From the hasher's perspective, full trust in the mining pool is required, as verification is nearly impossible without full transparency and cooperation from the mining pool. As mentioned above, this has not been a big issue in the past, as most mining revenue came from subsidies, with transaction fees being just a few satoshis sprinkled in, but this is not and cannot be the future of growing Bitcoin mining. In the future, miners will primarily receive revenue from transaction fees, which are much harder to predict and monitor than subsidies when using mining pools.

Compared to a payment scheme like PPLNS, where hashers are subject to greater variation (and the pool’s luck becomes the hasher’s luck), we see an extreme tendency for the mining ecosystem to prioritize consistency in payments over the ability to verify what was received. To make matters even more twisted, some hashers actually prefer this approach — hoping to present themselves to government agencies as a “hashing service” that is completely divorced from Bitcoin — and some are even proud of it. This is because FPPS is so far removed from the ideal miner/pool dynamic that it’s hard to describe what hashers are actually doing again, even as “Bitcoin mining.”

In reality, the FPPS pool is a large independent miner that pays hashers to solve its blocks. They then have an internal and opaque process by which they determine how much to pay the hashers. To really drive the point home, hashers can even (in some not-too-hard-to-imagine scenarios) pay their fees in something other than Bitcoin.

Why not? If you don’t care about finding any blocks, let alone what they look like before they’re built, then why not just get an independent miner to pay you in fiat to point your ASICs at them in the currency that’s most convenient? Bitcoin isn’t always the most frictionless option, but even if it is, it’s reasonable to imagine continuing down a path where “hashing” can be performed by as many entities as you like, but it’s all done on behalf of a small group of “mining pools” whose permission the entire network needs to actually record anything to the blockchain.

Who is doing the hashing anyway?

Let’s look at this in a broader context. We’ve already mentioned that some of the bigger players want to stay as far away from Bitcoin as possible and are therefore willing to delegate as much Bitcoin-related activity as possible to their mining pools. These mining pools are open to regulation and their large amounts of hash power are perfectly happy with it.

This again introduces economic irrationality from the perspective of the network itself, in the form of the act of mining blocks that meet some arbitrary criteria. When this has happened in the past, it has not lasted long due to community backlash and the absurdity of trying to actively please a jurisdiction’s ever-changing regulatory regime without being required to do so. But the fact that this is an option of choice exposes the risks of having a centralized block template build. Will miners in one jurisdiction try to ban or refuse to process transactions from another? Are miners simply extensions of bad government or influential actors? There are specific examples of some mining pools withholding transaction fees to profit illegally, sometimes simply to comply with regulatory pressure. This again appears economically irrational from the perspective of the network.

The most extreme recent example is the 19 BTC transaction fee paid in a transaction that was ultimately discovered by F2Pool, ostensibly as a mistake. As a FPPS pool, they became custodians of these 19 BTC mining fees and chose to return them to the person who made the mistake. This perfectly demonstrates the cost of placing too large an intermediary between miners and the Bitcoin network. In a PPLNS pool, this is less likely to happen. Not because the PPLNS pool is necessarily trustless or non-custodial, but because the fee income can be monitored and verified at the exact moment a block comes in, which may be more difficult for a mining pool to attempt, as it may have already internally deposited the miner's account with their share of the mining funds earned, causing a larger backlash. Although in principle there is no difference, until you compare what would happen if a mining pool paid its miners in the coinbase/generation transaction. In that case, the money is already in the hands of the miner and the mining pool cannot intercept the fee income. So in this example, the mining pool, wanting to appear generous or fair, caused its miners to lose $500,000 in fee income, a position it should never have been in to make a decision.

Next question: 51% attacks and other attacks

This should be pretty easy to explain: by now everyone knows what a 51% attack is. However, what is far less understood (until the network gets around it) is that 51% is a requirement for this type of attack to be guaranteed and consistently successful, rather than just destructive.

In reality, any entity with more than 20% of the network can cause problems through a number of attacks, some of which are executed in the wild but rarely discussed, and I will go into detail later. But before that, we can see that the network has only two entities with a total hashrate that reliably exceeds 51%. To make matters worse, one of the largest mining pools did not carefully cover up the fact that it found another 10% of the blocks through another large mining pool that has a strategic partnership with its parent company. The fact that this farce is still going on is not convincing.

There are two common responses to this. First, people point out that miners could simply change their voting pools if they banded together to perform a 51% attack. Second, it would be crazy for any mining pool to try to do this, for the simple reason that disrupting Bitcoin would cause the price to fall, and no one invested in the ecosystem wants that to happen. The second argument ignores human history and further assumes that people will never be forced to act destructively, thereby causing disruption simply for the sake of disruption or for other nefarious purposes. It also fails to take into account that markets are not necessarily always a good indicator that there are problems with Bitcoin, see the 2017 hard fork war.

However, the first argument is more solid, assuming that miners will always switch to other pools in the event that a pool does become too large. In fact, if a pool tries to do this, reality will intervene and we will realize that the pool is not actually the miner despite building 99% of the block templates. A famous case study is Ghash.io, which caused a fatal downward spiral by exceeding 40% of the hash rate.

So, so we have shown that this is not actually a problem and miners can be trusted to switch to other pools. In reality, this assumption would be less reliable if large mining operations were shackled by red tape, but let’s at least assume that we are fairly confident that this attack is unlikely to occur.

Unfortunately, the realization that any pool’s hash power will migrate elsewhere once it passes a scary threshold has caused them to self-regulate. This doesn’t help, because they don’t have to actually maintain hash power below the threshold, they just have to give the impression that they do. This basically amounts to accepting all the hash power they can get while forwarding it to other pools to avoid alerting the world to the chaos they are able to cause.

This way, we have an unknown picture of what is happening on the network. 30% of the blocks can be found by the largest mining pool in the open and accepted by everyone, while the other 10% of the total network hashrate still points to that pool, but is secretly transferred to one or more smaller pools. The miners responsible for this 10% may not realize that it is being used in this way, and using stratumV2 is even more difficult to detect. I will explain this in detail later.

This already suboptimal situation becomes even worse when considering that this redirected hashrate could be used to harm smaller mining pools via block mining attacks.

Specifically, the attackers mostly participated in the mining process as normal users of the victimized pools. As a result, they received a portion of the rewards from any block found by the pools as expected. The rewards ultimately flowed to the attackers, who could then pay the actual miners without losing any funds. So far, the only damage done was a false impression of the mining pools' hashrate, making it appear smaller than it actually was, but smaller pools remained unharmed.

Now, the harm can happen if the attacker decides not to tell the victim pool when they find a block. This will make the victim pool look unlucky. They will appear to simply find fewer blocks than they should, and pay out rewards to more participants who are actually mining honestly, i.e. necessarily lose money, assuming they don't make up for those losses in other ways.

If an FPPS pool is attacked in this way, they must burn revenue and pay miners out of pocket to make up the difference. If it is a PPLNS, miners will wonder why they are not getting what they should be getting. Either way, a block mining attack is anti-competitive and can destroy the victim pool by giving it a bad reputation.

From the perspective of the attacking pool, let's say they represent 5% of the victim pool's hashrate. This means they still receive 95% of their expected revenue, while the pool appears to be 5% less lucky than expected. This is enough to easily take down the pool, while in a larger pool the loss of 5% of redirected hashrate will be far less significant. If it only represents 1% of the larger pool's total hashrate, then the attacker has only lost 5% of 1% of their expected rewards - 0.05%. This is a clear advantage for any malicious, sizable mining pool that is prepared to act unethically.

The smaller the mining pools, the more vulnerable they are to this attack. The larger the pools, the more likely they are to block competing smaller pools. This risk increases as large pools approach a level where their total hashrate starts to cause panic in the community, further incentivizing them to at least hide their hashrate in smaller pools, even if they don’t actually use it to attack or the attacks aren’t frequent enough for the problem to ultimately boil down to variance. Larger pools already enjoy reduced variability due to the network providing more consistent payouts, meaning they are able to operate within tighter margins and thus be able to charge their miners less in fees. From the perspective of each unattacked miner/pool, this attack means they will enjoy a lower difficulty as the Bitcoin network adjusts to fewer total blocks.

Is block holding just theoretical? Absolutely not. Even in early 2015, several mining pools were attacked in this way. Preventing such attacks is very difficult because the pool must monitor all miners and make a deliberate decision whether to kick them out of the pool and/or refuse to pay them if they are lucky enough to the point where it is statistically improbable that the pool could reasonably assume they are acting maliciously. Such attacks also incentivize mining pools to want to "know their miners" and monitor payments, which of course makes life more difficult for those who want to mine permissionlessly.

Anyway, the overall effect of all of this is that people prefer to mine with larger pools for another reason.

We have heard large miners publicly say they are abandoning smaller pools because the payouts they are receiving are not meeting their expectations.

This is highly undesirable because larger pools and the larger miners who use them are more susceptible to regulatory burdens and therefore more likely to engage in behavior that harms Bitcoin, even beyond centralization of block templates and temporary escrow of all block rewards.

Mining pools effectively become agents, imposing bureaucratic bullshit “on behalf” of their miners. Currently, the two largest mining pools require users to jump through a bunch of cumbersome steps, including those that reveal their identity, which should not and does not have to be something that anyone mining Bitcoin has to go through outside of solo mining.

On the last point about block withholding, aside from the threat of making life more difficult for smaller pools and those who wish to mine with them, I say to those who might still be tempted to dismiss this as purely theoretical, even if it has clearly happened in the past, do we think that mining pools naturally maintain a consistent and apparently tolerable size? This means that new hashrate coming online is always distributed somewhat evenly. We have to believe that a mining pool can suddenly appear, grow rapidly, and then simply stop at around the threshold required before people get upset. Do we see mining pools begging people to stop mining with them, or simply limiting account creation and kicking out miners who exceed the allowed hashrate within existing accounts? Of course not.

The more likely two scenarios are that miners are collectively self-regulating. But this is unlikely, as it is well known that mining with smaller pools now means miners earn less Bitcoin, even if the reasons I present in this article don’t fully explain why. Not to mention that every time there is a high-profile example of mass migration from one pool to another, or that the mining pools are simply misleading people about the hashrate they are pointing to.

On top of that, smaller pools have another problem: they may not find a block for days on end, while larger pools don't exceed a few hours. This is a resolution problem, and the higher your hashrate, the closer you are to the desired outcome in the short term. Unfortunately, this leads to a lower limit below which pools cannot expect to make up for their losses during bad times, at which point it becomes impossible to compete.

The two-week period between difficulty epochs means that enough blocks must be found during those two weeks so that any bad luck has a chance to be balanced out by subsequent good luck. If not, for example, if the pool's projected block rate is 1 block every 13 days, and no blocks are found before the difficulty adjustment upwards causes them to drop to a projection of 1 block every 15 days, then the previous window of time will be closed forever. In the case of a PPLNS pool, the miners will earn less than they otherwise would. In the case of an FPPS pool, the pool will burn a lot of cash and/or go bankrupt.

This means that only so many mining pools can exist, at least the way they operate today. Simply put, there can’t be hundreds, as many of them may break down in bad times, and since they have less than 1% of the network hashrate, they may not even be able to reliably find a block per day, going through periods where there could be weeks without a block. This is a limitation that Bitcoin itself imposes on us.

How do miners and mining pools communicate?

The communication protocol between miners and mining pools is Stratum (which is gradually being replaced by StratumV2). StratumV1 is both old and deeply flawed. First, all communications are transmitted in clear text. This means that the Internet Service Provider not only knows that you are mining, but also knows how much you are mining. Anyone who can eavesdrop on your network traffic can perform a man-in-the-middle attack, causing your computer and computing power to be used to mine for someone else. This has been abused in the past by unknown attackers in order to steal computing power away from the intended mining pool.

In addition to some inefficiencies, StratumV1 failed to provide a practical way for miners to build their own block templates and still mine in a pool. All of these issues were addressed in the highly attractive StratumV2 (originally called "GBT" and then "Better Hash"), which we will return to later.

Hardware/Firmware

Before we discuss the mining pool/miner dynamics, we’ll digress for a moment because this article would be incomplete if we didn’t mention that there are only two companies that manufacture ASICs on any meaningful scale, Bitmain and MicroBT. There are other companies, but virtually all hashing occurs on machines manufactured by these two companies.

This is obviously bad, and the root cause is that chip manufacturing is very difficult and therefore too centralized.

It’s beyond the scope of this article to discuss solutions, but there are efforts underway to make home mining more practical (in North America, the main issues are the need for 220-240 volts and dealing with the piercing noise). Those working on “pleb-mining” projects argue that if enough regular Bitcoin users were able to do this, they could start accounting for a significant portion of the network’s total hashrate, which would be preferable for most scaled-up mining operations, which are more susceptible to regulatory intervention.

The difficulty of this task is that firmware is closed source. Even custom firmware that can "jailbreak" ASICs is often closed source to ensure that users pay the development fee, that is, the cost of the market firmware you use is on behalf of the production team to mine.

The stock firmware on ASICs, especially Bitmain's, is a good indication of how confident they have become in their market dominance. Besides being closed source, it is clearly malicious. When launching an Antminer, you are forced to mine for them, although miners can at least prevent this by blocking connections, or installing market firmware and then paying development fees, which cannot be prevented or the miner will refuse to mine. Bitmain has been caught adding malicious backdoors to the firmware of its miners multiple times (see Antbleed), and actively works to block market firmware developers.

The fact that the stock firmware does this is shocking, and clearly highlights the urgent need for competition in the ASIC manufacturing space.

Would anyone feel comfortable if the rules of the network were enforced by closed-source Bitcoin nodes? Moreover, imagine if we all knew that these nodes could cause users to lose Bitcoin to the developers of the software, would anyone be okay with that? When it comes to mining, almost no one cares about the sovereignty of the participants. Of course, node software and ASIC firmware are not of equal importance, and we certainly pay more attention to the former, as we should, but the latter is not insignificant and is clearly being overly neglected.

Having said that, let’s move on to discussing some solutions, specifically focusing on expanding the range of possibilities for miners and improving existing models.

P2POOL

There isn't much to say about this other than it essentially decentralizes every aspect of pooled mining. While this does many desirable things on a small scale, it requires every user to download, verify, and track every other user's share, and prove to each other that they calculated everything correctly in the template. Achieving this in an adversarial environment at any scale is essentially an impossible task. Due to the fundamental nature of pooled mining, the resources required are far greater than those required to run a Bitcoin full node, not to mention making the miner's job much more complicated.

For these reasons, it is mostly ignored by most people and is only used by more technical users or idealists who understandably cannot bring themselves to use other alternative methods of mining.

STRATUMV2

This is undoubtedly the easiest problem to solve, and it provides practical solutions to many of the problems mentioned in this article.

First, by allowing encrypted communication between pools and miners, ISPs and any other entity with access to your network traffic will no longer easily realize that you are mining, or to what extent you are mining. Therefore, it also becomes impossible, or much less easy, to "MITMing" you to hash in the name of the attacker.

Secondly, and perhaps most importantly, it would also allow miners to build their own block templates, so while mining pools would still be the trusted coordinators of reward distribution and likely still be custodians of block rewards, this would represent a shift in power from mining pools to miners, which is undoubtedly a good thing.

In a world where StratumV2 is the standard, miners are keen to build their own templates, and ideally mining pools will provide incentives for miners to build their own templates, which will lead to a more powerful Bitcoin.

The community is largely united in its commitment to upgrading the mining ecosystem to StratumV2, but historically miners have generally avoided these solutions due to the additional effort (albeit minimal compared to p2pool) and lack of incentives.

There is a lot of room for improvement, both with and without StratumV2. What is needed is a mining pool that provides miners with the ability to directly host their coins while mining. This requires a mining pool (or its miners) to construct block templates in which miners' rewards are paid directly in the coinbase/generation transaction included in each block. This is impractical in practice under an FPPS system, which means that any mining pool that implements this approach will face some reluctance from miners, but those miners who make the switch will enjoy greater transparency, as the Bitcoin itself will be paid directly to them above a certain threshold, with an easily verifiable distribution of subsidy and fee income. This can be combined with mining pools, even before StratumV2, to at least let miners understand the block template built for them before solving a block, and after StratumV2 simply verify that all miners have built templates that accurately reflect the reward distribution, without all miners having to constantly do this operation for the extension.

Mining pools could also address miners’ reluctance to make their own block templates by providing incentives for them to do so, such as by charging them lower fees. It seems likely that this additional incentive may be necessary if miners are still unwilling to take on this burden once it becomes practical again.

The above suggestions would greatly improve the situation.

Many new initiatives and announcements are emerging regarding ASIC manufacturing and mining pool infrastructure, which should be a welcome development for anyone looking to ensure that mining moves toward greater decentralization.