Technical Detailed Explanation of Chia Network Mining: How to Participate? What are the Profits?

Original title: "Chia Mining Technical Analysis and Suggestions"

Written by: Henry, Researcher at Empower Labs

First look at Chia network mining

I believe everyone is already very familiar with the mining of Bitcoin/Ethereum and other currencies. The principle behind mining is the consensus problem. Many miners participate in mining to obtain virtual currency and then make a profit. But in essence, the purpose of mining is to maintain an entire distributed computing network and maintain consensus. The virtual currency allocated to miners is just a reward for this process.

According to Satoshi Nakamoto's idea, a powerful distributed network can be built by using the idle computing power of the CPU in people's homes. However, with the emergence of ASIC mining machines, this computing power can be easily obtained by dedicated mining machines, thus becoming a game of capital. In addition, energy prices also significantly affect the computing power of Bitcoin.

According to Chia's white paper, the vision of the Chia project is "the green currency of the digital world". At the beginning of its design, the goal of selecting the computing power base was " a widely distributed, ASIC-resistant, oversupplied commodity " and had nothing to do with electricity prices - that is, hard disk space. (Broadly speaking, it also includes many forms of computer storage facilities, as long as they can provide a minimum capacity of 600MiB - this is the smallest plot file that the Chia mining software can provide, but it is very uneconomical and is no longer supported in the latest implementation, but it can still be used as a tool to test performance).

As of the time of writing this article, based on the Chia network's computing power, its total power consumption is estimated to be less than 1000KW, which is a huge difference compared to BTC.

They are both mining, so what makes Chia different?

In short, the mining process of Chia is like a lottery. Miners participating in mining should prepare files called plots in advance and wait for the broadcast of the next block. The contents of these files and the contents of the broadcast are hash values. The lucky person whose hash value is closest to the one in the broadcast, congratulations, he won the prize and got the XCH token.

According to the design of the Chia network, in the first three years after the mainnet is launched, the output is 64 XCH every 10 minutes, shared by 32 miners, that is, each miner has 4,608 chances to win every day, and then decreases in batches.

The essence of preparing a plot file is to calculate a huge amount of hash values ​​and store them on disk. The official wallet program can complete this calculation process. The wallet client also comes with a command line interface program (CLI) that can be used flexibly by miners to write scripts.

In order to increase the probability of winning, miners should prepare as many plots as possible. The computing power unit of the Chia network is the storage unit, i.e., MB/GB/TB/PB, etc. Note that the official unit used is a lowercase i, such as GiB, which means 1024 as the carry, not 1000 as used by hard disk manufacturers. So when you see the default parameters saying that it takes 239GiB of cache space to generate a plot, and you think that a 1T hard disk can run 4 plots at the same time, don't be too happy, because 239GiB is approximately equal to 251GB, and a 1T SSD actually has only more than 930GB of actual capacity. Of course, after practical exploration, a 1T SSD can also run 4 plots at the same time, but it requires a little skill in parameter setting.

Friends who have learned about Filecoin mining may think that Chia mining is very similar, which is true to some extent. Both require a large amount of storage devices and need to be prepared before the official mining to generate effective computing power. In Filecoin, this step is called data filling, and in Chia it is called Plot, or the more popular name - P disk. However, Filecoin mining has extremely high requirements for equipment. It is not just about piling up storage devices. It is far beyond the scope of public use. There are also very complex and expensive pre-mortgage requirements. Chia's requirements are very friendly to the people. There is no mortgage, no penalty mechanism, and it can be operated with a personal PC. This is the reason why Chia quickly became popular all over the country and successfully broke the circle.

However, the logic of using storage space by Filecoin and Chia is different. Filecoin mining is essentially an incentive for providing distributed storage, while Chia purely uses this special mechanism to maintain network consensus.

A simplified explanation of the Chia mining process

Since Chia became popular in mid-April, many impatient miners have already or are preparing to buy hard drives to get started. From the existing community discussions, a large number of computing resources have not been fully utilized. Therefore, it is necessary to understand the key points of Chia mining and optimize them according to your own conditions in order to achieve efficient mining.

Chia mining is not just about having a large hard drive. There are two essential parts: making a plot file (usually referred to as a P-map, because the official translation of plot is "cultivated land") and mining (official translation is "cultivation") .

For Chia, mining is actually a very easy task. What is important is capacity, not speed. Even the requirements for the mining host are so low that there is almost no requirement. The official even recommends using a Raspberry Pi worth tens of dollars plus several USB Hubs to drive up to 1PB of computing power. After all, the mining process is just waiting for a string of numbers to be broadcasted on the network, and then comparing it with the hash value in the hard disk. The hash comparison operation is extremely fast, and there is no need to read terabytes of data into the memory for any operation.

This is indeed in line with Chia's starting point - green. Anyone can join the Chia network using idle storage capacity. Mining consumes almost no additional energy. As long as the computer is turned on and connected to the network, you can enjoy 4,608 draws every day, and it is permanent. This is wonderful.

But in fact, the process of making a plot is still quite resource-intensive, but what is consumed is not electricity, but SSD. High-speed and large-capacity solid-state drives have become real consumables. This is thanks to (whether it is credit or blame, let's see the impact on the storage industry) the computing mechanism designed by Chia. In order to ultimately achieve the goal of providing Proof of Space and being able to retrieve quickly enough, the process of making a plot by Chia is actually divided into four steps:

1. 7 hash tables are generated by calculation, and the main algorithm used is forward propagation. This step requires a lot of calculations and is the most CPU-intensive. In fact, this step has generated all the data that is sufficient to support Proof of Space, but the efficiency is not good, so subsequent steps are needed.

2. Use the back propagation algorithm to clean up the 7 hash tables above, remove unnecessary hash values, and sort the tables. This step is also CPU-intensive, and the memory and cache usage peaked in versions 1.03 and earlier, but memory usage has decreased since 1.04.

3. Compress the results of the previous step and merge most of the tables. From this step on, the CPU consumption is reduced to a lower level, the memory usage remains high, but the cache disk usage begins to gradually decrease.

4. Compress the remaining tables into the final file format and transfer the files from the cache disk to the target location. This step is the only IO operation on the target disk.

It should be pointed out that Chia's plot file production, mining, node maintenance and other work are all completed by its wallet software. The software is still being improved and iterates quickly. In the past few minor versions, it has made great improvements in cache usage, speed calculation and other aspects. Some netizens on the Reddit forum recorded the resource usage of different versions of the software (1, 2) in the process of making plots. By comparison, it can be found that version 1.04 has reduced the cache usage by nearly half compared to earlier versions. It is recommended to always use the latest version of the official github.

How to dig?

For most miners, the first problem to be solved is the first step, that is, how to make plot files quickly and economically.

This process requires the use of the Chia wallet software. The original intention of Chia's design is actually very simple, and most of the obscure parameters have been carefully selected during the development process to ensure that they are already the optimal values. The rest is just the time difference caused by the performance difference of each computer. But it is precisely this performance difference that constitutes a completely different ecology. On the one hand, amateur players want to buy a few hard drives to test the waters, and on the other hand, the big crocodiles who are afraid of missing the wind are eagerly asking "What configuration should I buy with 10 million funds?" Behind this, everyone is concerned about the same question: How many coins can be mined in the first mining window period when the computing power of the entire network is low? After all, according to the statistics maintained by the Chia community, for the same plot process, with a computer configuration of less than 10,000 yuan, some people can complete it within 4 hours, while others need more than 24 hours. Such a computing power difference will naturally be reflected in the large difference in the number of rewards obtained.

In this regard, our suggestions are as follows.

If you are an amateur player , mainly relying on one or two idle computers, and at most buying a few extra hard disks, then you don't need to worry about it and can just use the default parameters. If you are willing to spend a little more time researching, it is best to test the number of concurrency several times to see which concurrency is most effective. Going further, you can observe the CPU, memory, and disk IO usage during the plot process. If any resource has a bottleneck, slightly reduce its allocation, and vice versa.

A few very useful tips:

• Based on the Reddit user record chart cited in the previous section, the utilization rate of various resources is different at different stages, which means that instead of pursuing the number of concurrency, it is better to scientifically allocate the delay time so that various resources are always at a higher utilization level. Let several concurrent tasks delay each other for a while before starting, which not only means that the CPU and memory can be better utilized. At the beginning of the release of version 1.04 of the software, netizens successfully tested and successfully ran four plots concurrently on a 1T SSD cache by setting a delay of 6 minutes. Considering that it is common to use ordinary mechanical hard disks as target disks in reality, and the IO performance of mechanical hard disks is weak, the delay time can also prevent multiple large files from competing for the IO of mechanical hard disks at the same time.

• Although the process of making a plot is CPU and memory intensive, over-allocating CPU cores and memory does not significantly improve performance. If you have a computer with 16G memory, 8 cores and 16 threads, there is no need to allocate all the memory and threads to a plot task.

• For individual miners, referring to the current computing power of the entire network, it is already very unrealistic to mine by relying solely on wallet software (commonly known as solo mining). Of course, as mentioned earlier, Chia mining is similar to lottery behavior, and even if there is only one bet, it is theoretically possible to win a prize. But from a more realistic perspective, individual miners are advised to join a mining pool to obtain more predictable output.

One thing to keep in mind : the lifespan of a civilian-grade SSD for home computers is generally several hundred TB of write and erase volume. According to the official Chia documentation, for every 0.1TB default parameter plot file, the actual write and erase volume is about 2TB, or 20:1. This means that a civilian SSD with a write and erase lifespan of 500TB can only produce 25TB of computing power at most before it dies. *This is also the reason why the official strongly recommends not to plot on the SSD built into a laptop.

If you are an ambitious new miner entering the market , our suggestion is to purchase some high-performance machines specifically for producing plot files, and equip a small number of low-power machines specifically for mining. In order to make efficient use of funds, it is not necessary to purchase high-performance complete machines, and it is best to build them on demand. Assuming there are no restrictions on power consumption, size, noise, etc., it is ideal to use a full-size motherboard to build an open host. Empower Labs believes that for a considerable period of time, it will be more popular to invest tens of thousands to hundreds of thousands of yuan to build a small Chia mining farm. Based on Empower Labs' analysis and cross-verification with industry insiders, the following suggestions are made:

• The core resources needed to make a plot are CPU, memory, and SSD, which need to be as balanced as possible.

• In terms of CPU, due to the intensive operations in the first and second stages, the CPU needs to have the highest possible main frequency. Secondly, in order to perform multiple concurrent tasks at the same time, the number of cores also needs to be large. Some miners chose second-hand multi-channel E5 systems but found that the performance was not satisfactory. The reason is that although there are many cores, the main frequency is not high.

• In terms of memory, the plot process does not consume much memory, and it has been decreasing with the iteration of software versions. According to actual observations, in version 1.03, a single plot process occupies about 4.6GB of memory, while versions 1.04 and 1.05 will not exceed 3.5GB at most. If there is a small interval between concurrent tasks, the memory usage will not reach the peak at the same time. If set reasonably, it is completely feasible to use the number of concurrent tasks * 3G or even less . At present, it has not been observed that the frequency of DDR memory will have a significant impact, so there is no need to deliberately pursue high frequency.

• The difference in SSD is quite significant. Enterprise-level large-capacity SSDs should be preferred, taking into account both performance and lifespan. The lifespan of enterprise-level SSDs is often measured in PB, which can produce more plots faster and at a lower amortized cost. Currently, the best-performing SSDs mostly use the U.2 interface, rather than the M.2 and SATA interfaces commonly found on civilian-grade motherboards, which also limits the choice of motherboards. Of course, choosing an adapter card that converts the U.2 to the PCIE 16X interface is also a reasonable option.

• In addition to SSD, the SAS hard disk interface commonly used on server motherboards is also a good choice, especially when you can obtain sufficient SAS hard disks. Using RAID cards to form a disk array has been proven to be better than a single SSD.

• In terms of operating system, Windows is not a must. Linux systems such as Mac and Ubuntu/CentOS take up fewer resources themselves, and Chia official wallets already support them. If you plan to carry out long-term uninterrupted production, it is best to use the command line interface (CLI) instead of the graphical interface plot program. The graphical interface of the Chia wallet actually relies on calling the command line program to implement plot. Taking Windows as an example, the command line program is located in C:\Users_XXXXXXXXX_\AppData\Local\chia-blockchain\app-1.0.x\resources\app.asar.unpacked\daemon\chia.exe. Among them, _XXXXXXXX_ is your current Windows user name. At present, the wallet GUI program often becomes unstable, and the wallet program itself is responsible for the synchronization task of the full node, which also consumes CPU. Therefore, CLI is a better choice for long-term mining.

Information revealed on the official website and in community discussions

After in-depth research on the official website documents and referring to a large number of discussions in the Chia community at home and abroad, it was found that some valuable information is not yet widely known.

About the -r option

The -r parameter is the parameter in the chia command line program that specifies the maximum number of CPU threads. In many discussions, miners tend to think that bigger is better. However, by digging deeper, we found that this is not the case.

According to Chia's Github discussion, the -r parameter only works in the first stage of the plot, when generating hash values . In theory, if -r 6 is set, the best result is that all 6 CPU threads are 100% involved in the work, and the total CPU number in this stage should be 600%. But in reality, the CPU participation is much less than this.

In our experiment, we used the i7-8700 CPU and set -r 6 to run a separate plot. The following output can be seen from the chia.exe log:

Starting phase 1/4: Forward Propagation into tmp files... Mon Apr 19 17:22:22 2021

Computing table 1

F1 complete, time: 172.346 seconds. CPU (183.87%) Mon Apr 19 17:25:15 2021

Computing table 2

Forward propagation table time: 964.964 seconds. CPU (312.470%) Mon Apr 19 17:41:20 2021

Computing table 3

Forward propagation table time: 1179.445 seconds. CPU (281.610%) Mon Apr 19 18:00:59 2021

... ...

The log shows that during the calculation of the 7 tables, the total CPU usage was less than half of the expected level. After many experiments, increasing the -r value can increase the total CPU usage to a certain extent, but not significantly. Therefore, we believe that the CPU frequency is the key factor affecting the plot speed, not the number of cores.

Of course, a more efficient approach is to run multiple plot queues in parallel. The significance of the above problems lies in that, first, the -r parameter is not exclusive, and second, the -r parameter has a short period of effect. The best approach is to grasp the time consumed in the first stage through experiments, and avoid it by setting intervals, so that each parallel task can always use a larger r value to monopolize the CPU as much as possible when executing the first stage.

About the -b option

The -b parameter specifies the maximum memory usage. Most miners think that the larger the better, but this is not the case. Compared with 1.04 and earlier versions, the peak memory usage during the plot process changed from the second stage to the first stage, which was caused by the algorithm improvement of the Chia team.

Now, the default parameter -b 3390 has been repeatedly optimized to ensure that the most efficient uniform sorting algorithm is always used during the sorting process. If this parameter is lowered, plotting can still be performed, but it may fall back to the relatively inefficient quick sorting algorithm. From the following log, we can clearly see this record. When the amount of available memory is less than qs_min, it will cause a complete failure.

Forward propagation table time: 2086.329 seconds. CPU (217.460%) Fri Apr 23 19:04:01 2021

Computing table 4

Bucket 0 uniform sort. Ram: 5.786GiB, u_sort min: 3.250GiB, qs min: 0.813GiB.

Bucket 1 uniform sort. Ram: 5.786GiB, u_sort min: 3.250GiB, qs min: 0.813GiB.

Bucket 2 uniform sort. Ram: 5.786GiB, u_sort min: 1.625GiB, qs min: 0.812GiB.

Bucket 3 uniform sort. Ram: 5.786GiB, u_sort min: 1.625GiB, qs min: 0.812GiB.

Bucket 4 uniform sort. Ram: 5.786GiB, u_sort min: 3.250GiB, qs min: 0.813GiB.

Bucket 5 uniform sort. Ram: 5.786GiB, u_sort min: 3.250GiB, qs min: 0.813GiB.

Bucket 6 uniform sort. Ram: 5.786GiB, u_sort min: 3.250GiB, qs min: 0.813GiB.

After observation, among the seven tables processed, the fourth and fifth tables use the most memory. The minimum memory threshold to achieve full uniform sorting is 3.250 GiB, which is exactly the effect that -b 3390 is intended to ensure.

If you increase the memory allocation, judging from the logs, the actual memory usage will increase slightly, and the logs always show uniform sorting, which is already the best algorithm, but there is no obvious effect on time reduction.

This means that even if there are up to 8 tasks running in parallel, a machine with 32G of memory is fully capable of handling the task.

About the -k option

In community discussions, the k value has always been a mystery. According to official documents, people never need k>32, unless they want to show off or there is only a little space left on the hard drive and you want to fill it up. But the Chinese community discussion does not seem to pay attention to it. Most people think that a larger k will bring a higher chance of winning, but it will increase the time to make the plot too much, which is not cost-effective. This is not the case at all.

The size of the K value directly affects the size of the plot file, but no matter how many k is equal to, the plot file produced is one vote. That's right, one file one vote. In fact, from the lottery process, it can be judged that if the content of T files needs to be read once to determine the winning eligibility, 10 minutes is not enough. Therefore, the lottery process is obviously just comparing file hashes.

Before the mainnet was launched, k could use a smaller value, for example, when k=25, a plot file could be created in a few minutes. This means that people with powerful computers can easily leave others behind and quickly get a large number of votes. This is why the official set the minimum allowed k=32 a few months before the mainnet was launched. According to the official plan, if Moore's Law continues to work in the future, the time required to create plot files will continue to decrease. It is expected that k=32 will no longer be applicable in 10 years, and the official will cancel the lottery qualification of k=32, and all k=32 files will be invalidated.

This also reminds miners who are prepared to continue to invest for a long time that at some point in the future, all computing power will need to be remade. When this time comes depends on the speed of computer hardware development. Perhaps this will be an important new factor in promoting Moore's Law.

Pain points of Chia mining

Chia's popularity did not happen overnight. In fact, the Chia project was founded in 2017, the test network was launched in 2019, and the main network was just launched on March 19 this year. As a star project with a long history, the main network is online, it will soon be listed on exchanges, and Chia's entity company plans to go public in the short term (especially with Coinbase's example). The significant impact of the explosion is the price increase and collective out-of-stock of hard disk devices, which also brought widespread controversy to the Chia project.

There is no doubt that the green concept designed by Chia has greatly reduced the threshold for participation in this blockchain project. For this reason, the problems we mentioned above are all about general hardware and some software parameter settings that are not very complicated. However, as a miner, the biggest difficulty at present is that it is impossible to obtain enough low-cost hard disks to build computing power. We learned from hard disk channel dealers that everyone from hard disk manufacturers to channel dealers was also deeply shocked by this. There is a certain degree of hoarding in the current hard disk market. But hard disk manufacturers and general generation are not worried about this. The global hard disk production capacity is as high as thousands of EB (1EB=1000PB), which far exceeds the current computing power used for Chia mining. The supply and price situation will gradually become clear in the future. Therefore, it is not recommended for miners to buy high-priced hard disks, and they should be especially careful to avoid substandard and second-hand hard disks.

Another potential problem is that although the prepared plot file is permanent computing power (currently the expectation for permanence should be about ten years) and can be stored on a cheap mechanical hard disk, the mechanical hard disk also has a lifespan. How to deal with the loss of hard disks caused by natural disasters and man-made disasters has not yet entered the consideration of most miners, but as time goes by, more and more miners will face this challenge. At present, the solution to this challenge seems to be to continuously update storage facilities, which has become a non-existent or very uneconomical option under the current supply of hard disks.

Expected Profits from Chia Mining

The development speed of Chia has long exceeded the expectations of many people, including the Chia team. When the white paper was released, the Chia team expected the computing power to reach 40~60PB when the main network was launched. However, on March 19, the second day after the launch, the computing power test showed that it had reached 120PB, and it has been rising exponentially since then, even exceeding the speed of most people producing plot files. This is not a good thing for miners.

As of the date of this article, the total storage of the Chia network has reached 900PB. This means that an investment of 1T of computing power can only produce an average of 0.06 Chia per day, and the computing power is rising sharply every day. According to the current equipment prices, the cost per T of self-built computing power mining has reached about 40-50 US dollars, excluding electricity and operation and maintenance expenses. Although the payback period is very short according to the current market futures prices and the computing power of the entire network. However, the computing power of the entire network is rising rapidly, and the average output per T of computing power formed after 30 days may be only 1/5 of the current one or even less. If the market enters a downward cycle in a few months, it is likely that the payback period of the newly added computing power will be extended to one year or even several years.

Summarize

The threshold for Chia mining is low enough. For home computer users, even if they do not make any additional investment, just using existing hard disk resources to create a few files and try to participate in network consensus is also a good experience. At the same time, you can also get a chance to win a lottery. If you can win a prize once a year, you can earn hundreds of dollars, which is also quite good. A large number of new users have learned about digital currency because of Chia, and they can have a more intuitive understanding of blockchain technology while participating in mining. Some of them will definitely choose to further explore this world. This is what we think is the biggest contribution that Chia mining has brought to the industry.

For professionals who are interested in participating in Chia mining, three factors need to be evaluated before entering the market. 1. The future price trend of XCH, 2. The growth curve of the computing power of the entire network, 3. The equipment purchase price, cycle and usage arrangement. It is recommended to take into account all possible variables of these three factors, calculate the upper limit of risk, and then arrange the funds to enter the market reasonably.

Finally, it is necessary to remind you that most of the mining machines currently on sale are used for both storage and mining, but a large number of resources will be idle after the P disk period ends. If you do not make reasonable plans, the final result may be that you buy a mining machine with a large amount of memory, CPU and other resources, but it is only used for P disk in the first month, and the rest of the life cycle is doing a job that a Raspberry Pi worth tens of dollars can do. As people gradually realize the characteristics of Chia mining, P disk work will gradually separate from mining work in the future, forming a more reasonable industrial collaboration.