The underlying logic of Filecoin mining

1/ As the saying goes: In the early years of Huashan Sect, there was no distinction between the Sword Sect and the Qi Sect. If we look at it from a broader perspective, the Internet is the Sword Sect and blockchain is the Qi Sect. If we only look at the Internet, commercial applications are the sword and digital infrastructure is the Qi. Similarly, if we only look at blockchain, mining infrastructure is the sword and application implementation is the Qi.

2/ Bitcoin is the founder of the blockchain mining sword sect. The key is to be fast and invincible. Even if the internal strength is mediocre, it can still defeat the enemy. Bitcoin mining machines and mining pools have created countless myths of getting rich overnight, and there is no need to argue.

3/ Cloud storage is the culmination of 20 years of development in the field of computer engineering. The Google File System, a secret recipe released by Google in 2003, established the ideological foundation of the current cloud storage system: cluster architecture, metadata and data separation, and ordinary commercial servers. Since then, GFS and subsequent Colossus have become references for the industry, giving birth to the Hadoop ecosystem and a large number of cloud storage vendors. At the same time, the practice of ordinary commercial servers has also set off a wave of de-IOE in China.

4/ Unlike the Internet's Web services, Filecoin is a blockchain + storage project, with a relatively pure business, which belongs to the category of infrastructure. Filecoin mining is actually storage mining, which requires the combination of two martial arts: Bitcoin's sword and cloud storage's qi.

6/ Filecoin mining is about moving the sword with Qi, and the unity of Qi and sword. Therefore, it is self-evident why Filecoin officials have been pushing the cluster version and outsourcing GPUs. Those who imitate the Bitcoin Jianzong model and go the mining machine route can be said to be using the imperial sword of the previous dynasty to kill the officials of the current dynasty, not to mention any effect. To clarify the source, I will disclose the Hongjun Nine Styles, the key to Filecoin mining.

7/ Hongjun Yishi: Reduce the total cost of ownership (TCO). Currently, the most discussed topic in Filecoin mining is ranking and input-output ratio (or return on investment ROI). Ranking of course says it all, and ROI refers to the ratio of income to investment. Those who talk about ranking are also talking about ROI, but these discussions generally stay on Capex investment, rather than discussing the total cost of ownership (TCO).

Why is the cloud model successful? Because the cloud model reduces TCO in all aspects. Should Filecoin's storage adopt the cloud model? The answer is obvious, why not stand on the shoulders of giants? There are many cloud models, self-built data centers, clouds, and a mixture of the two. The industry's practice results show that the hybrid model has the lowest cost (as shown in the following figure, the TCO comparison of the three deployment scenarios). In comparison, Filecoin mining has the following modes: mining machines, data center cluster mining pools, cloud cluster mining pools, and a mixture of the three. Similarly, the hybrid mode Filecoin mining pool has the lowest TCO cost.

Adopting a hybrid deployment model is just the first step. TCO includes the sum of all direct and indirect costs of building infrastructure, including all Capex, software development, monitoring, maintenance and support (Opex), operations, data centers, networks, downtime, sustainability, agility, etc. Refer to Google SRE's practical experience: Google's design experience in reliability engineering.

For Filecoin storage mining, you can refer to Google's experience, pay attention to the cost of data persistence and availability (storage market), and the cost of providing data services (retrieval market). If you keep the disk busy (stationary heads are also sunk costs), you will minimize the total storage TCO. Refer to this article: Google Colossus File System Design Experience.

Figuring out the TCO of infrastructure is not an easy task. TCO includes not only servers, storage, network and IT labor, but also annual capital costs and the impact of Moore's Law on future prices. Major cloud vendors have their own TCO calculators, so what about you?

8/ Hongjun Second Style: Maintain Sustainability. Sustainability refers to the ability of infrastructure to adapt to the evolving needs of the business. High utilization will reduce investment, but it will limit sustainability and have a negative impact on innovation and business growth.

Sustainability from the perspective of the cloud model. Most companies have hardware utilization rates well below 20% because the demand for resources to handle business peaks is too high. The cloud model reduces costs through economies of scale and provides services with elastic expansion, while users rent hardware, software, power supplies and other equipment on an hourly basis at a higher utilization rate. This is the scale effect and pay-as-you-go model of cloud computing. The blue line in the figure below represents the investment in purchasing hardware, which is often made on a regular basis. The red arrow represents the cost of excess capacity that the company must bear, and the red shaded area represents the situation where resource allocation is too low to meet business needs.

The value of this sustainability is difficult to calculate, so many people tend to ignore it, which would be a huge mistake. For Filecoin, asking a few questions can help you understand the sustainability of the mining system:

• How does the storage capacity output by the mining system change over time?

• How well does the mining system adapt to changes?

• How does the relative value of gains change as the computing power of the entire network changes?

• As the computing power of the entire network continues to rise, will the existing investment become a sunk cost?

• How do I upgrade? How much does it cost?

• Can mining rigs have storage attached? Can data be relocated?

9/ Hongjun Three Styles: Cluster Architecture. The cloud storage model, or the cluster architecture and distributed architecture of storage, is the idea that Filecoin mining should review and learn from the most. Turing Award winner Jim Gray gave two conclusions (computing close to data and combining data from multiple distributed sites) and two warnings in his article "Distributed Computing Economics" (Beowulf clusters have completely different network economics and Moore's Law will strengthen the two conclusions). Pat Helland also summarized the economics of data centers: data centers can provide computing, storage and network services at a relatively cost-effective price through measures such as cheap electricity, proximity to backbone networks, and shared management. Therefore, for the economics of Filecoin mining, what needs to be learned is: building a cluster or outsourcing to a cluster to achieve the separation of computing (seal) and storage, and the separation of mining (chain) and storage. For most people, managing their own mining pool or data center is unlikely to be strategically significant, and outsourcing or investment is also a good choice. The logic behind this is: it can reduce the total cost of ownership (TCO).

10/ Hongjun's Four Styles: Separation of Computing and Storage. With the in-depth development of cloud native, the separation of computing and storage has become a basic paradigm of IT systems. Google started the earliest in this regard, but there is little external information. Amazon published the design of Aurora database at SIGMOD 2017, leading the wave of separation of computing and storage in cloud databases in the industry. Facebook gave a presentation on the separation of computing and storage at the Apache Spark & ​​AI Summit in 2018. In addition to following the economic principle of improving cost efficiency, the separation of computing and storage can also allow the system to expand at a finer granularity (computing and storage) to maintain sustainable development.

11/ Hongjun Five Styles: The ratio of computing and storage. How does the Filecoin mining pool arrange the ratio of computing and storage? According to the operating experience of cloud computing and cloud storage (as shown in the figure below), the cloud hosting scenario requires more computing power than storage capacity. For Filecoin mining, evaluating the ratio of computing and storage has a great impact on the deployment of mining systems and reducing TCO. For sustainable scale growth, the cost of building a data center is lower. For a single mining machine, since it violates the principle of separation of computing and storage in Hongjun Four Styles, the ratio of computing and storage is out of the question, let alone sustainability.

12/ Hongjun Six Styles: Innovative Storage Architecture. This involves the optimization of the entire storage stack for Filecoin mining and storage scenarios. There is a lot of content, and it is hard to describe it in detail. You can refer to the design experience of Google's Colossus file system.

13/ Hongjun Seven Styles: The size of the mining pool. What is the best size of the mining pool? Is the larger the mining pool, the better? As shown in the figure below, as the amount of data increases, the TCO of cloud storage increases rapidly. Combined with Filecoin's vision of building a retrieval market, the size of the mining pool is not the larger the better.

14/ Hongjun Eight Styles: Customize digital center disks to balance IOPS and capacity. Disks will be the core of Filecoin mining, GPUs are not, and tapes are not. SSDs have higher IOPS and are the future of storage, but the problem is that the cost ($/GB) is still too high and will not drop much in the next decade. Google's exploration of data center disks is worth learning from, mainly including focusing on disk collections rather than individual disks (also including evaluating the redesign of 3.5-inch disks in the cloud era, as well as optimizations in security, firmware, tail latency, etc.) and the balance between IOPS and capacity.

For a collection of disks, the TCO of a disk-based storage solution is mainly composed of disk acquisition costs, disk power costs, disk connection overhead (disk slots, computing, and network), and repair and maintenance overhead. Google recommends focusing on disk collections rather than individual disks and considering the global maximum of the following five key indicators:

• Higher input/output per second (IOPS), usually limited by disk head seeks,

• Higher capacity (in GB)

• Lower tail latency,

• Satisfy safety requirements, and

• Lower total cost of ownership (TCO).

For the balance between IOPS and capacity, Google uses a thinking space shown in the figure below: The letters represent the various types of disks, expressed in $/GB (in terms of TCO) and IOPS/GB. Larger disks increase GB/$ but reduce IOPS/GB, and smaller disks do the opposite. Higher disk speeds increase IOPS/GB but reduce GB/$, and lower disk speeds do the opposite. The ideal direction of development is downward and right. New disks of the same type tend to cost less (less $/GB) but are further to the left (because the capacity becomes larger, not faster). Switching to a smaller platter size is a way to move to the right. In general, we can aggregate disks to achieve the model from C to D (rightward and upward), downward and right is what we dream of (such as E to F), and our actual path is often the red line in the figure.

15/ Hongjun Nine Styles: The further you look back, the further you can see forward. This is what Churchill said. Blockchain itself is a model of combinatorial innovation, and blockchain + storage itself is a cost revolution. Therefore, the problems and challenges faced by blockchain storage can also be found in the historical archives. There is no need to reinvent the wheel or make a wheel.