Distributed and clustered are inseparable. A brief analysis of the optimization direction of Filecoin cluster architecture

The exploration of Filecoin clusters has a long history and has never stopped.

If Filecoin mining is compared to building a high-rise building, then the cluster is like laying the foundation. The structure of the foundation, the quality of materials, the construction plan, etc. all affect how high and wide the house can be built, how many people it can accommodate at most, and whether it can remain standing and unshakable when disturbed by natural and man-made factors such as typhoons, heavy rains, and earthquakes.

If Filecoin is the underlying infrastructure of blockchain, then the cluster architecture is the underlying infrastructure of Filecoin. Next, we will talk about the optimization direction of Filecoin cluster from the three stages of cluster development.

Distributed and clustered

What is a cluster?

A cluster refers to deploying the same application or service module on multiple different servers. In simple terms, a cluster refers to bringing several servers together to implement the same business.

A cluster generally refers to physical concentration and unified management. In terms of form, a cluster is just a physical form, but not a working method. In terms of working method, clusters are also divided into stand-alone structure clusters, cluster structure clusters and distributed structure clusters.

A cluster consisting of a single machine structure

The stand-alone structure mentioned here does not refer to a single mining machine, but a cluster composed of individual mining machines. It is called a stand-alone structure because this cluster does not have specific functional subdivisions, and one mining machine includes almost all functions.

Taking Filecoin mining as an example, assuming that Filecoin mining has six major processes (P1, P2, C1, C2, Lotus, miner ), a cluster composed of a single-machine structure means that these six major processes are placed on one mining machine. These mining machines also form a cluster , but each mining machine is an independent individual, and the system interactivity is relatively low.

The cluster composed of a single machine structure is relatively rough, and all processes are presented on one mining machine, which will reduce processing efficiency and reduce the composability between various functional modules. As far as Filecoin mining is concerned, this may increase the time period for generating effective computing power.

Cluster structure composed of clusters

The cluster composed of cluster structure is also composed of mining machines. The main difference from the cluster composed of single machine structure is that the single mining machine in the cluster structure does not undertake all the processes at one time, but focuses on a single process. Therefore, the cluster composed of cluster structure has higher subdivision and granularity than the cluster composed of single machine structure .

Taking Filecoin mining as an example , each process such as P1, P2, C1, C2, etc. is provided by multiple mining machines. Therefore, the processing power and efficiency of the cluster can be improved exponentially, which can reduce the coupling between systems and improve management efficiency and cluster effectiveness.

Compared with a stand-alone structure, when a company's business scale expands, a cluster structure can add corresponding equipment more quickly and more specifically to meet business needs.

Distributed Cluster

The cluster composed of cluster architecture takes the route of "refinement" and separates all functions, which helps to maximize efficiency, but there is also a certain degree of resource waste.

We all know that not all processes in Filecoin mining are completely independent. Many of them even interact with each other. For example, the work of P1 and P2 is related to data encapsulation, and the work of C1 and C2 is related to zkSNARK . Therefore, these related processes can be put together to improve the utilization rate of mining machines, increase the reusability of services , and reduce costs while ensuring maximum efficiency.

Although each type of cluster has its own advantages and corresponds to different scenarios, in general, compared with a single-machine cluster, a distributed cluster can reduce costs and increase efficiency. Compared with a cluster composed of a cluster architecture, it can reduce resource waste and improve service reusability .

In fact, the times are changing rapidly, and the architecture of clusters is also constantly developing and improving. Distribution and clusters must be combined after a long period of separation, and a truly excellent cluster manager and builder will definitely be able to select the cluster model that best suits the company's current situation, and then carry out personalized deployment based on business conditions.

Filecoin cluster architecture optimization direction

Combining the knowledge mentioned above, let’s think about the following two questions together to have a deeper understanding of the optimization direction of the Filecoin cluster.

Filecoin has built a decentralized storage trading market for cloud storage based on the IPFS protocol, which mainly includes two markets: "storage" and "retrieval". Currently, miners obtain block rewards mainly by mining in the storage market.

The four stages of Filecoin mining (software aspect)

Filecoin mining (miners receive block rewards) generally goes through four stages: "P1, P2, C1, C2":

• The P1 phase mainly involves breaking up and sharding the files and calculating the Merkle tree of the original data. This phase mainly consumes CPU and takes about 4 hours.

• In the P2 stage, Replica needs to be generated and Column Hash needs to be calculated. This stage requires a lot of gas. The GPU is the main processor in this stage, which takes about 20 minutes.

• The C1 stage is unwinding and takes only a few seconds.

• The C2 stage mainly involves the circuit processing of zero-knowledge proof and the process of generating zero-knowledge proof, which takes about forty minutes.

P1 takes 4 hours, which is essential. This explains why during the first phase of Filecoin testing, the second phase of testing, and the start of the space race, the fastest time for miner nodes to go online was after four hours, meaning that users can generally only see data at noon .

Question 1: “How can miners pack the most data in the shortest time?”

For example, if P1 takes 4 hours, should we run P2, C1, and C2 after running P1, or should we consider running a combination of 1 P1 and 3 P2s? What combination can achieve the fastest effective computing power packaging speed?

The composition of Filecoin mining machine (hardware matching)

A computer cluster refers to the integrated hardware and software that are closely connected, layered, and coordinated to complete a task together. Therefore, we should not only think about the software combination, but also the hardware matching and how to match the software and hardware.

As we all know, the hardware of Filecoin mining machine is mainly composed of three parts: Worker, Miner and Storage. Worker is responsible for writing data, which requires a high-performance AMD processor; Miner is responsible for sending and receiving data, and its main job is Post, which means that graphics cards and GPUs are needed; Storage is responsible for storing data, which is equivalent to giving data a home, and at this time, high-quality hard disks are needed.

Question 2: "How should we match the various components so that they can be independently developed, deployed, and troubleshooted, while also matching each other to reduce coupling, making the system more scalable and flexible, thereby maximizing cluster performance"?

Seeing this, I believe you have already understood that the first question is mainly about the principle of Filecoin mining, and the software optimization is carried out, and its main purpose is to encapsulate the most valid data in the shortest time; the second question is mainly about the Filecoin hardware configuration, and its main purpose is to reasonably match the various components of the Filecoin mining machine, so as to avoid waste of resources, simplify the process, and improve the "granularity" of hardware management, thereby improving performance.

A truly excellent cluster manager and architect must not only achieve the optimal solution for one aspect, but also be able to balance the conflicts between software and hardware, so that the cluster can store the most data at the lowest unit cost and the fastest speed.

After all, in the Filecoin network, the more data is stored per unit time, the faster the effective storage capacity grows, the more effective computing power there is, and the more block rewards you get.