The latest and most comprehensive Filecoin mining guide!

A lot has changed since we published the last version of the Filecoin testnet storage miner mining guide. At that time, we had just launched the first version of the testnet, and then we received many suggestions for uncertainty about the testnet. We are very grateful to everyone who participated in the testnet. Thanks to your support, we have been able to quickly mature our protocol and implementation. Today, we are now in the second phase of the testnet and are steadily moving forward with the mainnet launch.

In this post, we provide an updated guide to Filecoin mining and discuss recently announced incentives and opportunities for members of the growing Filecoin community.

Please note that Filecoin is still a work in progress, and new insights from experiments provided by the testnet are being actively incorporated into the final specification. Therefore, important network parameters are still subject to change before the mainnet launch, and we recommend that miners purchase small-scale hardware for testing, experimentation, and benchmarking before investing large amounts of capital.

Join the Filecoin network

Currently, a node plays two main roles in the Filecoin network: storage and retrieval . We expect that specialized miners will be needed to perform specific roles.

Storage Market

In the Filecoin network, storage nodes have the ability to contract with clients to store their data for an agreed-upon period of time in exchange for Filecoin.

Nodes that provide storage for the Filecoin network are called storage miners. These nodes are periodically awarded the ability to extend the Filecoin blockchain with blocks they create . When they create a new block, storage miners will receive Filecoin rewards, and they can charge transaction fees to other nodes that send messages in the block.

Search Market

Nodes can additionally participate in a retrieval protocol, providing clients with specified files in exchange for Filecoin. This incentivizes well-located nodes with high-throughput, high-bandwidth connections to participate in the network, thereby facilitating widespread and rapid distribution of files, especially those in high demand.

Other roles

A number of other roles are currently being developed (e.g., “repair” nodes that facilitate self-healing of the network) but have not yet been finalized in any implementation , however, the network is fully functional without these improvements.

How do storage miners work?

The role of storage miners is to store files on behalf of the Filecoin network. Storage miners must cryptographically prove that they have fulfilled their promise to store files - this is achieved through the Proof of Replication (PoRep) and Proof of Spacetime (PoSt) mechanisms . Pledge storage to the Filecoin network itself requires FIL, which is used as collateral to ensure that storage miners fulfill their contractual obligations.

Storing Data

In the Filecoin network, data is stored in sectors of fixed size. Typically, storage miners fill these sectors with data stored on behalf of clients , who contract with storage miners for a specific period of time through transactions, however, storage miners are not forced to make transactions. If storage miners do not find any attractive transaction proposals, they can choose to make a capacity commitment to fill sectors with arbitrary data. This allows them to prove that they are reserving space on behalf of the network. If necessary, sectors created to act as capacity commitments can be "upgraded" later to provide contract storage space for future transactions.

Proof of Replication

Once the sector has been filled , the Proof of Replication prompts the Storage Miner to seal the sector. Sealing is the result of an intensive computational process in which a unique representation of the data is created. Once the data is sealed, the Storage Miner generates a Proof of Replication; compresses the proof by running a zero-knowledge proof; and finally, submits the compressed result to the blockchain as proof of storage commitment. The storage reserved for the network through this process is called a pledge.

Proof of Space and Time

Once Proof of Replication is complete, Storage Miners must continually prove that they are still storing the data they committed to storing. This is done through Proof of Spacetime , a process whereby Storage Miners are issued cryptographic challenges that can only be answered correctly by consulting the sealed sectors directly. Storage Miners must answer this challenge within strict time limits; the computational difficulty of sealing ensures that Storage Miners must maintain ready access to and integrity of the sealed sectors.

In Filecoin, Proof of Replication faces two different challenges: WindowPoSt and WinningPoSt.

WindowPoSt

WindowsPoSt is a mechanism for auditing the commitments made by storage miners. It divides each 24-hour period into a series of windows. Accordingly, each storage miner's set of guaranteed sectors is divided into subsets, one for each window. Within a given window , each storage miner must submit a proof of spacetime for each sector in its respective subset, which requires access to each challenged sector at any time and publishes the evidence that leads to SNARK compression as a message in a block to the blockchain. As a result, sectors that create committed capacity are audited at least once every 24 hours, and a permanent, verifiable public record is kept to prove the continued commitment of each storage miner.

The Filecoin network expects storage files to have continuous availability. Sectors that fail to submit WindowPoSt as required will result in a failure, and the storage miners who provide the sector will be slashed, that is, part of their collateral will be confiscated and their effective computing power will be reduced. Storage miners will have a limited time to recover from the failure before they are considered to have completely abandoned their storage commitment. If necessary, storage miners will also have the ability to preemptively declare a failure, which will reduce penalties, but must still be resolved within a reasonable time.

WinningPoSt

WinningPoSt is a mechanism by which storage miners can be rewarded based on the size of their contributions. In the Filecoin network, time is fragmented into a series of small time periods, and the height of the blockchain corresponds to the number of time periods that have passed. Whenever a new block is created, a small number of miners are selected to mine the new block (Filecoin uses tipsets, which allow multiple blocks to be mined at the same height). Each elected miner who successfully creates a block will receive a FIL reward and have the opportunity to charge fees to other nodes for sending messages in the block.

The probability of a storage miner being elected corresponds to its storage capacity, and in a process similar to the underlying WindowPoSt, the storage miner is tasked with submitting a compressed storage proof of a specified sector before the end of the block. Storage miners who fail to complete WinningPoSt within the necessary window will lose the opportunity to mine a block, but will not be penalized for failing to do so.

Storage Capacity

The power of a Filecoin storage miner corresponds to the probability that a storage miner mines a block, roughly proportional to the amount of storage they seal on behalf of the network. To further incentivize the storage of "useful" data beyond simple capacity commitments, storage miners have an additional opportunity to compete for special deals offered by verified clients. Such clients are certified in their intent to offer deals involving the storage of meaningful data, and the power that storage miners gain for these deals is enhanced through a multiplier. After calculating this multiplier, the total hashrate of a given storage miner is called the quality-adjusted hashrate.

Implementation of Filecoin

The Filecoin distributed storage network is open source and has multiple implementations.

At the time of writing, the most mature implementation is the Go-based Lotus (which should be used to access the current testnet implementation) . The Lotus client is able to run on Linux and macOS, and detailed instructions on installing and using Lotus are available through its documentation.

There are currently at least three other implementations under active development. These include go-filecoin (another Go-based implementation) , forest (a Rust implementation developed by ChainSafe) , and fuhon (a C++ implementation developed by Soramitsu) .

Hardware Considerations

Participants in the Filecoin network will need to ensure that their systems are adequately equipped for the roles they intend to play.

Running the Lotus client without mining

If you don’t want to mine, but still want to run the Lotus client, either to keep a wallet or connect to the network, a system with 2-4 CPU cores, 8GiB RAM, and enough storage for the Filecoin blockchain should be sufficient (the current testnet chain grows by about 12GiB per week; improvements to reduce this storage requirement are in the works).

Storage Mining

It should be noted that in its current state, Filecoin storage mining requires powerful hardware to meet the storage and proof requirements. These requirements are mainly driven by the design constraints imposed by the Proof of Replication and Proof of Spacetime mechanisms, and by the balance between computational feasibility and cryptographic security.

Filecoin storage mining is not proof-of-work mining, sealed storage is the only way to gain power on the network, but fast and efficient hardware is required to compute the necessary proofs in an acceptable time frame. Protocol Labs is currently investigating ways to reduce these requirements (for example, by introducing efficiencies in the proof mechanism itself, or by outsourcing SNARK computations to eliminate the need for expensive GPUs) . In the meantime, before making a large investment in hardware, storage miners should carefully consider and experiment with the composition of their systems to ensure they have the performance they need.

Mining machine configuration parameters

The optimal system composition will depend heavily on the storage miner's operating model, including capital expenditures and operating costs; therefore, Protocol Labs cannot provide any specific recommendations. However, we have published some of our own designs, including outlines of machines that are currently suitable for testing and small-scale mining.

We expect storage miners to be able to adapt their configurations to their needs; mining on the testnet is possible using alternative configurations, and we expect many of these configurations to exceed the efficiency of our own builds. We encourage miners to experiment and ask interested community members to share their own benchmark scores on GitHub.

General Hardware Issues

While we cannot provide specific advice, we can offer some general guidance.

CPU : As a rule of thumb, a multi-core CPU with a high clock frequency will speed up the sealing process, allowing storage miners to put storage on the network faster. Protocol Labs’ own test data shows that AMD processors with SHA extensions have a significant advantage over processors from other brands.

GPU : A powerful GPU is necessary to complete SNARK computations within the required time constraints. Lotus is currently designed to support chips made by NVIDIA. We anticipate supporting cards from other manufacturers in the future, and our benchmarks can help provide insight into which chips will be successful.

RAM : The current Filecoin network only supports sealing of 32GiB and 64GiB sectors. Performing necessary calculations on these larger sectors requires a corresponding increase in RAM. It is recommended that the mining system be equipped with at least 128GiB.

Storage : There are many considerations involved in choosing the right storage solution, perhaps the most important being the specific revenue model employed by the mining operation. Storage miners currently need to guarantee 1TiB of raw storage in order to begin mining, but there are many more influencing factors beyond this requirement that miners may find useful to consider.

• First, storage miners should keep in mind that data loss will receive severe penalties; even a flipped data bit may result in severe penalties, so storage miners need to implement data redundancy.

• For storage miners seeking to join the retrieval market, it may also be wise to consider incorporating additional storage in preparation to provide a “hot” copy of the sealed data. Although it is possible to open a sector to recover the original data, a Filecoin implementation that supports this use case would remove this computational burden (this is a feature currently under development by Lotus) .

• Another factor to consider is the Filecoin network’s expectation of high availability. While in theory storage miners should be able to integrate with most commodity hard drives, SSDs, or other non-cold storage solutions , not all storage solutions can guarantee that the mining pool will work 24/7 and with high performance.

• Currently, storage miners also require enough space to store the blockchain itself. Reducing the footprint of the blockchain on disk is a feature that Lotus is actively developing, and Filecoin implementations may also require additional disk storage, equivalent to a small fraction of the committed storage, for bookkeeping.

• Finally, Protocol Labs found in testing that in systems with low amounts of RAM (128GiB), using NVMe storage as swap space can be a supplement, otherwise storage miners may experience out-of-memory issues during certain operations (sealing requires a lot of memory).

Network : A high-performance network is recommended when using distributed Lotus Seal workers (see Advanced Mining Considerations below) (10GbE+ network cards and switches are recommended), and is also recommended when using network-attached storage.

Advanced Mining Notes

As mentioned earlier, Filecoin storage mining is dominated by Proof of Replication and Proof of Spacetime. Proof of Replication itself consists of several stages and Lotus implementation, and Filecoin facilitates the sealing of different machine delegations in these stages to maximize efficiency. Protocol Labs has developed an example architecture designed to leverage these features for large-scale mining. Here, we break down the different bottlenecks to consider when designing a similar system.

Sealing Pre-Commit Phase 1 : In this phase, PoRep SDR encoding is performed. This phase is CPU-bound and single-threaded (by design, it is not amenable to parallelization) . This phase is expected to take several hours. The exact time depends on the size of the sector to be sealed and of course on the specifications of the machine doing the sealing. As mentioned earlier, Protocol Labs found that AMD processors with SHA extensions greatly speed up this process. Using a more efficient CPU will also improve performance.

Sealed Pre-Commit Phase 2 : In this phase, the Merkle tree is generated using the Poseidon hashing algorithm. This process is primarily GPU-dependent, CPUs can be used as an alternative but should be expected to be much slower. When using a GPU, this phase is expected to take 45 minutes - 60 minutes.

Sealed Commitment Phase 1 : This is an intermediate phase that performs the preparation work necessary to generate the proof. It is CPU-bound and typically completes in tens of seconds.

Sealing Commitment Phase 2 : Finally, this sealing phase involves the creation of a SNARK that is used to compress the necessary proofs before transmitting them to the blockchain, a GPU-intensive process that is expected to take 20 to 30 minutes to complete.