A brief discussion on storage and retrieval transactions in Filecoin

The Filecoin network consists of hundreds of storage providers distributed around the world. Content addressing and cryptographic storage proofs verify that data is correctly and securely stored on miners' hardware for a long time, creating a strong and reliable service.

This article explains the various stages of the operation of two types of transactions in Filecoin, storage transactions and retrieval transactions, and explains their life cycle in detail. It also explains how cryptographic proofs are used to verify whether participants in the system fulfill their responsibilities as promised.

Data on Filecoin

To store files on Filecoin, users must first import the files in their local Filecoin node. This step generates a data CID - a content identifier, a unique ID that describes the content . After that, the data is passed to the miners.

Importing data into the local Filecoin node can be done through the lotus client import command. Remember the generated data CID (which can also be obtained on the local node later) because it will be used when retrieving data from the miner later.

After importing the data into the local node, the user needs to initiate a transaction. This step is completed through the lotus client deal command. This command takes a data CID as input, generates a Filecoin Piece, and interactively guides the user to complete the storage transaction process.

Filecoin Piece is the main negotiation unit for users to store data on the Filecoin network. Filecoin Piece does not have a specific size, but is capped at the sector size and controlled by network parameters. If a Filecoin Piece is larger than the sector size supported by the miner, it must be split into more pieces so that each piece fits into a sector.

Filecoin Piece

Each Filecoin Piece is a CAR file containing an IPLD DAG with corresponding data CID and piece CID.

CAR stands for Content Addressable Archive. Each CAR file is a serialized representation of an IPLD DAG, which is a string of data blocks plus header information describing the DAG graph (and the root CID).

When a user wants to store a file in the Filecoin network, they first need to use UnixFS to create an IPLD DAG of the file (this is what the lotus client import command does). The hash representing the root node of the DAG is an IPFS-style CID, called a data CID.

UnixFS is a protobuf-based format used to describe files, directories, and soft links in IPFS. In Filecoin, UnixFS is the file format standard, and files are submitted to the Filecoin network in this format.

The generated CAR file is padded with extra zero bits so that the file is written as a binary merkle tree.

Storage transaction process

Users store and access data in the Filecoin network through transactions. Participants in the network, including miners (supply side) and users (demand side), interact with each other through storage transactions and retrieval transactions.

The life cycle of a storage transaction is as follows:

1. Discovery

The user first determines the miner and its pricing, which is the price per GiB per epoch (30 seconds) that the miner wants to receive in order to accept the transaction, in attoFIL. Currently, the minimum term for a transaction in Filecoin is 180 days.

You can query the synchronized nodes through the JSON RPC API and list all currently active miners using the Filecoin.StateListMiners method. You can select miners based on their reputation and capabilities in the network. Miner reputation metrics have not yet entered the Filecoin protocol.

After you have selected a miner, you can use methods such as Filecoin.StateMinerInfo to obtain the miner's PeerID, which is used to establish a secure connection with the other party in the libp2p protocol.

Next, you can use the Filecoin.ClientQueryAsk method to get a signed StorageAsk.

The result includes details of the deal that the miner is willing to accept, such as the range of accepted Filecoin Piece sizes and the price per GiB per epoch. It should be noted that proposing a storage deal that matches the miner's storage requirements is only a prerequisite, but not sufficient to ensure that the deal is accepted - the storage provider may run its own decision logic afterwards.

2. Negotiation and data transmission

At this stage, both parties reach an agreement on the transaction terms, such as transaction costs, transaction period, transaction start time, etc.

Then, the data is sent from the user to the miners.

3. Release

Transactions are published on-chain via the PublishStorageDeals message, making storage providers publicly responsible for transactions.

4. Finish

Once a transaction is published on the chain, it will be handed over to the mining subsystem, packaged into sectors, and then encapsulated, and then continuously proven to be available.

Storage Mining Subsystem

The storage mining subsystem ensures that the data of the Filecoin network is effectively stored by miners and:

• Participate in the Filecoin storage market, take over user data, and participate in storage transactions.

• Participate in the Filecoin storage computing power consensus, verify and produce blocks, grow the Filecoin blockchain, and obtain block rewards.

The system oversees the following processes:

Committing new storage and registering new sectors

In order to register a sector in Filecoin, a miner must seal the sector. The sealing process requires a lot of computation to produce a unique representation of the data in the form of a proof, known as a Proof of Replication, or PoRep. Once the proof is generated, the miner compresses it and submits the result to the blockchain. This proves that the miner did make a copy of the data they agreed to store.

To prove that the storage is continuously available, all storage miners need to continuously submit on-chain proofs to verify that the sectors are fully stored.

Announcing and recovering from storage failures, if the above proofs required for a sector are not successfully submitted, will result in a failure and the miner will be penalized.

Storage miners and users' considerations, as mentioned above, storage transactions are activated and sealed only after they are published on the chain. This is very important because publishing transactions will lock the user's funds in on-chain custody. Only in this way can the miners' income be guaranteed after the data is sealed into the sector.

Publishing a transaction on the chain can be thought of as signing a contract, and encapsulating and activating the transaction can be thought of as starting the work of fulfilling the commitment.

From a user's perspective, if you want to use Filecoin to store data , the transaction generally goes through the following stages:

1. Transaction deposit, users lock funds into escrow

2. Proposing transactions to miners

3. Check the intention to accept the transaction

4. Data is transferred to miners to make transactions, which is done through the GraphSync protocol. GraphSync is a protocol for synchronizing IPLD graphs between nodes. The protocol allows a local node to make a request to a remote node to obtain the results of a search by selector on the remote node's IPLD graph. Lotus uses ipfs/go-graphsync, an implementation of the GraphSync protocol.

5. Check for acceptance - Make sure the miner has accepted the transaction and published it on-chain.

6. Packaging - The transaction is on-chain and miners are in the process of packaging the sectors containing the transaction.

7. Activated - The transaction has been sealed and is active. From here on, storage providers/miners should periodically prove that they continue to store the data.

From the perspective of miners, who provide services by storing user data , transactions generally go through the following stages:

1. Verify transaction - Receive a transaction proposal and check its parameters (size, price, etc.).

2. Check if there is locked funds - Make sure the user has locked funds to pay for the transaction.

3. Waiting for data - receiving transaction data provided by the customer.

4. Provide collateral for transactions for on-chain transactions.

5. Publish transactions on the chain.

6. To encapsulate sector activation transactions, storage providers (miners) periodically submit WindowPoSt to prove that they are continuously storing data.

• Discovery - Users find out which miners have the data they need and ask them for retrieval quote details - price per byte, unblocking price, payment interval.

• Setting up a payment channel - The user needs to set up a payment channel with the miner (if one does not already exist).

• Data Transfer and Payment - Miners send data to users until payment is required. When a certain threshold is reached, payment processing is requested and data transfer can continue. Depending on whether the miner has the data in their block storage, they may need to unseal the data first - this is an unconventional and non-instantaneous operation, which is the reverse of the sealing described in the storage transaction section.

• At this point, the user has not yet obtained complete data.

• WindowPoSt

• WinningPoSt

• Sector Failure Fee : Payable daily by each sector in a failed state. The size of this fee is slightly higher than the block reward that the sector is expected to receive per day. If a sector is in a failed state for more than 2 consecutive weeks, the sector will pay a termination fee and be removed from the blockchain state.

• Sector Fault Detection Fee : This is a one-time fee that is charged if the fault is detected by the on-chain mechanism rather than reported honestly by the miner. Given the probabilistic nature of the Proof of Spacetime check, this fee is set to the block reward for the corresponding sector over a number of days.

• Sector Termination Fee : A sector may be terminated before the expiration date due to a fault or the initiative of the miner. The termination fee charged is in principle equivalent to the current income generated by a sector, and does not exceed a limit to avoid hindering long-term sectors.