Filecoin verifies storage, eliminating trust issues in the decentralized storage market

This is the second blog post in a series focusing on features unique to the Filecoin protocol. Learn about the first post, which explored off-chain data transfers.

The Filecoin market works efficiently because clients can be confident that storage miners on the network are storing their data correctly. This confidence comes from Filecoin's own unique proof system, which ensures that storage on the Filecoin network works as contracted.

With Filecoin, miners’ storage can be publicly audited. In other words, our storage proofs provide assurance to every user of the Filecoin network that all miners are storing the data as promised. Storage verification is the glue that holds the Filecoin market together; it ensures the integrity of all data stored on the network.

In order for these data integrity checks to provide storage users with the assurances they need, the checks must be part of Filecoin’s most fundamental protocol. That’s where Filecoin’s proof system comes into play.

A proof system is a cryptographic protocol that participants (usually miners) use to verify storage services. Filecoin’s proof system solves a previously difficult problem for decentralized storage: How can miners prove that they actually stored the data they said they did, at the time they said they did, and in the physical space they dedicated? To understand this, we need to look at how Filecoin implements decentralized storage and what makes its proof system unique.

In centralized storage services, we trust a reputable company or entity that can vouch for the integrity and security of its systems. This third-party entity is key to maintaining confidence in the network. For example, when we use Amazon S3 for cloud storage, we don’t really trust the data server network that Amazon uses. Rather, we trust Amazon to guarantee that the network will operate as promised - that the data we store on Amazon will be safe and available according to the company’s service level agreement.

Trust in decentralized storage networks is more difficult to achieve. On the Filecoin network, anyone in the world can provide storage space. But to maintain trust on a decentralized network like Filecoin, you need a way to establish trust on the global network itself.

Filecoin’s proof system is achieved by establishing trust in the protocol layer of the Filecoin blockchain. The Filecoin protocol uses storage proof as the core mechanism for reaching consensus in the Filecoin blockchain.

On every public blockchain, “consensus” defines the agreement about the data entries that exist on the blockchain. To reach consensus, blockchains rely on consensus algorithms to generate new blocks and protect the blockchain from fraudulent data entries. These algorithms determine precisely how miners contribute to building and maintaining the blockchain.

However, while other blockchains rely on consensus algorithms that require miners to perform wasteful work (e.g., proof-of-work blockchains), Filecoin’s consensus mechanism causes miners to perform useful work in providing and verifying the integrity of stored data (various proofs of storage, namely proof-of-replication and proof-of-spacetime).

This means that the Filecoin blockchain itself verifies the storage actions performed by miners. This is how Filecoin builds trust in the protocol layer of its decentralized network. At the core of the Filecoin consensus algorithm are two proof mechanisms that together make data storage publicly verifiable on the Filecoin blockchain: Proof of Replication and Proof of Spacetime.

To verify storage on the Filecoin decentralized network, you need to prove two things. First, you need to prove that the correct dataset is stored in a given storage space. Second, you need to prove that the same dataset has been stored continuously for a given period of time.

Filecoin's proof algorithm can perform these verification tasks. Proof of Replication proves that a given miner is storing a physically unique copy of the client's original data, while Proof of Spacetime proves that the client's data is stored continuously over time. The two proofs work as follows.

Proof of Replication (PoRep) starts with a process called “sealing.” A miner dedicates a portion of available storage space, called a “sector,” to store clients’ data. Once that sector is filled, it is sealed. Sealing is a set of operations that gradually transforms a sector into a unique copy of the original data. That copy is associated with the Filecoin miner’s public key. To perform the final PoRep, the miner then submits a cryptographic hash of the copy (its CommR) to the public Filecoin blockchain.

With PoRep, miners can provide public proof that they are storing a unique encoding of the client's data when they perform the proof. Only miners with all the original data can submit the correct CommR (on-chain commitment to the copy) to the Filecoin blockchain. However, a PoRep alone cannot verify that the storage will be continuous over time.

This is where Proof of Spacetime (PoSt) comes in. With PoSt, randomly selected miners are asked to provide a PoRep for a randomly selected storage sector they maintain. This is done through a procedure where miners are issued a cryptographic challenge that can only be answered correctly by consulting the sealed sector directly. Miners must answer this challenge within a strict time limit. Each miner must prove all of their storage every day, and miners are also randomly selected to prove storage in order to win a block. The computational difficulty of sealing ensures that miners must maintain ready access and integrity to the sealed area.

PoRep and PoSt together constitute Filecoin's unique proof system, which is a combination of storage proof and space proof.

In addition to the proof system, the Filecoin network relies on game-theoretic incentives to discourage malicious or negligent activity. All miners who agree to store data on the Filecoin network must provide collateral in the form of Filecoin at the time of agreement. Any storage miner that fails the PoSt check is penalized, and a portion of that collateral is lost whenever a penalty is imposed. After too many penalties are imposed, all collateral is lost, preventing the miner from providing storage to clients again. You can learn more about some of these crypto-economic mechanisms here.

The Filecoin protocol uses novel cryptography, consensus protocols, and game-theoretic incentive design to provide a decentralized storage network and market that clients and miners can use with confidence. At the core is Filecoin's unique storage verification method. You can read more about verifying storage on Filecoin in our latest ProtoSchool tutorial.