From entry level to advanced level: an article to understand IPFS

IPFS is a peer-to-peer (p2p) file sharing system that aims to fundamentally change the way information is distributed globally and around the world.

IPFS consists of multiple innovations in communication protocols and distributed systems that combine to produce an unparalleled file system.

Therefore, to try to understand the full breadth and depth that IPFS reaches, it is important to understand the technological breakthroughs that make it possible.

Communication protocols and distributed systems

For two people to exchange information, they need a common set of rules that define how and when information is transmitted. These rules are broadly called communication protocols, but this is cumbersome, so we simply call them languages.

If you've ever traveled to a country where the language isn't your own, you've probably experienced a breakdown in "communication protocol."

This is the case with computers.

They were unable to communicate with each other and existed as isolated computing devices until the first communication protocols for computing were invented in the early 1980s.

“What the protocol communicates is the computation”

In computers, communication protocols usually exist in multi-layered bundles called protocol suites. For example, the Internet protocol suite consists of 4 layers, each responsible for a specific function.

In addition to communication protocols, the important relationship to understand is the basic structure of the interconnections between computers. This is called system architecture.

There are two types which are "client-server network" and "peer-to-peer network".

Currently, the Internet is dominated by a "client-server" relationship that relies on the Internet Protocol Suite, among which the Hypertext Transfer Protocol (HTTP) is the basis of communication.

1. Data is stored in centralized servers and accessed through location-based addressing.

This makes it easier to distribute, manage, protect data, and expand the capacity of servers and clients. However, there are many weaknesses in terms of security, privacy, and efficiency: control of the server translates into control of the data.

This means that your data can be accessed, altered, and deleted by any party under the control of the server. This could be an entity with legitimate authority over the server, or a malicious hacker.

2. In location-based addressing, data is identified by its location rather than its content.

This limitation means that you must go all the way to a specific location to access a piece of data, even if the same data is available at a closer location.

But the client-server model and HTTP have served the Internet fairly reliably for most of its history.

That's because the HTTP Web is very efficient at moving around small files like text and images. In the first twenty years of the Web, the average web page only grew in size from 2KB to 2MB.

resource

HTTP is great for loading websites, but it wasn't designed for transferring large amounts of data, such as audio and video files.

These limitations have allowed alternative file-sharing systems to emerge and become more mainstream, such as Napster, which primarily streams music, and BitTorrent, which can stream movies and just about anything else.

Fast forward to 2018, and on-demand HD video streaming and big data have become ubiquitous; we will continue to produce and consume more and more data and develop increasingly powerful computers to process it.

Significant advances in cloud computing have helped sustain this transition, but the underlying infrastructure used to distribute all this data has remained largely unchanged.

Interplanetary File System

IPFS attempts to address the shortcomings of the client-server model and the HTTP Web through a novel p2p file sharing system.

The system is a synthesis of several new and existing innovations.

IPFS is an open source project created by Protocol Labs, an R&D lab for network protocols and a former Y Combinator startup. Protocol Labs also develops complementary systems such as IPLD and Filecoin.

Hundreds of developers around the world contribute to the development of IPFS, so its orchestration has been a difficult task.

The following are the main components

1. Distributed Hash Table

A hash table is a data structure that stores information as key/value pairs.

In a distributed hash table (DHT), data is distributed across a network of computers and efficiently coordinated to enable efficient access and lookup between nodes.

The main advantages of DHT are decentralization, fault tolerance, and scalability.

Nodes do not require central coordination, the system can operate reliably even if a node fails or leaves the network, and the DHT can scale to accommodate millions of nodes. These features together form a system that is generally more resilient than a client-server structure.

2. Block Trades

The popular file-sharing system Bittorrent is able to successfully coordinate data transfer between millions of nodes by relying on an innovative data exchange protocol, but it is limited to the torrent ecosystem.

IPFS implements a generalized version of this protocol called BitSwap.

BitSwap can serve as a market for any type of data, and this market is the foundation of Filecoin: a p2p storage market built on IPFS.

3. Merkel Dai

A merkle DAG is a hybrid of a Merkle tree and a directed acyclic graph (DAG).

Merkle trees ensure that blocks of data exchanged on a p2p network are correct, uncorrupted, and unaltered. This verification is done by organizing blocks of data using a cryptographic hash function. This is simply a function that takes an input and computes a unique alphanumeric string (hash) corresponding to that input.

It's easy to check whether an input results in a given hash, but it's hard to guess the input from the hash.

Individual chunks of data are called "leaf nodes," which are hashed to form "non-leaf nodes." These non-leaf nodes can then be combined and hashed until all chunks of data can be represented by a single root hash.

Here's a simpler way to conceptualize it:

DAG is a way to model a topological sequence of information without cycles. A simple example of a DAG is a family tree. Merkle DAG is basically a data structure where hashes are used to reference data blocks and objects in the DAG.

This creates several useful functions:

Everything on IPFS can be uniquely identified because each block of data has a unique hash. Plus the data is tamper-proof because making changes to it changes the hash value, like this:

A central tenet of IPFS is to model all data on a universal merkle DAG. It is difficult to overstate the importance of this security feature.

4. Version Control System

Another powerful feature of the Merkle DAG structure is that it allows you to build a distributed version control system (VCS). The most popular example is Github, which allows developers to easily collaborate on projects at the same time.

Files on Github are stored and versioned using merkle DAG. It allows users to independently copy and edit multiple versions of a file, store those versions, and then merge the edits with the original file.

IPFS uses a similar model for data objects: as long as the object corresponding to the original data and any new versions can be accessed, the entire file history can be retrieved.

Assuming that data blocks are stored locally across the network and can be cached indefinitely, this means that IPFS objects can be stored permanently.

Additionally, IPFS does not rely on access to the Internet Protocol. Data can be distributed across an overlay network, which is simply a network built on top of another network. These features are very compelling because they are core elements in a censorship-resistant network.

The changes IPFS can bring to our lives can be a useful tool in promoting free speech to combat the global epidemic of internet censorship, but we should also recognize the potential for bad actors to abuse their power.

5. Self-certifying file system

The last fundamental component of IPFS is the Self-Certifying File System (SFS). It is a distributed file system that does not require special permissions to exchange data.

This is "self-authenticating" because the data provided to the client is authenticated by the file name (signed by the server). You can securely access remote content with the transparency of local storage.

IPFS builds on this concept to create an interplanetary namespace. It is an SFS that uses public key cryptography to self-authenticate objects published by network users. Earlier we mentioned that all objects on IPFS can be uniquely identified, but this also extends to nodes.

Here is a quick review of the key components of IPFS

1. Using a distributed hash table, nodes can store and share data without central coordination;

2. IPFS allows for the immediate pre-authentication and verification of exchanged data using public key cryptography;

3. Merkle DAG can achieve unique identification, tamper-proof and permanent storage of data;

4. You can access past versions of edited data through a version control system.

Simple conceptual framework

So, why does all of this matter?

IPFS provides high throughput, low latency, data distribution. It is also decentralized and secure. This opens up several interesting and exciting use cases. It can be used to deliver content to websites, store files globally with automatic versioning and backups, facilitate secure file sharing and encrypted communications.

The following are projects built on IPFS

Origin, a peer-to-peer marketplace for the sharing economy;

Akasha, the next generation social network;

Balance3, a three-entry accounting platform;

BlockFreight, an open global freight network;

Digix, a platform for tokenizing physical gold;

Infura, an infrastructure provider for DApps;

Livepeer, a decentralized live video streaming platform;

UPORT, autonomous identification system.

The diversity of these applications demonstrates the versatility of IPFS in several different use cases. It is also used as a supplementary file system for public blockchains and other p2p applications.

At the time of writing, storing a kilobyte of data in an Ethereum smart contract can cost several dollars. This is a major constraint at a time when there is a massive growth in new decentralized applications (DApps).

IPFS is interoperable with smart contracts and blockchain data, so it can add reliable, low-cost storage capacity to the Ethereum ecosystem. An attempt to make Ethereum blockchain data natively accessible on IPFS is a separate protocol called IPLD (InterPlanetary Linked Data).

IPFS Challenges

Although IPFS performs well, there are still some problems that have yet to be fully resolved. First, content addressing on IPNS is not very user-friendly yet.

Your IPFS link will look like this:

ipfs.io/ipns/QmeQe5FTgMs8PNspzTQ3LRz1iMhdq9K34TQnsCP2jqt8wV/

These links can be simplified to simpler names using the Domain Name System (DNS), but this introduces an external point of failure for content distribution. However, the content can still be accessed through the original IPNS address. Some users have also reported that IPNS can be slow to resolve domain names, with delays of up to several seconds. It is unclear what the source of this problem is.

On IPFS, nodes have little incentive to maintain long-term backups of data on the network. Nodes can choose to purge cached data to save space, which means that in theory, if there are no nodes left hosting the data, files will eventually "disappear" over time.

At current adoption levels this is not a huge issue, but in the long term, backing up large amounts of data will require strong economic incentives.

Storage Market

Filecoin is a separate protocol that aims to add economic incentives for file storage on IPFS and promote a distributed storage market that competes with enterprise cloud storage such as Amazon S3.

Instead of centralized infrastructure with fixed prices, IPFS + Filecoin stores data on a global network of local providers who are free to determine prices based on supply and demand.

Instead of a Proof-of-Work consensus algorithm (like Bitcoin), Filecoin uses Proof-of-Storage to ensure security and reliability.

This way, anyone can join the network, provide unused hard drive space on their computing devices, and be rewarded in Filecoin tokens for data storage and retrieval services.

The network is developed on Ethereum, so smart contract integration can produce advanced features such as custody, insurance, etc. in the storage market.

In theory, this economic model should create a competitive free market with potentially lower costs than large providers. But Filecoin isn’t live yet, so it will be interesting to see how these concepts play out in reality.

Getting Started with IPFS

IPFS is a very ambitious undertaking, and obviously the precise mechanics of how the system functions are much more complex than what can be described in this guide. We leave those details to cryptographers and computer scientists.

You don’t have to be an expert to use IPFS, so if any of the benefits or use cases seem useful or appealing to you, download IPFS and get started here.

If you have GB or TB of unused storage space and want to make full use of that idle capacity, you can sign up to become an early Filecoin miner when the network launches. If you are interested in becoming an early storage user, you can also sign up.

Using IPFS is compelling, and learning about the technical wizardry that makes it possible is even more exciting.

If successful, IPFS and its complementary protocols could provide a resilient infrastructure for the next generation of the Web. Promising to deliver a distributed, secure, and transparent web.