The IPFS white paper discusses a peer-to-peer, version-controlled approach to a file system that has no single point of failure. Since data is content-addressed, nodes in the network do not need to trust each other. According to this paper, IPFS has the potential to replace HTTP and make the web distributed.IPFS -A fast, indexable, versioned, peer-to-peer file systemsummaryIPFS (Inter Planetary File System)is a peer-to-peer distributed file system designed to connect all computer devices with the same file system. In some ways, IPFS is similar to the web, but the web is centralized, while IPFS is a single Bittorrent cluster with distributed storage using git repositories. In other words, IPFS provides a high-throughput content-addressed block storage model with content-addressed hyperlinks. This forms a generalized Merkle DAG data structure that can be used to build version file systems, blockchains, and even permanent websites. IPFS combines a distributed hash table, block exchange with an incentive mechanism, and a self-certification namespace. IPFS has no single point of failure, and nodes do not need to trust each other.1. IntroductionThere have been many attempts to develop globally distributed file systems. Some systems have achieved significant success, while many have failed completely. Among academic attempts, AFS[6] is a successful example and is now widely used, however, others[7, ? ] have not achieved the same results. Outside of academia, the most widely used are peer-to-peer file sharing systems for audio and video media. Most notably, the file distribution systems deployed by Napster, KaZaA, and BitTorrent[2] support 100 million simultaneous users. Even today, BitTorrent maintains tens of millions of active nodes per day. Applications based on these academic file system theories have a large number of users, however, these system theories are at the application layer, not at the base layer. As a result, there is no universal file system infrastructure that can provide low-latency distribution to the world.Maybe it's because HTTP is a "good enough" system that already exists. So far, HTTP has been used as a "distributed file system" protocol and has been widely deployed, combined with browsers, and has a huge technical and social impact. It has now become the de facto standard for transferring files on the Internet. However, it has not adopted dozens of advanced file distribution technologies invented in the last 15 years. On the one hand, it is almost impossible to continuously evolve the infrastructure of the http web due to backward compatibility limitations and the current investment in new models. But from one perspective, since the emergence of http, many new protocols have emerged and are widely used. Upgrading the http protocol can introduce new features and strengthen the current http protocol, but it will degrade the user experience.Some industries have gotten away with using HTTP for so long because moving small files is relatively cheap, even for small organizations with large traffic volumes. However, we are entering a new era of data distribution, with new challenges.
Hosting and distributing petabyte-scale datasets;
Big data computing across organizations;
High-definition on-demand or real-time media streaming at high volumes;
Versioning and linking of large-scale datasets;
Prevent accidental loss of important files, etc.
A lot of this can be summed up as "lots of data, everywhere". We have already abandoned the HTTP distribution protocol for different data due to critical functionality and bandwidth issues. The next step is to make them part of the web itself.Orthogonal to efficient data distribution, version control systems have also managed to develop important data collaboration workflows. Git is a distributed source code version control system that has developed many useful methods for modeling and implementing distributed data operations. The Git toolchain provides flexible version control capabilities that are severely lacking in a large number of file distribution systems. New solutions inspired by Git are emerging, such as Camlistore[? ], a personal file storage system, and the Dat[? ] data collaboration toolchain and dataset package manager. Git has influenced distributed file system design[9] because its content involves the Merkle DAG data model, which enables powerful file distribution strategies. It remains to be explored how this data structure affects the design of high-throughput file systems and how to upgrade the Web itself.This paper introduces IPFS, a novel peer-to-peer versioned file system that aims to reconcile these problems.IPFS combines the advantages of many previous successful systems. IPFS produces outstanding results, even better than the sum of these reference systems. The core principle of IPFS is to model all data as part of the same Merkle DAG.2. BackgroundThis section reviews the important properties of the successful peer-to-peer system technology adopted by IPFS.2.1 Distributed Hash Table (DHT)Distributed hash tables (DHTs) are widely used to coordinate and maintain metadata about peer systems. For example, Mainline DHT is a decentralized hash table that keeps track of all peer nodes.2.1.1 KADEMLIA DHTKademlia[10] is a popular DHT that provides:
Efficient querying over large networks: querying an average contact is O(log2N) nodes. (e.g., a 20-hop network with 100k nodes)
Low coordination overhead: optimizes the number of control messages sent to other nodes.
Resistant to all kinds of attacks, prefers long-lived nodes.
Widely used in peer-to-peer applications, including Gnutella and BitTorrent, forming a network of over 20 million nodes[16].
2.1.2 CORAL DSHTWhile some peer-to-peer file systems store data blocks directly in the DHT, this “wastes storage and bandwidth by storing data on nodes where it is not needed”[5]. The Coral DSHT extends Kademlia in three particularly important ways:
Kademlia stores values in the key nodes whose ids are the "closest" (using XOR-distance). This does not take into account the locality of the application data, ignores "far" nodes that may already have the data, and forces the "closest" nodes to store it, whether they need it or not. This wastes a lot of storage and bandwidth. Coral, in contrast, stores addresses whose peers can provide the corresponding data blocks.
Coral changed the DHT API from get_value(key) to get_any_values(key) ("sloppy" in DSHT). This is still because Coral users only need a (working) peer, not a full list. In return, Coral can distribute only subsets to the "nearest" nodes, avoiding hotspots (overloading all nearest nodes when a key becomes popular).
Additionally, Coral organizes a hierarchy of independent DSHTs called clusters based on region and size. This allows nodes to query peers in their region first, “finding nearby data without querying remote nodes” [5] and greatly reducing the latency of lookups.
2.1.3 S/Kademlia DHTS/Kademlia[1] extends Kademlia to prevent malicious attacks. There are two methods:
S/Kad provides a solution to ensure that NodeId generation is resistant to Sybill attacks. It requires nodes to generate PKI public-private key pairs, derive their identities from them, and sign each other. One solution uses POW proof of work, making the generation of Sybills expensive.
S/Kad nodes search for direct links on disjoint paths, ensuring that honest nodes can link to each other even if there are a large number of dishonest nodes in the network. Even if half of the network is dishonest, S/Kad can achieve an 85% success rate.
2.2 Block Exchange - BitTorrentBitTorrent[3] is a widely used and successful peer-to-peer shared file system that can distribute individual file data pieces in a collaborative network of untrusted peer nodes (clusters). IPFS draws inspiration from the key features of BitTorrent and its ecosystem as follows:
BitTorrent's data exchange protocol uses a bit-for-tat incentive strategy that rewards nodes that contribute to other aspects and punishes nodes that only exploit the other party's resources.
BitTorrent peers track the availability of files and prioritize sending rare pieces. This reduces the burden on seeds nodes and allows non-seeds nodes to trade with each other.
BitTorrent’s standard tit-for-tat strategy is very vulnerable to some exploitative bandwidth sharing strategies. However, PropShare[8] is a different peer-to-peer bandwidth allocation strategy that can better resist exploitative strategies and improve cluster performance.
2.3 Version Control System - GitVersion control systems provide facilities for modeling files that change over time, and distributing different versions efficiently. The popular version control system Git provides a powerful Merkle DAG object model that captures changes to a file system tree in a distribution-friendly way.
Immutable objects represent files (blobs), directories (trees), and changes (commits).
By encrypting the contents of the hash object, the object is made addressable.
Links to other objects are embedded, forming a Merkle DAG. This provides many useful integrity and workflow properties.
A lot of version metadata (branches, tags, etc.) is just pointer references, so it is cheap to create and update.
Version changes simply update references or add objects.
Distributed version changes appear to other users as simply moving objects and updating remote references.
2.4 Self-certifying file system - SFSSFS [12, 11] proposes two compelling implementations: (a) a distributed trust chain, and (b) an equally shared global namespace. SFS introduces a self-construction technology - the registry file: addressing remote file systems uses the following format:/sfs/ : Location: represents the service network location HostID = hash(public_key || Location)Therefore, the name of the SFS file system authenticates its service. Users can verify it through the public key provided by the service, negotiate a shared private key, and ensure all communications. All SFS instances share a global namespace, and the name allocation of this namespace is encrypted and not controlled by any centralized body.3. IPFS DesignIPFS is peer-to-peer, and no node is privileged.IPFS nodes store IPFS objects in local storage, and nodes connect to each other and transfer objects, which represent files and other data structures. The IPFS protocol is divided into a set of sub-protocols responsible for different functions:A Merkle DAG of linked, content-addressed, immutable objects. Used to represent arbitrary data structures, such as file hierarchies and communication systems. See Section 3.5 for details.
Identity - Manages node identity generation and verification. Described in Section 3.1.
Network - manages connections to other peers, using various underlying network protocols. Configurable. See section 3.2 for details.
Routing - Maintains information to locate specific peers and objects. Responds to local and remote queries. Defaults to DHT, but can be changed. Described in Section 3.3.
Swap - A novel block exchange protocol (BitSwap) that supports efficient block distribution. Simulates the market, weakens data replication. Trade strategies are interchangeable. Described in Section 3.4.
Object IPFS is a distributed file system that synthesizes successful ideas from previous peer-to-peer systems, including DHT, BitTorrent, Git, and SFS. The contribution of IPFS is to simplify, evolve, and connect mature technologies into a single cohesive system that is greater than the sum of its parts. IPFS provides a new platform for writing and deploying applications, as well as a new distribution system for versioned big data. IPFS can even evolve the network itself.
Files - A versioned filesystem hierarchy inspired by Git. See Section 3.6 for details.
Naming - A self-certifying mutable name system. See section 3.7 for details.
These subsystems are not independent, they are integrated together and take advantage of each other's properties. However, it is useful to describe them separately, building the protocol stack from the bottom up. Notation: The following data structures and functions are specified in the Go language.3.1 IdentityNodes are identified by a NodeId, which is a cryptographic hash of a public key created using S/Kademlia's static cryptographic puzzle [1]. Nodes store their public and private keys (encrypted with a password). Users are free to set a "new" node identity on each startup, though this will result in a loss of accumulated network benefits. Incentivized nodes remain the same.type NodeId Multihash type Multihash []byte // Self-describing cryptographic hash summary type PublicKey []byte type PrivateKey []byte // Self-describing private key type Node struct { NodeId NodeID PubKey PublicKey PriKey PrivateKey }IPFS identity generation based on S/Kademlia:difficulty = n = Node{} do { n.PubKey, n.PrivKey = PKI.genKeyPair() n.NodeId = hash(n.PubKey) p = count_preceding_zero_bits(hash(n.NodeId)) } while (p < difficulty) When first connecting, the peers exchange public keys and check that: hash(other.PublicKey) is equal to other.NodeId. If not, the connection is terminated.Notes on encryption functions:Rather than locking the system to a specific set of function choices, IPFS supports self-describing values. Hash digest values are stored in a multi-hash format, which includes a header specifying the hash function used and the digest length in bytes. For example: 1This allows the system to (a)choose the best feature use case (e.g., stronger security vs. faster performance) and (b) evolve as feature selection changes. Self-describing values allow for compatible use with different parameter choices.3.2 NetworkIPFS nodes regularly communicate with hundreds of other nodes in the network, potentially across a wide area network.The IPFS network stack features:
Transport layer: IPFS can use any transport protocol and is best suited for WebRTC DataChannels [? ] (for browser connections) or uTP (LEDBAT [14]).
Reliability: If the underlying network does not provide reliability, IPFS can use uTP (LEDBAT [14]) or SCTP [15] to provide reliability.
Connectivity: IPFS can also use ICE NAT wall-penetration technology[13].
Integrity: A hash checksum can be used to check the integrity of the message.
Verifiability: The authenticity of a message can be checked using HMAC using the sender's public key.
3.2.1 Notes on peer node addressing:IPFS can use any network;but it does not bear the responsibility of obtaining an IP and does not directly depend on the IP layer. This allows IPFS to be used in overlay networks. IPFS stores addresses as multi-layer addresses, which are composed of byte strings to facilitate the use of the underlying network. Multi-layer addresses provide a way to represent addresses and their protocols, which can be encapsulated into a format that is easy to parse. For example: # an SCTP/IPv4 connection /ip4/10.20.30.40/sctp/1234/ # an SCTP/IPv4 connection proxied over TCP/IPv4/ip4/5.6.7.8/tcp/5678/ip4/1.2.3.4/sctp/1234/3.3 RoutingIPFS nodes require a routing system that can be used to find: (a)the network addresses of other peers, and (b) peer nodes dedicated to serving specific objects.IPFS uses a DSHT based on S/Kademlia and Coral, which was described in detail in Section 2.1. IPFS is similar to Coral[5] and Mainline[16] in terms of object size and usage patterns, so the IPFS DHT differentiates the stored values according to their size. Small values (equal to or less than 1KB) are stored directly on the DHT. For larger values, the DHT only stores the value index, which is the NodeId of a peer node that can provide specific services for that type of value.The interface of DSHT is as follows: type IPFSRouting interface { FindPeer(node NodeId) // Get the network address of a specific NodeId. SetValue(key []bytes, value []bytes) // Store a small metadata to DHT. GetValue(key []bytes) // Get metadata from DHT. ProvideValue(key Multihash) //Declare that this node can provide a large data. FindValuePeers(key Multihash, min int) // Get the nodes that serve the big data. }Note: Different use cases will require fundamentally different routing systems (e.g. DHT in WAN, static HT in LAN). Therefore,the IPFS routing system can be replaced based on the needs of the user.As long as the above interface is used, the system will continue to operate normally.3.4 Block Exchange - BitSwap ProtocolThe BitSwap protocol in IPFS is inspired by BitTorrent and distributes data by exchanging data blocks between peer nodes.Like BT, each peer node continuously uploads the downloaded data to other peer nodes while downloading. Unlike the BT protocol, BitSwap is not limited to the data blocks in a torrent file. There is a permanent market in the BitSwap protocol. This market includes all the block data that each node wants to obtain. Regardless of which blocks are part of the .torrent file, these blocks may come from completely unrelated files in the file system. This market is composed of all nodes. While the concept of a barter system means that virtual currencies can be created, this would require a global ledger to track ownership and transfers of currencies. This could be implemented as the BitSwap strategy and will be explored in a future paper.In the basic case, BitSwap nodes must provide direct value to each other in the form of chunks. This only works well if the distribution of chunks across nodes is complementary, so each party gets what it needs. This is often not the case, and in some cases nodes must work for their own chunks. In cases where a node does not have what its peers need (or does not have it at all), it will place a lower priority on finding chunks that its peers want. This incentivizes nodes to cache and propagate rare pieces, even if the node is not interested in those pieces.3.4.1 BITSWAP CreditThis protocol must have an incentive mechanism to motivate nodes to seed blocks needed by other nodes, even though they themselves do not need these blocks. Therefore, BitSwap nodes are very active in sending blocks to peer nodes in the hope of getting paid. However, leech attacks (empty-load nodes never share blocks) must be prevented. A simple credit-like system solves these problems:
Peers keep track of their balances (via byte-wise authentication).
As the debt increases, the probability decreases, and peers probabilistically send blocks to debtors.
Note that if a node decides not to send to a peer, the node then ignores the peer's ignore_cooldown timeout. This prevents senders from trying to send multiple times (flooding attacks) (BitSwap defaults to 10 seconds).3.4.2 BITSWAP’s StrategyBitSwap peers adopt many different strategies, which have different and dramatic effects on the overall performance of the block exchange. In BT, the standard strategy is clearly defined (tit-for-tat), and other different strategies have been implemented, ranging from BitTyrant [8] (share as much as possible) to BitThief [8] (exploit a vulnerability, never share), to PropShare [8] (share proportionally). BitSwap peers can similarly implement a range of strategies (good and malicious). When choosing features, one should aim for:
Maximize transaction capacity for entire transactions and nodes.
To prevent empty-load nodes from exploiting and compromising transactions.
Effectively resist unknown strategies.
More lenient towards trusted peers.
Exploring the gaps in these strategies is a matter for the future. One alternative function used in practice is the sigmoid, scaled by the debt ratio:Let debt ratio be between a node and its peers:According to r, the probability of sending to the debt node is:As you can see in Figure 1, when the node debt ratio exceeds twice the node's established credit, the probability of sending to the indebted node will drop rapidly.Figure 1 The probability of sending when r increasesThe debt ratio is a measure of trust: it is more lenient with debts to nodes that have successfully exchanged a lot of data before, and more strict with untrusted nodes that are unknown to it. This (a) creates a barrier for attackers who create a lot of nodes (sybill attack). (b) It protects the relationship between nodes that have successfully traded with each other before, even if the node is temporarily unable to provide data. (c) It eventually blocks communication between nodes whose relationships have deteriorated until they can be proven again.3.4.3 BITSWAP LedgerBitSwap nodes keep a ledger of transactions with all other nodes. This allows nodes to track history and prevent tampering. When a link is activated, BitSwap nodes exchange their ledger information. If the ledger information is not exactly the same, the ledger will be reinitialized and accrued credits and debts will be lost. Malicious nodes will intentionally lose "these" ledgers in the hope of clearing their debts. It is unlikely that a node will accumulate enough debt to authorize certification without accruing credits. Partner nodes are free to view this as misconduct and reject transactions. type Ledger struct { owner NodeId partner NodeId bytes_sent int bytes_recv int timestamp Timestamp }Nodes are free to retain the ledger history, which is not required for correct operation, as only the current ledger entries are useful. Nodes are also free to collect the ledger as needed, starting with the less useful ones: old (other peers may not exist) and small.3.4.4 Detailed explanation of BITSWAPThe BitSwap node has the following simple protocol.// Additional state kept type BitSwap struct { ledgers map[NodeId]Ledger // Ledgers known to this node, inc inactive active map[NodeId]Peer // currently open connections to other nodes need_list []Multihash // checksums of blocks this node needs have_list []Multihash // checksums of blocks this node has } type Peer struct { nodeid NodeId ledger Ledger // Ledger between the node and this peer last_seen Timestamp // timestamp of last received message want_list []Multihash // checksums of all blocks wanted by peer // includes blocks wanted by peer's peers } // Protocol interface: interface Peer { open (nodeid : NodeId, ledger : Ledger); send_want_list (want_list : WantList); send_block(block: Block) -> (complete:Bool); close(final: Bool); }Sketch of the life cycle of a peer connection:
Open: peers send ledgers to each other until they agree.
Sending: want_lists and blocks are exchanged between peer nodes.
Close: The peer node disconnects.
Ignored: (Special) The peer is ignored (wait timeout) if the node adopts the prevent send strategy.
Peer.open(NodeId, Ledger).When a connection occurs, the node will initialize the linked ledger, either by saving a linked ledger or creating a new one that is cleared to zero. Then, it will send an open message carrying the ledger to the peer node.After receiving an open message, a peer can choose whether to accept the connection. If, according to the receiver's ledger, the sender is an untrustworthy agent (transfers are below zero or have a large outstanding debt), the receiver may choose to ignore the request. Ignoring requests is implemented probabilistically with the ignore_cooldown timeout, to allow time for errors to be corrected and attackers to be thwarted.If the connection is successful, the receiver initializes a Peer object with the local ledger and sets the last_seen timestamp. Then, it compares the received ledger with its own ledger. If the two ledgers are exactly the same, the connection is opened. If the ledgers are not exactly the same, the node creates a new zeroed ledger and sends it. Peer.send_want_list(WantList)When a connection is open, nodes will broadcast their want_list to all connected peers. This is done (a) after opening the connection, (b) after a random interval timeout, (c) after the want_list changes, or (d) after receiving a new block. After receiving a want_list, the node will store it. Then, it will check whether it has any blocks it wants. If so, it will send out the blocks required by the want_list according to the BitSwap strategy mentioned above. Peer.send_block(Block)Sending a block is straightforward. The node simply transmits the block of data. When all the data has been received, the receiver calculates a multi-hash checksum to verify that it is the data it needs, and then sends an acknowledgment.After a correct block transfer is completed, the recipient moves the block from need_list to have_list, and finally both the receiver and the sender update their ledgers to reflect the number of additional data bytes transferred.If a transfer fails verification, the sender is either at fault or attacking the receiver, who may choose to reject future transactions. Note that BitSwap is expected to operate over a reliable transmission channel, so transmission errors (which may result in a penalty for honest sender errors) are expected to be caught before data is sent to BitSwap.Peer.close(Bool)The last parameter passed to close indicates whether closing the connection is the sender's intention. If the parameter value is false, the receiver may reopen the connection immediately, which avoids closing the connection prematurely.A peer node closes the connection in two cases:
The silence_wait timeout has expired and no information has been received from the peer node (BitSwap uses 30 seconds by default), the node will send Peer.close(false).
When the node exits and BitSwap is closed, the node will send Peer.close(true).
After receiving the close message, the receiver and sender will disconnect and clear all stored states. The ledger may be saved for future convenience, of course, only if it is considered that the ledger will be useful in the future. Note:
Non-open messages on an inactive connection should be ignored. When sending a send_block message, the receiver should check the block to see if it is what it needs and if it is correct, and if so, use this block. In general, any irregular messages will cause the receiver to trigger a close(false) message and force the reinitialization of the connection.
3.5 Merkle DAG ObjectDHT and BitSwap allow IPFS to construct a large peer-to-peer system for fast and stable distribution and storage. Most importantly, IPFS builds a Merkle DAG, a loop-free directed graph where links between objects are hash-encrypted and embedded in the source and target. This is a generalization of the Git data structure. Merkle DAGS provide IPFS with many useful properties, including:
Content addressable: All content is uniquely identified by multiple hash checksums, including links.
Tamper-proof: All content is verified with its checksum. If the data is tampered or corrupted, IPFS will detect it.
Deduplication: All objects with the same content are stored only once. This is useful for indexing objects, such as git trees and commits, or common parts of data.
The format of an IPFS object is:type IPFSLink struct { Name string // Alias of this link Hash Multihash // encrypted hash of the target Size int //Total target size } type IPFSObject struct { links []IPFSLink //links array data []byte //Opaque content data }The IPFS Merkle DAG is a very flexible way to store data. It only requires that object references be (a) content addressable, and (b) encoded in the above format. IPFS allows applications to have complete control over the data domain; applications can use data in any custom format, even data that IPFS cannot understand. A separate internal object link table allows IPFS to do:
List all object references in object form, for example:
> ipfs ls /XLZ1625Jjn7SubMDgEyeaynFuR84ginqvzb XLYkgq61DYaQ8NhkcqyU7rLcnSa7dSHQ16x 189458 less XLHBNmRQ5sJJrdMPuu48pzeyTtRo39tNDR5 19441 script XLF4hwVHsVuZ78FZK6fozf8Jj9WEURMbCX4 5286 template
Solve string path lookup, such as foo/bar/baz. Given an object, IPFS will parse the first path component, hash it and put it into the object's link table, then get the second component of the path, and repeat this process. Therefore, any string path in any data format can be used in Merkle DAG.
Recursively resolve all object references:
> ipfs refs --recursive /XLZ1625Jjn7SubMDgEyeaynFuR84ginqvzb XLLxhdgJcXzLbtsLRL1twCHA2NrURp4H38s XLYkgq61DYaQ8NhkcqyU7rLcnSa7dSHQ16x XLHBNmRQ5sJJrdMPuu48pzeyTtRo39tNDR5 XLWVQDqxo9Km9zLyquoC9gAP8CL1gWnHZ7zThe public link structure is a necessary component of IPFS to build any data structure. It is easy to see how Git's object model is applied to DAG. Some other potential data structures:
Key-value storage
Traditional relational data
Triple data storage
Document publishing system
Communication Platform
Cryptocurrency Blockchain
These systems can all apply IPFS Merkle DAG, which enables more complex applications of these systems to use IPFS as a transmission protocol.3.5.1 RouteIPFS objects can traverse a string path. The path format is consistent with traditional UNIX file systems and the Web. Merkle DAG links make traversal easy. The full path format in IPFS is: # Format /ipfs/ /# example/ipfs/XLYkgq61DYaQ8NhkcqyU7rLcnSa7dSHQ16x/foo.txtThe /ipfs prefix allows mounting on an existing system as long as there are no mount point conflicts (the mount point name is of course configurable). The second path component (the first in IPFS) is a hash of the object. This is usually the case, because there is no global root. A root object would have an impossible task of handling the consistency of millions of objects in a distributed (and possibly disconnected) environment. Therefore, we simulate the root by addressing it. All objects are accessible via the hash. This means that given a path object /bar/baz, the resulting object can be accessed by all:/ipfs/ /bar/baz /ipfs/ /baz/ipfs/3.5.2 Local ObjectsIPFS clients need a local storage, an external system to store and retrieve local raw data for objects managed by IPFS.The type of storage varies depending on the node use case. In most cases, this storage is just a portion of the hard disk space (either managed by the local file system using a key-value store such as leveldb, or managed directly by the IPFS client), in other cases, such as non-persistent caches, the storage is part of the RAM.Ultimately, all blocks are accessible in IPFS, and blocks are stored in local storage on some nodes. When a user requests an object, the object is found, downloaded, and stored locally, at least temporarily. This provides fast lookups for some configurable amount of time.3.5.3 Object LockingA node that wants to ensure the survival of a specific object can lock that object. This ensures that the specific object is saved on the node's local storage. It can also recursively lock all related derived objects. This keeps all specified objects on local storage. This is particularly useful for long-term storage of files, including references. This also makes IPFS a web where links are permanent and objects can ensure the survival of other specified objects.3.5.4 Publishing ObjectsIPFS is globally distributed.It is designed to allow thousands of users to have files coexisting.DHT uses content hash addressing technology to make publishing objects fair, secure, and fully distributed. Anyone can publish an object by adding the object's key to the DHT and joining it as a peer node, and then giving the path to other users. It is important to note that objects are inherently immutable, just like in Git. New versions have different hashes, so they are new objects. Tracking versions is the job of additional version objects.3.5.5 Object-level encryptionIPFS is capable of handling object-level cryptographic operations. An encrypted or signed object is wrapped in a special framework that allows encryption and verification of raw bytes.type EncryptedObject struct { Object []bytes // Encrypted original object data Tag []bytes // Optional encryption tag type SignedObject struct { Object []bytes //Signed raw object data Signature []bytes // HMAC signature PublicKey []multihash //Multiple hash identity key }The encryption operation changes the hash value of the object, defining a new and different object. IPFS automatically verifies the signature and decrypts the data using the keychain specified by the user. Links to encrypted data are also protected, and objects cannot be traversed without the decryption key. There is also a phenomenon that the parent object may be encrypted with one key, while the child object is encrypted with another key or not encrypted at all. This ensures the security of links to shared objects.3.6 FilesIPFS also defines a set of objects for modeling versioned file systems on Merkle DAG. This object model is similar to Git:
Block: A variable-sized block of data
List: a collection of blocks or other linked lists
Tree: a collection of blocks, linked lists, or other trees
Commit: a snapshot of the tree in the version history
I would have liked to use a model consistent with the Git object format, but that would have required some separation to introduce certain features useful in a distributed file system, such as
Fast size lookup (total byte size already added to the object)
Deleting duplicates of large files (adding to list object)
Commits are embedded in trees.
However, IPFS file objects are very similar to Git, and it is possible to communicate between the two. Moreover, a series of Git objects can be imported and converted without losing any information (UNIX file permissions, etc.).Notation: The following file object format uses JSON. Note that although IPFS includes JSON conversion, the structure of the file object still uses protobufs binary encoding.3.6.1 File Object:BLOBThe blob object represents a file and contains an addressable unit of data. IPFS blobs are like Git blobs or file system data blocks. They store user data. It should be noted that IPFS files can be represented by lists or blobs. Blobs do not have links. { 'data': 'some data here', // blobs have no links }3.6.2 File Object: LISTList objects represent large files or deduplication files connected by several IPFS blobs. Lists contain ordered blob sequences or list objects. To some extent, IPFS list functions are like a file system with indirect blocks. Since lists can contain other lists, it is possible to include linked lists and balanced trees. The same nodes in directed graphs appear in multiple different places to allow deduplication in files. Of course, loops are not possible because they are forced by hash addressing. { 'data': ['blob', 'list', 'blob'], //lists has an array of object type as data 'links': [ { 'hash': 'XLYkgq61DYaQ8NhkcqyU7rLcnSa7dSHQ16x', 'size': 189458 }, { 'hash': 'XLHBNmRQ5sJJrdMPuu48pzeyTtRo39tNDR5', 'size': 19441 }, { 'hash': 'XLWVQDqxo9Km9zLyquoC9gAP8CL1gWnHZ7z', 'size': 5286 } //In links lists have no name ]}3.6.3 File object:treeThe tree object in IPFS is similar to that in Git. It represents a directory and a map of name to hash value. The hash value represents blobs, lists, other trees, or commits. Note that the naming of traditional paths has long been implemented by Merkle DAG. { 'data': ['blob', 'list', 'blob'],//trees has an array of object type as data 'links': [ { 'hash': 'XLYkgq61DYaQ8NhkcqyU7rLcnSa7dSHQ16x', 'name': 'less', 'size': 189458 }, { 'hash': 'XLHBNmRQ5sJJrdMPuu48pzeyTtRo39tNDR5', 'name': 'script', 'size': 19441 }, { 'hash': 'XLWVQDqxo9Km9zLyquoC9gAP8CL1gWnHZ7z', 'name': 'template', 'size': 5286 }//trees has a name ] }3.6.4 File object: commitThe commit object in IPFS represents a snapshot of any object in the version history. Similar to Git, but it can represent any type of object. It also links the initiator object.
3.6.5 Version Control A Commit object represents a specific snapshot of an object in a historical version. Comparing objects (and sub-objects) in two different commits can reveal the difference between two different versions of file systems. As long as the references to commit and all its sub-objects are accessible, all previous versions are available, and all history of all file system changes is accessible, which is separated from the Merkle DAG object model. All features of the Git version control tool are available to users of IPFS. The object model is not exactly consistent, but is also compatible. This may
Build a Git tool version to transform it into an IPFS object graph.
Build a FUSE file system, mount an IPFS tree as a Git repository, and convert the read/write of the Git file system into IPFS format.
3.6.6 File system path As we see in Merkle DAG, IPFS objects can be traversed using the string path API. IPFS file objects are designed specifically to make it easier to mount IPFS to UNIX file system. File objects limit trees to have no data, so that they can represent directories. Commits can appear in the form of representing directories or can be completely hidden in the file system. 3.6.7 Separate files into Lists and Blobs One of the most important challenges in version control and distributing large files is to find a correct way to separate them into separate blocks. Instead of thinking that IPFS can provide the correct way to separate each different type of file, IPFS offers the following options:
Just like in LBFS[?], use Rabin Fingerprints[?] to select a more appropriate block boundary.
Use rsync[?] rolling-checksum algorithm to detect changes in blocks between versions.
Allows the user to specify 'fast-separated' functions that are adjusted for specific files.
3.6.8 Path search performance Path-based access requires traversing the object graph. Getting each object requires finding their key in DHT, connecting to peers, and then getting its blocks. This creates considerable overhead, especially when the paths found are composed of many subpaths. The following method can slow down the overhead:
tree cache: Since all objects are hash-addressed, they can be cached infinitely. In addition, trees are generally smaller, so IPFS will cache trees first compared to blobs.
flattened trees: For any tree, a special flattened tree can build a linked list, and all objects can be accessed from this tree. In the flattened tree, the name is a path separated from the original tree, separated by a slash.
For example, for the flattened tree of ttt111 above, as follows: { 'data': ['tree', 'blob', 'tree', 'list', 'blob' 'blob'], 'links': [ { 'hash': ' ', 'size': 1234 'name': 'ttt222-name' }, { 'hash': ' ', 'size': 123, 'name': 'ttt222-name/bbb111-name' }, { 'hash': ' ', 'size': 3456, 'name': 'ttt333-name' }, { 'hash': ' ', 'size': 587, 'name': 'ttt333-name/lll111-name'}, { 'hash': ' ', 'size': 22, 'name': 'ttt333-name/lll111-name/bbb222-name' }, { 'hash': ' ', 'size': 22 'name': 'bbb222-name' } ] } 3.7 IPNS:Naming and volatile state So far, IPFS has formed a peer-to-peer block exchange to form a content-addressable DAG object. This provides publishing and obtaining unchangeable objects. This can even track the version history of these objects. However, there is a key component here that is missing: volatile naming. Without this, all new content communications will definitely be biased. Now it is necessary to have some ways to get the volatile state of the same path. This is worth explaining the reason—if the ultimate volatile data is necessary—we have put a lot of effort into building an immutable Merkle DAG. Just think that IPFS is separated from the characteristics of Merkle DAG: objects can (a)be obtained through hash values, (b) check for integrity, (c) link other objects, and (d) unlimited cache.In a sense:objects are eternal. These are the key features of a high-performance distributed system, where moving files between network links is very expensive. Object content addressable builds a web with the following features, (a) excellent broadband optimization, (b) untrusted content services, (c) eternal links, (d) permanently reserved any object and its references.
Immutable content addressable objects and named Merkle DAGs, with variable pointers pointing to Merkle DAGs, instantiating a dichotomy that appears in many successful distributed systems. These systems include Git's version control system, using immutable objects and mutable references, and the UNIX distributed successor Plan9[?] file system, using mutable Fossil and immutable Venti[?]. LBFS[?] also uses mutable indexes and immutable blocks. 3.7.1 Self-certification name Using the naming scheme in SFS [12, 11], we provide a way to build self-authentication names, which is mutable in an encrypted global namespace. The IPFS scheme is as follows:
Recall in IPFS:
NodeId = hash(node.PubKey)
We assign each user a variable namespace under this path:
/ipns/
A user can publish an object signed with his or her private key under this path, for example:
/ipns/XLF2ipQ4jD3UdeX5xp1KBgeHRhemUtaA8Vm/
When other users acquire objects, they can detect whether the signature matches the public key and NodeId.This verifies the authenticity of the user's published object and achieves a variable state acquisition.
Pay attention to the following details:
IPNS (Inter Planetary's namespace) separate prefix creates an easily recognizable distinction between mutable and immutable paths, for the sake of programs and for the convenience of human reading.
Because this is not a content-addressable object, publishing it depends on the only variable state allocation system in IPFS, the routing system. The process is (1) First publish this object as a conventional immutable IPFS object, and (2) publish the hash value of this object as the value of metadata to the routing system: routing.setValue(NodeId, )
Any links in the published object act as a subname in the command space: /ipns/XLF2ipQ4jD3UdeX5xp1KBgeHRhemUtaA8Vm/ /ipns/XLF2ipQ4jD3UdeX5xp1KBgeHRhemUtaA8Vm/docs /ipns/XLF2ipQ4jD3UdeX5xp1KBgeHRhemUtaA8Vm/docs/ipfs
It is generally recommended to use historical version records when publishing a commit object or other object, because this way the user can find the name he used before. However, since this is not always necessary, leave a user to choose for himself.
Note that when a user publishes an object, he cannot use the same method to publish the object. 3.7.2 Human friendly name IPNS is indeed a good way to allocate and allocate names, but it is not very user-friendly because it uses a long hash value as the name. It is well known that such names are difficult to remember. IPNS is enough to deal with URLs, but it is not so easy to use for many offline transmission work. Therefore, IPFS uses the following technology to increase the user-friendliness of IPNS. Peer node Links Inspired by SFS, users can directly link other users' objects to their own objects (command space, home directory, etc.). One advantage of this is to create a trusted web (also supports old authenticity authentication models): # Alice links to Bob ipfs link //friends/bob / # Eve links to Alice ipfs link /<eve-pk-hash/friends/alice / # Eve can also access Bob /<eve-pk-hash/friends/alice/friends/bob # Visit Verisign authentication domain / /foo.com
DNS TXT IPNS RecordIf /ipns/ is a valid domain name, IPFS will look for key ipns in the DNS TXT record. IPFS will translate the found value into a hash value of an object or the path to another ipns: # DNS TXT Record ipfs.benet.ai. TXT 'ipfs=XLF2ipQ4jD3U ...' # Behave as a symbolic link ln -s /ipns/XLF2ipQ4jD3U /ipns/fs.benet.ai
Proquit Readable IdentifierThere is always a way to translate binary encoding into readable files. IPNS supports Proquit[?]. As follows: #proquit statement /ipns/dahih-dolij-sozuk-vosah-luvar-fuluh # Decompose into the following form /ipns/KhAwNprxYVxKqpDZ
Shorten name serviceThere will be many servers that provide shortened names and provide users with their namespaces. Just like the URLs for DNS and Web we are seeing now: # Users can get a link from below /ipns/shorten.er/foobar # Then put it in your namespace /ipns/XLF2ipQ4jD3UdeX5xp1KBgeHRhemUtaA8Vm 3.8 Using IPFS IPFS is designed to be used in many different ways, and here are some of the usages I will continue to pursue:
As a mounted global file system, mounted under /ipfs and /ipns
As a mounted personal synchronization folder, it automatically performs version management, publishing, and backup any writes
As an encrypted file or data sharing system
As a version package manager for all software
The root file system as a virtual machine
Start the file system as VM (under the hypervisor)
As a database: the application can directly write data into the Merkle DAG data model to obtain all versions, buffers, and IPFS-provided allocations
As a linked (and encrypted) communication platform
As a CDN check for the integrity of large files (without using SSL)
As an encrypted CDN
On a web page, as a web CDN
As a links, there will always be a new eternal web
Objectives achieved by IPFS:
An IPFS library can be exported to your own application for use
Command line tools can operate objects directly
Mount the file system using FUSE[?] or kernel model
4. The future The idea of IPFS is the product of decades of successful exploration and open source of distributed systems.IPFS integrates many excellent ideas in systems that have been successful so far. In addition to the new BitSwap protocol, the biggest feature of IPFS is the coupling of the system and the synthesis of design. IPFS is an ambition of decentralized network infrastructure, and many different types of applications can be built on IPFS. At the lowest level, it can be used as a global, mountable, version-controlled file system and namespace, or as a next-generation file sharing system. The best case is that IPFS can upgrade the Web to a level. When publishing a valuable information, anyone interested can publish it without forcing it to only allow publishing agencies to publish it. Users can trust the content of the information, and it is irrelevant to trust the sender of the information. Another feature is that some important but very old files will not be lost. IPFS looks forward to taking us into a world of eternal Wdb. 5. Thanks IPFS is a complex of many great ideas and systems. Without standing on the shoulders of giants, IPFS wouldn’t dare to have such an ambitious goal. I would like to thank those who have participated in the long-term discussion of these ideas: David Dalrymple, Joe Zimmerman, and Ali Yahya, and in particular: uncovering the overall architecture of Merkle DAG (David, Joe), rolling hash blocking (David), s/kademlia sybill protection (David, Ali), and especially thanks to David Mazieres for his very smart ideas before. Original address of the white paper (for external network):https://ipfs.io/ipfs/QmR7GSQM93Cx5eAg6a6yRzNde1FQv7uL6X1o4k7zrJa3LX/ipfs.draft3.pdf
