From concepts to underlying technologies, understand blockchain architecture design in one article (with knowledge graph)

Preface

As an implementation of architectural design, blockchain is quite different from the knowledge base of basic languages ​​or platforms. Blockchain is the technology behind cryptocurrency and is one of the hottest technologies on par with VR. It is not a new technology in itself, similar to Ajax. It can be said that it is a technical architecture, so let's talk about the technical implementation of blockchain from the perspective of architectural design.

No matter what programming language you are good at, you can refer to this design to implement a blockchain product. At the same time, we will sort out the related knowledge graphs and systems to help everyone study and research systematically.

Basic Concepts

The concept of blockchain is very popular recently. It comes from the implementation of cryptocurrencies such as Bitcoin, but at present, this technology has been gradually applied in various fields. What is blockchain technology? In order to understand this problem intuitively, we can use the example of Google Earth as an analogy. Ajax is not a new technology, but it is combined to create the product Google Earth. Similarly, blockchain is not a new technology, but it is combined with encryption and decryption technology, P2P network, etc., to create Bitcoin. Technical personnel, especially Web development engineers, first learned about Ajax technology because they were attracted by the cool effects of Google Earth. Now, history repeats itself again. Many people are attracted by the crazy development of Bitcoin and begin to study the technology behind it - blockchain.

Blockchain was originally a unique way for cryptocurrencies such as Bitcoin to store data. It is a self-referencing data structure used to store a large amount of transaction information. Each record is linked in order from back to front, and has the characteristics of openness, transparency, inalterability, and easy traceability. In fact, this characteristic also directly reflects the characteristics of Bitcoin as a whole, so it is very intuitive and appropriate to use blockchain to summarize the technical implementation behind cryptocurrency. Blockchain is a technology, and cryptocurrency is a type of product developed and implemented by it (including tokens, and there are blockchain products without tokens), which cannot be equated or confused. Compared with cryptocurrency, the name blockchain puts aside the concept of tokens, is more figurative, technical, and depoliticized, and is more suitable as a technology to study and promote.

Therefore, when people talk about blockchain alone, they are referring to blockchain technology, which is a product architecture design method that realizes data openness, transparency, and traceability, and is considered a broad blockchain. When talking about blockchain in specific products, it can refer to a data storage method similar to Bitcoin, perhaps a database design, or a file design, which is considered a narrow blockchain. Broadly speaking, blockchain technology must include four aspects: peer-to-peer network design, application of encryption technology, implementation of distributed algorithms, and use of data storage technology. Other aspects may involve distributed storage, machine learning, VR, the Internet of Things, big data, etc. Narrow blockchain only involves data storage technology, database or file operations, etc. The blockchain in this article refers to a broad blockchain.

Architecture diagram

From the perspective of architectural design, blockchain can be simply divided into three layers: protocol layer, extension layer, and application layer. Among them, the protocol layer can be divided into storage layer and network layer, which are independent of each other but inseparable. As shown in the figure:

Protocol Layer

The so-called protocol layer refers to the lowest level of technology. This level is usually a complete blockchain product, similar to the operating system of our computer. It maintains the network nodes and only provides APIs for calling. Usually the official will provide a simple client (commonly known as a wallet). The functions of this client wallet are also very simple. It can only establish addresses, verify signatures, transfer payments, check balances, etc. This level is the foundation of everything. It builds the network environment, sets up transaction channels, and formulates node reward rules. As for what you want to trade and what you want to do, it does not ask and cannot ask. The typical example is naturally Bitcoin, and various second-coins, such as Litecoin, etc., and the Yishu Coin introduced in this book is also. This level is where developers gather at this stage, which shows that cryptocurrency is still in its infancy.

In terms of the technologies used, the protocol layer mainly includes four aspects: network programming, distributed algorithms, encryption signatures, and data storage technology. Network programming ability is the main consideration for choosing a programming language, because distributed algorithms are basically business logic implementations, and any language can do it. Encryption signature technology is a direct and simple use (please read the relevant encryption and decryption articles in the book, it is not recommended to play freely, there is no excessive coding logic), and database technology is also mainly at the usage level. Only the implementation of point-to-point networks and concurrent processing are the difficulties of development, so people are particularly fond of languages ​​with strong network programming capabilities and simple concurrent processing. Therefore, Nodejs has gradually become more popular in developing blockchain applications, and the Go language is also gradually rising.

In the above architecture design diagram, I further divided this layer into the storage layer and the network layer. Data storage can be relatively independent and has greater freedom of choice, so it can be discussed separately. The principle of selection is nothing more than performance and ease of use. We know that the overall performance of the system mainly depends on the I/O performance of the network or data storage. There is not much room for network I/O optimization, but the I/O of local data storage can be optimized. For example, Bitcoin chose Google's LevelDB. It is said that this database has good read and write performance, but many functions need to be implemented by developers themselves. At present, a major problem that plagues the industry is that the transaction processing volume of cryptocurrency is far less than that of the current centralized payment system (banks, etc.). In addition to I/O, all-round breakthroughs are needed.

Distributed algorithms, encrypted signatures, etc. must be used in the process of realizing peer-to-peer networks, so they are naturally matters of the network layer, and are also the focus and difficulty of coding. The entire book "Developing Cryptocurrency with Nodejs" basically shares this part of the content. Of course, there are also separate implementations of peer-to-peer networks, separating the logic of node search, data storage and verification, and putting consensus algorithms, encrypted signatures, data storage and other operations together to form the core layer. No matter how they are combined, these two parts are the most core and lowest parts, and are the content of the protocol layer.

Extension Layer

This level is similar to the driver of a computer, which is to make blockchain products more practical. There are two types of trading markets. One is various trading markets, which are important channels for fiat currency to cryptocurrency. They are simple to implement, quick to make money, low cost, but also high risk. The second is the expansion implementation in a certain direction. For example, based on the Yishu side chain, customized services can be provided for third-party publishers, forum websites and other content producers. It is particularly worth mentioning that the concept of "smart contract" that everyone hears the most is a typical application development at the expansion level. The so-called "smart contract" is a "programmable contract", or "contract intelligence". The "intelligence" here refers to the intelligence in execution, that is, when a certain condition is met, the contract is automatically executed, such as automatic transfer of securities, automatic payment, etc. There is no mature product yet, but it is undeniable that this will be an important development direction of blockchain technology.

There are no restrictions on the technologies used in the extension layer, which can include many of the above-mentioned distributed storage, machine learning, VR, Internet of Things, big data, etc. The choice of programming language can be more free, because it can be completely separated from the protocol layer, and the programming language can also be different from the development language used in the protocol layer. In development, except for interacting with the protocol layer during transactions, try not to mix it with the development of the protocol layer at other times. This layer is closer to the application layer, and can also be understood as the server in the B/S architecture product. This is not only more scientific in architectural design, making blockchain data smaller and the network more independent, but also ensures that the development of the extension layer is not constrained.

From this perspective, blockchain can be used to develop any type of product, not just in the financial industry. In the future, as the underlying protocol becomes more complete, any product that requires third-party payment can easily use blockchain technology; any information that requires confirmation of ownership, credit investigation, and traceability can be achieved with the help of blockchain. I personally think that this goal should be achieved soon.

Application Layer

This level is similar to various software programs in computers. It is a product that ordinary people can really use directly. It can also be understood as the browser side (Browser) in B/S architecture products. Applications at this level are almost blank at present. The market is in urgent need of such applications to ignite the market and form a real expansion trend, so that blockchain technology can quickly enter ordinary people and serve the public. The various light wallets (clients) used by everyone should be regarded as the simplest and most typical applications at the application layer. Soon, Yishu will launch a document collaboration tool based on the Yishu network, which is a typical product at the application layer.

Limited by the current development of blockchain technology, Yishu can only start from the protocol layer and aim at the application layer, while providing strong support for the extension layer for third-party developers. Doing so can not only avoid greed, but also avoid failure to land, which is a truly rational development route. Because pure development of the protocol layer or extension layer cannot truly understand and verify the application layer, it will be out of touch with reality and make it difficult for third-party developers to use it. If only the application layer is considered, there will be no truly solid and easy-to-use protocol layer or extension layer products on the market. Therefore, we have to make full efforts, adopt a completely open source and open attitude, and use the power of the community to work together to do something meaningful, which can also be regarded as a little technical accumulation and contribution to the development of China's blockchain technology.

Programming Implementation

Many people are used to combining their own technical background to understand the above architecture design. Here, we briefly introduce several products in combination with specific programming languages ​​for reference only.

(1) C/C++

These two languages ​​are insurmountable. When any developer encounters a bottleneck, they will basically be found. Naturally, they should be the first to be introduced. At the same time, the originator of blockchain technology, Bitcoin (protocol layer), was developed in C++, and so far, there is no blockchain product more successful than Bitcoin. Therefore, no matter what language you use to develop, you should first study Bitcoin before officially entering this industry. The official Bitcoin client wallet uses Qt, and third-party wallets are developed in Python. In particular, many third-party development libraries (Api packages) are designed by Nodejs. The architecture of Bitcoin is basically the same as the above architecture design. In addition, because the consensus algorithm uses the proof of work mechanism (PoW: Proof of work), there are some special mining processes. Other competing coins are directly derived from Bitcoin branches, so the programming language is the same, and there may be improvements in specific technology selection and technical implementation. For example, Litecoin uses other encryption algorithms.

Official website: https://bitcoin.org/

Source code repository: https://github.com/bitcoin
(2) Nodejs/Javascript

The Nodejs platform has powerful network programming capabilities and the simplicity and speed of the js scripting language, so it is naturally indispensable in the blockchain field. Yishu is such a blockchain product. Yishu Coin is its protocol layer, which uses the famous express development framework and is developed based on the http protocol. At the same time, it adopts the Delegated Proof of Stake (DPoS) mechanism. The algorithmic improvements make it lighter when processing transactions and greatly improve its processing power. It provides a powerful collaboration mechanism, which facilitates digital publishing and copyright protection; it expands the side chain function and can develop any decentralized application based on it, thus providing a lot of convenience for professional authors, bloggers and developers. The book "Nodejs Development of Cryptocurrency" fully shares its source code, from the basic concepts of blockchain to code implementation, from basic principles to development design ideas, and has made a relatively detailed exploration. So far, there are very few books that explain the implementation of blockchain technology in-depth from the protocol level. This is one of them.

Official website: http://ebookchain.org/

Source code repository: https://github.com/Ebookcoin
(3) Python

If you are a Python language enthusiast, I suggest you study the Python implementation of Ethereum. Although it has caused a lot of controversy due to The Dao incident, it is still worth learning from the perspective of technical implementation. Ethereum is officially positioned as a platform for developing and managing distributed applications. Its main focus is "smart contracts" and it has customized a programming language Solidity for it. The core of Ethereum is the Ethereum Virtual Machine (EVM), which allows users to create operations as they wish. Ethereum provides implementations in multiple languages ​​such as Go, Java, and Python. The Python-based implementation mainly includes three parts: Pyethapp is the client part; pyethereum is the core library, which implements functions such as blockchain, Ethereum simulator, and mining; pydevp2p is a peer-to-peer network library, which implements functions such as node discovery, contract code transmission, and encrypted signatures. The combination of these three is a complete blockchain implementation, and the latter two core libraries together constitute the protocol layer. In addition, go-ethereum is a complete implementation of the go language; Ethereum(J) is a pure Java implementation, which is provided as a library that can be embedded in any Java/Scala project. On the client side, there are also implementations in languages ​​such as Rust, Ruby, and Javascript.

Official website: https://ethereum.org/

Source code repository: https://github.com/ethereum/pyethapp
(4) Go

In the multi-core era, Go language is very popular. It allows you to easily achieve high concurrency in a synchronous way, especially in the fields of distributed systems and network programming. Therefore, there are many projects using Go language in the field of blockchain development. Among them, Hyperledger, led by the Linux Foundation, is one of them. The name of the version library is Fabric. The project attempts to create an open distributed ledger standard for a new generation of transaction applications, supporting permissioned blockchains (this method may not reproduce the strong network effect of Bitcoin). Fabric's development environment is built on a VirtualBox virtual machine. The deployment environment can be self-built or directly deployed on BlueMix. The deployment method can be dockerized and supports the development of smart contracts in Go and JavaScript. It uses the PBFT distributed algorithm, uses gRPC for P2P communication in network programming, and uses Protocol Buffer to serialize the data structure to be transmitted. In terms of architectural design, Fabric may be different from blockchain products such as Bitcoin, but the above basic components are still indispensable.

Official website: https://www.hyperledger.org/

Source code repository: https://github.com/hyperledger
Other programming languages, such as C#, also have specific examples, which are not listed here. In short, for different programming languages, the specific coding or architecture design may be different, even very different, but the technology used in the protocol layer has not changed much. Among them, network programming is the focus and difficulty. Most of them do not have ready-made frameworks available, and they are designed and developed using the libraries provided by the programming language itself. Therefore, it is relatively low-level and very challenging for developers to code.

Knowledge Graph

Following the above analysis, we can already understand what blockchain is and how to implement it. By the way, if we sort out the programming technical knowledge involved, it will naturally become much clearer.

Based on my personal understanding, I divide the knowledge related to blockchain into the following five aspects:

(1) Basic knowledge

Blockchain is a new technology, and it is associated with a large number of new concepts and theories behind it. Although this knowledge is not directly reflected in the code, it is the basic knowledge to understand blockchain and master blockchain technology. Therefore, it should become an indispensable part of blockchain technology. This part starts with the basic concepts and describes the working principles, which can cover all the basic knowledge of blockchain.

(2) Technical Implementation

Blockchain is a technology, but from the above analysis, it can be seen that it should be an architecture application, and the implementation of the architecture should be the core of our knowledge base. As you can see, for any blockchain product, the protocol layer must include four parts: peer-to-peer network, encryption signature, data storage, and distributed algorithm. The application layer must also provide basic applications such as wallets and client browsers. Therefore, it is reasonable to separate this part.

In the extension layer, blockchain technology can be connected to various applications, such as finance, Internet of Things, network security, copyright protection, e-commerce, etc. Many existing technologies can be used here. However, how to combine with blockchain and how to achieve cross-industry use are naturally the research topics of this part. Therefore, the technologies listed or involved here should be classified as an important part of technology implementation.

(3) Development environment

Blockchain is a combination of multiple technologies and has its own complexity. Some applications are highly dependent on the development environment. Development tools and environment building are important for developers to get started quickly.

(4) Project practice

It is said that in just a few years, there are already thousands of blockchain products in the world, many of which are innovative applications. Some excellent open source products and project practices are the best learning and research materials.

(5) Development documentation

Needless to say, each product will have its own development documentation. Another thing is that some resources compiled by caring developers can help us save a lot of query time.

When I was thinking about this knowledge system, I was mainly thinking about whether readers could quickly master blockchain technology and eventually develop and implement a blockchain product by following these tags to read articles. In addition, I also deliberately avoided words related to specific programming languages ​​and specific fields. The only thing that can be distinguished is the corresponding article tags under these nodes. Therefore, these classifications appear to be very neutral. I also considered using the implementation of specific fields such as Bitcoin, competitive coins, smart contracts, digital assets, and smart assets as a classification method, but I was afraid that it would limit the readers' thinking. At the same time, with the development of blockchain, this map will continue to be modified. Here, I would like to appeal to the friends who read this article to provide your valuable opinions, so that we can make this knowledge classification map about blockchain more scientific and reasonable, and more convenient to use.

Summarize

In this article, we describe the basic architecture of blockchain technology. It is important to emphasize that this is only one implementation method, and not all blockchain products are the same. We also look forward to more innovations and believe that they will appear. Programming implementation lists several programming languages ​​and their typical products. Because the protocol layer technology is relatively low-level, there are not many ready-made frameworks to introduce or discuss. At the same time, the specific technical details cannot be listed clearly in a few lines. Fortunately, these products are open source products. You can further check the corresponding product source code based on your own technical background and quickly understand the mystery.
About the author: Zhu Zhiwen, founder of Yishu, guest editor of CSDN blockchain knowledge base, creator and initiator of China Blockchain Club, loyal fan of Bitcoin, evangelist of blockchain technology, and author of the masterpiece "Developing Cryptocurrency with Nodejs".