The era of data explosion: What is the mission of distributed networks?

Humanity’s data explosion

Yet, despite the almost incalculable amount of data that exists today, its rate of growth is even more staggering. Back in 2012, IBM calculated that 90% of the world’s data had been created in the previous two years. Since then, the total amount of data in the world has continued to grow exponentially, and it looks like this trend will continue. In fact, IDC predicts that in the next three years, humanity will create more data than in the previous thirty years.

The obvious question is: what has changed? Why are we suddenly generating more data than ever before? Clearly, smartphones are part of this story. Everyone now literally carries a mobile computer in their pocket, dwarfing previous generations of desktop computers. These machines are constantly connected to the internet, even when they’re idle, constantly receiving and transmitting data. The average American Generation Z adult, born between 1996 and 2010, unlocks their phone 79 times a day, or about every 13 minutes. The always-on nature of these devices results in a flood of new data, with 500 million new tweets, 4,000 megabytes of Facebook posts, and 65 billion new WhatsApp messages entering cyberspace every 24 hours.

However, smartphones are only the most visible part of the new data world. You might think that video platforms such as Netflix and YouTube account for the largest share of global data, but in fact, the share of data generated by consumers as a whole is only about 50%, and this proportion is expected to gradually decline in the coming years. So, what is the rest?

The rise of the Internet of Things and connected devices has further expanded our global data footprint. In fact, the fastest year-over-year growth is in a category of information known as embedded and productivity data. This is information from sensors, connected machines, and automatically generated metadata that exists behind the scenes, beyond the visibility of the end user.

Take self-driving cars, for example. They use technologies such as cameras, sonar, lidar, radar, and GPS to monitor traffic conditions, plot routes, and avoid danger. According to Intel's calculations, an average self-driving car using current technology will generate 4 trillion terabytes of data per day. To put this into perspective, the amount of data generated by a single car per day is equivalent to that of nearly 3,000 people.

On the one hand, this data will help schedule service intervals and diagnose technical problems most efficiently. It can also be used as part of a distributed system to coordinate traffic flows and minimize energy consumption in a particular city. Finally, and perhaps most importantly in the short term, it will be crucial in resolving legal disputes in the event of an injury or accident.

Self-driving cars are just one small part of the picture. According to McKinsey & Company, the percentage of businesses using IoT technology rose from 13% to 25% between 2014 and 2019, and the total number of devices worldwide is expected to reach 43 billion by 2023. From the Industrial Internet of Things to entire smart cities, the future economy will have a huge number of connected devices that will generate potentially highly sensitive and even critical data.

Is the end of Moore's Law imminent?

There are two factors to consider, both of which point to the growing utility of decentralized networks. First, while we have more data than ever before to address global challenges like climate change, financial instability, and airborne viruses like COVID-19, we may be approaching a difficult technological boundary: the limit of information that can be processed in real time by centralized computers. While the amount of data has grown exponentially in recent years, the ability to process it has not grown at the same rate.

In the 1960s, Intel co-founder Gordon Moore proposed Moore's Law, which states that as the number of transistors on a microchip doubles every two years, computing power will grow at a corresponding rate. But Moore himself admitted that this is not a scientific law; it is more like a fleeting statistical observation. In 2010, he acknowledged that with transistors now approaching the size of atoms, computer processing power will reach a strict technological limit in the next few decades. After that, more core components can be added to the processor to increase speed, but this will increase the size, cost and power consumption of the device. So to avoid the bottleneck effect, we need to find new ways to monitor and respond to data.

The second factor to consider is cybersecurity. In an increasingly connected world, millions of new devices are coming online. The data they provide can impact the control of power grids, the management of healthcare, and the regulation of transportation. As a result, edge security—the safety of data that resides outside the core of the network—becomes critical. This presents a complex challenge for cybersecurity experts, as the many different devices and protocol combinations present new attack surfaces and opportunities for intrusion.

Learning from Nature’s Networks

If centralized processing is too slow and insecure for the data-rich economies of the future, what are the alternatives? Some experts have been looking to nature for inspiration, arguing that we should move from a top-down model of monitoring and responding to data to a bottom-up model. Take ant colonies, for example. While each individual ant has relatively modest intelligence, collectively the colony manages to create and maintain a complex, dynamic network of foraging paths that can connect multiple nests and transient food sources. They do this by following a few simple behaviors and responding to stimuli in the local environment, such as pheromone trails from other ants. Over time, however, evolution has unearthed instincts and behaviors at the individual level, resulting in a system that is remarkably efficient and robust at the macro level. If a path is damaged by wind or rain, the ants will find a new one, without any individual ant even being aware of the overall goal of maintaining the network.

What if the same logic were used to organize computer networks? Similar to an ant colony, in a blockchain network, many nodes with modest processing power combine to produce a global result that is greater than the sum of its parts. Just as in nature where instincts and behavior are crucial, the rules that govern how nodes interact are also crucial in determining how successfully a network will achieve at a macro level.

Aligning the incentives of each individual participant and forming a mutually beneficial network takes nature thousands of years to foster. Therefore, it is also a difficult challenge for human designers of decentralized networks. However, genetic mutations in animals are essentially random, and we have an advantage in that we can purposefully simulate and design incentive mechanisms to achieve a common overall goal.

With such a carefully designed incentive structure, a distributed network can greatly enhance edge security. Just as an ant colony’s pathfinding network can continue to work even if an ant is lost or dead, a decentralized network is equally powerful and can remain fully functional even if a single node crashes or goes offline. In addition, no single node in the entire network needs to process or understand all the data. In this way, some researchers believe that we can create an economic incentive structure to automatically detect and respond to common challenges in a distributed manner.

in conclusion

The total amount of data produced by humanity is exploding, and the ability to monitor and respond to that data using centralized computer networks is approaching its limits. For this reason, decentralized networks are uniquely suited to the challenges ahead. Much research, testing, and experimentation remains to be done, but the fundamental vibrancy of blockchain and the utility of the underlying technology have already been demonstrated. As humanity moves toward a data-rich, hyper-connected world, distributed networks may play an important role in reaping the greatest economic and social benefits.

Original link : https://cointelegraph.com/news/the-role-of-decentralized-networks-in-a-data-abundant-hyperconnected-world