What is big data?

Technology supporting big data

NoSQL (Not only SQL)

NoSQL is defined as a term that represents databases other than RDBMS (relational database management system), which is a matrix-type storage format, such as MySQL and PostgreSQL. NoSQL usually stores data only in key-value pairs and specializes in data read/write speed. Since data can be held in a simple format, it is easy to store various unstructured data. It can also be used on cloud services such as Alibaba Cloud’s Table Store.

MapReduce and Hadoop

In October 2003, Google released a distributed file system called Google File System, and in December 2004, a big data processing technology called MapReduce. The technology presented in this paper will bring about a major change in data processing technology ranging from several terabytes to several petabytes, which was considered difficult until then. Although the implemented code was not released by Google, the Apache Nutch project, a pioneer of OSS, adopted this technology, and around July 2005, the project became independent as Apache Hadoop. In 2006, Yahoo adopted Hadoop for the backend of its search engine, and the project rapidly accelerated, after which it was used by many IT companies such as Facebook, Twitter, and LinkedIn.

The mechanics of Hadoop were revolutionary at the time, greatly reducing the cost of structuring and storing large amounts of data. However, Hadoop does not support high-speed query processing, leaving issues for scenes that require real-time performance.

Deep learning

Deep learning is famous for being the spark that sparked the third AI boom, but the origin of the technology itself dates back to 1957. A method called perceptron, which mimics the human brain, was devised, and in 2006, a method of stacking networks in multiple layers was devised. However, the amount of computation (CPU, GPU) of computers at that time was not enough to demonstrate sufficient performance. From around 2010, the hardware will start to have enough performance to withstand the technology. After that, deep learning surprises the world with various news such as the identification of cat images and victory over the world champion of Go. Behind it was the big data that the algorithm learned.

Big data and personal information

Revision of the Personal Information Protection Law

As mentioned above, with the rapid development of technology, the amount, quality, and type of information handled in the world have changed greatly. In 2017, the Japanese government will enforce the revised Act on the Protection of Personal Information to ensure that information is used appropriately in such circumstances. This law stipulates the addition of personal identification codes, the establishment of special care-required personal information, the securing of traceability, and the establishment of anonymously processed information as a means of deregulation.

Among these items, the items related to anonymously processed information have a particularly significant meaning in the use of big data. By processing information that can be used as personal identifiers from the data held by companies, ensuring a safety management system, and announcing the use of anonymously processed information, companies will no longer be subject to rules regarding personal information.

GDPR and Cookie Restrictions

The trend of personal information protection is a big trend not only in Japan but also in the world. In Europe, GDPR (EU General Data Protection Regulation) was enacted in 2018. As a result, IP addresses and cookies will also be treated as personal information. In January 2020, Google announced that within two years, Google will start advertising on the Google Chrome web browser. We are announcing that we are phasing out support for third-party cookies on Purpose.

Secure computing and Secure AI

Even if a company wants to request analysis of its data, such as personal DNA and health information, there are cases where it is not possible to provide information to the outside due to the risk of information leakage. On the other hand, research on a method called secure computation, which analyzes data while it is encrypted, has been progressing for the past few years, and demonstration cases are starting mainly in the financial field. If this method is established, a scheme will be established that can provide a large amount of data that was not provided for analysis, so it is expected that the learning accuracy of AI will improve dramatically.

Summary

With IoT and edge computing, more and more information will be digitized and accumulated as big data in the future. In addition, with the spread of the cloud, an environment is being prepared in which companies can easily analyze the acquired data. However, as laws and regulations on personal information are being strengthened, there is a need for even more careful handling of information. In providing value and ensuring safety using big data, companies need to face both sides.