Explain in detail what you don't know about "big data"

Explain in detail what you don't know about "big data"

The word 20 12 big data has been mentioned more and more. People use it to describe and define the massive data generated in the era of information explosion, and to name the related technological development and innovation. It once appeared on the cover of the Wall Street Journal column, entered the White House official website news, appeared in some Internet-themed lecture salons in China, and was even written into the investment recommendation report by sensitive securities companies.

I. Background of Big Data

The word 20 12 big data has been mentioned more and more. People use it to describe and define the massive data generated in the era of information explosion, and to name the related technological development and innovation. It once appeared on the cover of the Wall Street Journal column, entered the White House official website news, appeared in some Internet-themed lecture salons in China, and was even written into the investment recommendation report by sensitive securities companies.

The rapid expansion of data determines the future development of enterprises. Although enterprises may not be aware of the hidden dangers caused by explosive data growth, people will become more and more aware of the importance of data to enterprises as time goes by. The era of big data poses new challenges to human data control ability, and also provides unprecedented space and potential for people to gain deeper and more comprehensive insight.

It was McKinsey, a world-renowned consulting firm, that first proposed the arrival of the era of big data. McKinsey said: "Data has penetrated into every industry and business functional field today and become an important factor of production. The mining and application of massive data indicates a new wave of productivity growth and the arrival of consumer surplus. " "Big data" has existed in the fields of physics, biology, environmental ecology, military, finance, communications and other industries for some time, but it has attracted people's attention because of the development of the Internet and information industry in recent years.

Big data in the Internet industry refers to the phenomenon that Internet companies generate and accumulate user network behavior data in their daily operations. The scale of these data is so huge that it can't be measured by G or T. The starting unit of big data measurement is at least P( 1000 t), E (1000t) or Z (1000t).

Second, what is big data?

In the field of information technology, there have been concepts such as "massive data" and "large-scale data", but these concepts only pay attention to the data scale itself and fail to fully reflect the data processing and application needs under the background of data explosion. The new concept of "big data" not only refers to large-scale data objects, but also includes the processing and application activities of these data objects, which is the unity of data objects, technologies and applications.

1, bigdata, that is, huge data, refers to information that involves so much data that it can't be captured, managed, processed and sorted by current mainstream software tools, so that it can achieve a more positive purpose and help enterprises make business decisions within a reasonable time. Big data objects may be actual and limited data sets, such as a database held by a government department or enterprise, or virtual and unlimited data sets, such as all information on Weibo, WeChat and social networks.

Big data is a massive, high-growth and diversified information asset, which needs a new processing mode to have stronger decision-making, insight and discovery, and process optimization capabilities. In terms of data, "big data" refers to information that cannot be processed or analyzed by traditional processes or tools. It defines those data sets that are beyond the normal processing range and size, forcing users to adopt unconventional processing methods.

Amazon Web Services (AWS) and big data scientist JohnRauser mentioned a simple definition: Big data is any massive data that exceeds the processing power of a computer. R&D team's definition of big data: "Big data is the biggest propaganda technology and the most fashionable technology. When this phenomenon occurs, the definition becomes very confusing. " Kelly said: "Big data may not contain all the information, but I think most of it is correct. Part of the view on big data is that it is so big that analyzing it requires multiple workloads, which is the definition of AWS.

2. Big data technology refers to the technical ability to quickly obtain valuable information from all kinds of big data, including data collection, storage, management, analysis and mining, visualization and other technologies and their integration. Technologies suitable for big data include MPP database, data mining power grid, distributed file system, distributed database, cloud computing platform, Internet, extensible storage system and so on.

3. Big data application refers to the integration and application of big data technology to a specific big data collection to obtain valuable information. For different businesses in different fields and enterprises, or even the same business of different enterprises in the same field, the big data technology and big data information system adopted may be quite different due to different business requirements, data collection and analysis, and mining objectives. Only by adhering to the simultaneous development of the trinity of "object, technology and application" can we fully realize the value of big data.

When your technology reaches the limit, it is the limit of data. "Big data is not about how to define it, the most important thing is how to use it. The biggest challenge is which technologies can make better use of data and how to apply big data. Compared with traditional databases, the rise of open source big data analysis tools such as Hadoop and the value of these unstructured data services.

Third, the types and value mining methods of big data

1, the types of big data can be roughly divided into three categories:

1) traditional enterprise data: including customer data, traditional ERP data, inventory data and account data of CRMsystems.

2) Machine generated/sensor data: including CallDetailRecords, smart meters, industrial equipment sensors, equipment logs (usually digital exhaust), transaction data, etc.

3) Socialdata: including user behavior records, feedback data, etc. Social media platforms such as Twitter and Facebook.

2. There are four main methods for big data mining business value:

1) subdivide customer groups, and then customize special services for each group.

2) Simulate the real environment, explore new demands and improve the return on investment.

3) Strengthen departmental contact and improve the efficiency of the whole management chain and industrial chain.

4) Reduce service costs, find hidden clues and innovate products and services.

Fourth, the characteristics of big data.

The industry usually uses four V's (namely, volume, category, value and speed) to summarize the characteristics of big data. Specifically, big data has four basic characteristics:

1 is a huge amount of data.

A large amount of data refers to a large data set, which is generally around 10TB. However, in practical application, many enterprise users put multiple data sets together, which has formed a PB-level data volume. According to Baidu's data, its new homepage navigation needs to provide more than1.5pb (1Pb =1024tb) every day, and if printed, it will exceed 500 billion A4 sheets. It has been confirmed that up to now, the data volume of all printed matter produced by human beings is only 200PB.

2. There are various types of data.

There are many kinds of data, which come from various data sources, and the types and formats of data are increasingly rich, which has broken through the previously defined category of structured data, including semi-structured and unstructured data. Nowadays, data types are not only text, but also pictures, videos, audio, geographic information and other types of data, and personalized data accounts for an absolute majority.

3, the processing speed is fast.

In the case of huge amount of data, real-time data processing can also be realized. Data processing follows the "1 second law", and high-value information can be quickly obtained from all kinds of data.

4. It is high-value authenticity and low density.

The data is highly authentic. With people's interest in new data sources such as social data, enterprise content, transaction and application data, the limitations of traditional data sources have been broken, and enterprises increasingly need effective information power to ensure their authenticity and security. Take video as an example. An hour of video, in the process of uninterrupted monitoring, may only have one or two seconds of useful data.

Verb (abbreviation for verb) The role of big data

1, the processing and analysis of big data is becoming the node of the new generation of information technology integrated application.

Mobile Internet, Internet of Things, social networks, digital home, e-commerce and so on are the application forms of the new generation of information technology, and these applications constantly produce big data. Cloud computing provides a storage and computing platform for these massive and diverse big data. Through the management, processing, analysis and optimization of data from different sources, the results are fed back to the above applications, thus creating great economic and social value.

Big data has the energy to promote social change. But releasing this energy requires strict data governance, insightful data analysis and an environment to stimulate management innovation (RamayyaKrishnan, Dean of Hindes College of Carnegie Mellon University).

2. Big data is the new engine for the sustained and rapid growth of the information industry.

New technologies, new products, new services and new formats will emerge in the big data market. In the field of hardware and integrated equipment, big data will have an important impact on the chip and storage industry, and will also give birth to markets such as integrated data storage and processing servers and memory computing. In the field of software and services, big data will lead to the development of fast data processing and analysis, data mining technology and software products.

3. The use of big data will become a key factor to improve core competitiveness.

The decision-making of all walks of life is changing from "business-driven" to "data-driven". The analysis of big data allows retailers to grasp the market dynamics in real time and respond quickly; It can provide decision support for merchants to formulate more accurate and effective marketing strategies; Can help enterprises to provide consumers with more timely and personalized services; In the medical field, it can improve the accuracy of diagnosis and the effectiveness of drugs; In the public sector, big data has also begun to play an important role in promoting economic development and maintaining social stability.

4. In the era of big data, the methods and means of scientific research will undergo major changes.

For example, sampling survey is the basic research method of social science. In the era of big data, we can monitor and track the massive behavior data generated by research objects on the Internet in real time, conduct mining and analysis, reveal regular things, and put forward research conclusions and countermeasures.

Sixth, the commercial value of big data.

1. Segmentation of customer base

"Big Data" can subdivide customer groups and then take unique actions for each group. Marketing and service for specific customer groups has always been the pursuit of businesses. The massive data in cloud storage and the analysis technology of "big data" make it possible to segment consumers in real time and with high cost performance.

Step 2 simulate reality

Use "big data" to simulate the real situation, explore new demands and improve the return on investment. Nowadays, more and more products are equipped with sensors, and the popularity of cars and smart phones makes the data that can be collected explode. Social networks such as blogs, Twitter, Facebook and Weibo are also generating huge amounts of data.

Cloud computing and "big data" analysis technology enable enterprises to store and analyze these data and transaction behavior data in real time, which is very cost-effective. Trading process, product use and human behavior can all be digitized. "Big Data" technology can integrate these data for data mining, so in some cases, we can judge which scheme has the highest return on investment under different variables (such as different promotion schemes in different regions) through model simulation.

3. Improve the return on investment.

Improve the sharing of "big data" results in relevant departments and improve the return on investment of the entire management chain and industrial chain. Departments with strong "big data" capabilities can share the results of "big data" with departments with weak "big data" capabilities through cloud computing, the Internet and internal search engines, and help them create business value by using "big data".

4. Lease of data storage space

Both enterprises and individuals have the demand for massive information storage. Only by storing these data correctly can we further explore their potential value. Specifically, this business model can be subdivided into two categories: personal file storage and enterprise users. Mainly through the easy-to-use API, users can easily put all kinds of data objects in the cloud, and then charge according to the usage like water and electricity. At present, many companies have launched corresponding services, such as Amazon, Netease and Nokia. Operators have also launched corresponding services, such as China Mobile's Cai Yun service.

5. Manage customer relationships

The purpose of customer management application is to deeply analyze and understand customers from different angles according to their attributes (including natural attributes and behavioral attributes), so as to increase new customers, improve customer loyalty, reduce customer churn rate and increase customer consumption. For small and medium-sized customers, specialized CRM is obviously big and expensive. Many small and medium-sized enterprises use Fetion as their main CRM. For example, add old customers to Fetion group, release new product announcements and special sales notices in the circle of friends, and complete pre-sales and after-sales services.

6. Personalized and accurate recommendation

Within operators, it is common to recommend various services or applications according to users' preferences, such as app store software recommendation and IPTV video program recommendation. After intelligent analysis algorithms such as association algorithm, text abstract extraction and sentiment analysis, it can be extended to commercial services, and data mining technology can be used to help customers carry out precise marketing. Future profits can come from the share of customer value-added parts.

Take the daily "spam messages" as an example, the information is not all "junk" because the recipient doesn't need it and is regarded as junk. After analyzing the user behavior data, we can send the needed information to those who need it, so that "spam messages" can become valuable information. In McDonald's in Japan, users download coupons on their mobile phones and then go to restaurants to pay with the mobile wallet of operator DoCoMo. Operators and McDonald's collect relevant consumption information, such as what hamburgers they often buy, which stores they go to, and how often they consume, and then accurately push coupons to users.

7. Data search

Data search is not a new application. With the advent of the era of "big data", people's demand for real-time and all-round search is getting stronger and stronger. We need to be able to search various social networks, user behaviors and other data. Its commercial application value lies in the connection of real-time data processing with analysis and advertising, that is, the real-time advertising business and social service of in-app mobile advertising.

The information of users' online behavior mastered by operators makes the obtained data "have more comprehensive dimensions" and have more commercial value. Typical applications such as "Pangu Search" of China Mobile.

Seven, the important impact of big data on the economy and society

1, which can promote the realization of huge economic benefits.

For example, the contribution to the growth of retail net profit in China, and the reduction of R&D and assembly costs of manufacturing products. It is estimated that in 20 13 years, global big data will directly and indirectly drive information technology expenditure to reach120 billion US dollars.

2. It can promote the improvement of social management level.

The application of big data in the field of public services can effectively promote the development of related work, improve the decision-making level, service efficiency and social management level of relevant departments, and generate great social value. By analyzing the traffic flow data collected in real time, many cities in Europe can guide drivers to choose the best route, thus improving urban traffic conditions.

3. Without high-performance analysis tools, the value of big data cannot be released.

We must keep a clear understanding of the application of big data, neither blindly believe its analysis results nor deny its important role because it is not completely accurate.

1) Due to various reasons, the data objects to be analyzed and processed will inevitably contain all kinds of wrong data and useless data, while the technologies such as data analysis and artificial intelligence, which are the core of big data technology, are not yet fully mature, so it is impossible to require the computer to complete the big data analysis and processing results to be completely accurate. For example, by analyzing the search content of hundreds of millions of users, Google can predict the outbreak of influenza faster than professional organizations, but this prediction is inaccurate many times due to the interference of useless information on Weibo.

2) It must be clearly positioned. The focus of the role and value of big data is to guide and stimulate the innovative thinking of big data users and assist decision-making. Simply put, when you deal with a problem, usually people can think of one method. Big data can provide ten reference methods, even if only three are feasible, it will triple the thinking of solving the problem.

Therefore, objectively understanding and giving play to the role of big data, without exaggerating or shrinking, is the premise of accurately understanding and applying big data.

Eight. abstract

Regardless of whether the core value of big data is prediction or not, the decision-making model based on big data has brought profits and word of mouth to many enterprises.

1. From the value chain analysis of big data, there are three modes:

1) holding big data, but not making good use of it; Typical are financial institutions, telecommunications industry, government agencies and so on.

2) I don't have data, but I know how to help people with data use it; Typical IT consulting and service enterprises, such as Accenture, IBM and Oracle Bone Inscriptions.

3) There are both data and big data thinking; Typical ones are Google, Amazon, MasterCard, etc.

2. The most valuable things in the field of big data in the future are two things:

1) People with big data thinking can turn the potential value of big data into real benefits;

2) Business areas that have not been touched by big data. These are unexplored oil wells and gold mines, also known as the blue ocean.

Big data is a typical field where information technology and professional technology, information technology industry and various industries are closely integrated, with strong application demand and broad application prospects. In order to grasp the new opportunities brought by this emerging field, it is necessary to continuously track and study big data, continuously improve the cognition and understanding of big data, adhere to the synergy between technological innovation and application innovation, accelerate the development and utilization of big data in various fields of economy and society, and push the data application demand and application level of countries, industries and enterprises into a new stage.