Application and Challenges of Big Data in Financial Industry

Application and Challenges of Big Data in Financial Industry

A has four basic characteristics.

The financial industry is basically the most dependent on data among all industries in the world, and it is also the easiest to realize data realization. Bloomberg, the world's largest financial data company, was founded in 198 1, before the concept of "big data" appeared. The initial product of Bloomberg is Investment Market System (IMS), which mainly provides real-time data and financial analysis for various investors.

With the advent of the information age, Bloomberg, with a valuation of 1983 of only 1 100 million dollars, exchanged 30% of its shares for an investment of 30 million dollars from Merrill Lynch, and successively launched various products such as Bloomberg terminals, news, radio and television. 1996 Bloomberg bought back 10% of its shares from Merrill Lynch for $200 million with a value of $2 billion. In 2004, Bloomberg built a 246-meter-high skyscraper in the center of Manhattan. By the subprime mortgage crisis in 2008, Merrill Lynch was facing bankruptcy, and its remaining 20% stake in Bloomberg became a lifeline. Bloomberg used Merrill Lynch's crisis to redeem all its shares, and its valuation jumped to $22.5 billion. In 20 16, Bloomberg had 192 offices around the world, with15,000 employees, with an annual income of about10 billion and a valuation of about10 billion, exceeding the Wall Street benchmark Goldman Sachs with a market value of 65 billion dollars in the same year.

The concept of big data was formed around 2000 and was originally defined as a collection of massive data. 20 1 1 year, McKinsey & Company of the United States first proposed in the report "The Next Frontier of Big Data: Innovation, Competition and Productivity" that big data refers to data sets whose scale exceeds the collection, storage, management and analysis capabilities of typical database software tools.

Specifically, big data has four basic characteristics:

First, the large amount of data refers to a large data set, which is generally around 10TB. However, in practical application, many enterprise users put multiple data sets together, which has formed a PB-level data volume.

Second, the data types are large, coming from various data sources, and the types and formats of data are increasingly rich, which breaks through the previously defined structured data category and includes semi-structured and unstructured data. Nowadays, data types are not only text, but also pictures, videos, audio, geographic information and other types of data.

Third, the processing speed is fast, and the data can be processed in real time under the condition of huge data. Data processing follows the "1 second law", and high-value information can be quickly obtained from all kinds of data.

Fourth, the data is highly authentic. With the rise of new data sources such as social data, enterprise content, transaction and application data, the limitations of traditional data sources have been broken, and the authenticity and security of information are extremely important.

Compared with other industries, financial data has a close logical relationship and requires higher security, stability and real-time. It usually includes the following key technologies: data analysis, including data mining, machine learning, artificial intelligence and so on. , mainly used for customer credit, clustering, characteristics, marketing, product correlation analysis, etc. ; Data management, including relational and non-relational data, integration, data extraction, data cleaning and transformation; Data usage, including distributed computing, memory computing, cloud computing, stream processing, task configuration, etc. Data display, including visualization, historical flow and spatial information flow display, is mainly used for monitoring and early warning of financial product health, product development trend, customer value change, anti-money laundering and anti-fraud.

B reshape the new pattern of competition in the financial industry

After "internet plus", with the rapid rise of "Big Data+"in the world, the financial industry has quietly undergone the following changes:

The characteristics of big data have increased from "3 V" in traditional data to "5 V". On the basis of volume, speed and variety, the value and authenticity will be further improved. Authenticity includes the credibility, source and reputation, validity and auditability of data.

The financial industry is divided into business models according to business products. The traditional financial industry is divided into five categories: banking, securities, futures, insurance and funds. With the rise of big data industry and the development of mixed operation, the modern financial industry is divided into three categories: deposit and loan, investment and insurance.

The big data market has developed from monopoly to full market competition. The number of enterprises in the global big data market has increased rapidly, the differences between products and services have increased, the technical threshold has gradually decreased, and the market competition has become increasingly fierce. Industry solutions, computing and analysis services, storage services, database services and big data applications have become the five major market segments.

Big data forms a new economic growth point. According to Wikibon data, in 20 16, the global big data hardware, software and services market grew by 22%, reaching $2810 billion. It is estimated that by 2027, the compound annual growth rate of global big data hardware, software and services will be 12%, reaching about $97 billion.

Data and IT technology have replaced "repetitive" business work. Eurekahedge, a data service company, tracked 23 hedge funds and found that the total salary of five hedge fund managers was $654.38 billion or more. In the past 10 years, the physicist and mathematician "Kwank" who analyzed financial markets through mathematical models has always been the darling of hedge funds. In fact, big data+artificial intelligence is better at this. In 2000, the new york Stock Cash Trading Department of Goldman Sachs had 600 traders, but now there are only two, all of which are handled by machines. Experts say that in 10 years, Goldman Sachs will definitely have fewer employees than it does today.

The development of big data in the United States is at the forefront of the world. The US government declared: "Data is a valuable national capital and should be open to the public, not limited to the government system." As the source and innovation leader of big data, the development of big data in the United States has always been at the forefront of the world. Since the 20th century, the United States has promulgated a series of laws and regulations to make specific provisions on data collection, release, use and management. In 2009, the U.S. government launched the Data.gov government data open platform, which is convenient for developers in the application field to use the platform to develop applications, meet public needs or innovate and start businesses. 20 10 the us congress passed the update bill, which further improved the accuracy of data collection and the frequency of reporting. 20 12 in March, the Obama administration launched a big data research and development plan, and big data ushered in a new round of rapid development.

Britain is a European financial center, and big data has become one of its leading technologies. In 20 13, Britain invested 654.38+89 million pounds to develop big data. On 20 15, 73 million pounds was added to establish the data.gov.uk website of "British Database". In 20 16, more than 22,000 scientific and technological activities were held in London. In the same year, Britain invested more than 6.8 billion pounds in digital technology, while its revenue exceeded/kloc-0.7 billion pounds. In addition, the British Bureau of Statistics uses government resources to conduct a "virtual census", which alone saves 500 million pounds a year.

Establish an effective financial supervision system

As the core of modern digital technology, the soul of big data is prediction.

Discover and combat tax evasion, money laundering and financial fraud.

The annual economic loss caused by fraud in the world is about 3.7 trillion dollars, and the loss caused by fraud in enterprises is usually 5% of annual income. SAS Company is one of the largest software companies in the world. It cooperates with tax, customs and other government departments as well as banks, insurance, health care and other institutions around the world to effectively deal with increasingly complex financial crimes. For example, before issuing the license, it is necessary to test whether the customer has criminal records such as bribery and fraud. Through the analysis of the previous data, it is determined whether to issue loans or customs clearance. The system developed by SAS has been internationally recognized as the standard software for statistical analysis and is widely used in various fields. The British government used the big data detection behavior model to recover 20 billion pounds of tax evasion and fraud, and recovered billions of dollars in losses. Texas Capital Bank (TCBank), which was rated as the best bank in the United States by Forbes, has continuously invested in big data technology, and its anti-financial crime system has kept pace with the development of banks. In the past three years, its assets have increased from $9 billion to $2 1 billion. CZ, the third largest life insurance company in the Netherlands, relies on big data to detect fraudulent insurance and false claims, and blocks them in advance before payment, effectively reducing judicial relief after fraudulent insurance.

Big data risk control establishes customer credit scoring and monitoring system.

According to the statistics of American Association of Certified Fraud Auditors (ACFE), enterprises that lack anti-fraud control will suffer high losses. FICO, the mainstream personal credit scoring tool in the United States, can automatically compare the historical data of borrowers with the overall credit habits of all borrowers in the database, predict the behavior trend of borrowers, and evaluate their similarity with various bad borrowers. American SAS Company conducts real-time anti-fraud analysis by browsing, analyzing and evaluating the basic information of customers' bank accounts, historical behavior patterns and ongoing behavior patterns (such as transfer), and combining with intelligent rule engines (such as searching for customers to transfer money for the only user in emerging countries or conducting online transactions in new locations). ).

An Internet credit rating agency in the United States analyzes the information left by customers on social platforms such as Facebook and Twitter, conducts risk assessment on bank credit and insurance applicants, and sells the results to banks and insurance companies. , thus becoming a partner of many financial institutions.

Data integration is difficult.

Application of economic index forecasting system to analyze market trend

Using big data information technology, IBM has successfully developed the "Economic Index Forecasting System", which is refined and integrated based on a single data, predicts the trend by searching, counting and analyzing the words related to stock price indicators, such as "new orders" appearing in news, and then analyzes the relationship between it and stock price by combining other relevant economic data and historical data, so as to obtain the market forecast results.

Track the huge amount of information on social media and evaluate market changes.

Nowadays, Weibo, WeChat, forums, news comments and e-commerce platforms on search engines, social networks and smart phones generate hundreds or even hundreds of billions of words, audio and video, videos and data every day, covering manufacturers' trends, personal emotions, industry information, product experience, product browsing and transaction records, price trends and so on. , which contains great wealth value.

20 1 1 in may, 2008, DC market, a British hedge fund with a scale of $40 million, used big data to analyze the information content of Twitter, perceive market sentiment and guide investment. It made a profit in the first month, beating other hedge funds with a yield of 1.85%, with an average yield of only 0.76%.

A doctor at Pace University in the United States used big data to track the onlookers of Starbucks, Coca-Cola and Nike on social media, and compared their stock prices, proving that the number of fans on Facebook, Twitter and Youtube is closely related to the stock prices.

Provide a wide range of investment options and transaction switching.

Money Design, a Japanese personal investment and wealth management product, uses algorithm+artificial intelligence in the application of Theo, with a minimum threshold of $924. Users can trade and switch 1. 1.9 million stocks in 35 different currencies in 65 countries with an annual management fee of only 1%. Currency design can also automatically balance the account amount of users according to their investment objectives. It is estimated that more than $2 trillion will be invested in such products in 2020.

Using cloud database to provide accounting services for customers

Money Forward, a Japanese wealth management tool provider, provides cloud-based bookkeeping services, which can manage wages, collect and pay, send invoices and bills, and push new financial projects in a targeted manner. Its software system connects and integrates all kinds of accounts of 2,580 financial institutions, displays the current wealth status of users by using the intelligent dashboard of big data analysis, and can also analyze the past data of users and predict the future financial trajectory. At present, it has 500,000 merchants and 3.5 million individual users, and jointly developed a new APP with Yamaguchi Financial Group, with a market value of 2.5 trillion US dollars.

Customize differentiated products and marketing programs for customers.

Financial institutions urgently need to master more user information, and then build a 360-degree three-dimensional portrait of users, so as to carry out precise marketing, real-time marketing and smart marketing for segmented customers.

Some overseas banks analyze and calculate general life nodes around customers' "life events", which effectively stimulates customers' willingness to buy high-value financial products. For example, through big data analysis, an Australian bank found that customers who are about to give birth at home have the greatest potential demand for life insurance products, so it monitored the phenomenon that expectant mothers began to buy anti-abortion drugs and baby-related products through bank card data, identified families who are about to add babies, and accurately launched customized financial product packages, which received positive responses from customers, greatly improving the success rate compared with the traditional SMS group sending mode.

Give birth to and support artificial intelligence transactions.

Simmons, the king of quantitative investment, is recognized as the most profitable fund manager. Since 1988 founded Medalian Fund, the flagship product of Fuxing Technology Company, it has created an average annual net return rate of 35% in 20 years, which is higher than Soros 10% and higher than Warren Buffett 18%. In the ranking of hedge fund managers published annually by Alpha magazine, Simmons ranked first in the world with a net income of $65.438+0.5 billion in 2005, fifth with $65.438+0.7 billion in 2006, back to the top with $65.438+0.3 billion in 2007 and back to the top with $2.5 billion in 2008.

Promote the innovation of financial products and services.

E faces three major challenges.

At present, the data growth rate of various industries in the world is amazing, especially in key industries such as finance, transportation, telecommunications and manufacturing in China. The deepening of informatization is further giving birth to more new massive data.

According to statistics, in 20 15 years, the total amount of data in China reached more than 1700EB, up by 90% year-on-year, and it is estimated that this value will exceed 8,000 EB by 2020. Taking the banking industry as an example, the banking industry generates an average of 654.38+0.30 GB of data for every 6.5438+0 million yuan of income, and the data density ranks first in all industries. However, in financial enterprises, data is in a fragmented state. Branches such as business lines, functional departments, channel departments and risk departments are often the real owners of data, and there is no smooth * * * sharing mechanism, which leads to massive data being scattered and "sleeping". Although the financial industry has "rich" data, it is "stretched" when it is really used.

Hidden dangers of data security

The essence of big data is openness and enjoyment, but how to define and protect personal privacy has become a legal issue. There are also many risks in the process of big data storage, processing, transmission and sharing, which require not only technical support, but also relevant laws and regulations and self-discipline of financial institutions. Many practical cases show that even a large amount of harmless data will breed various hidden dangers. The object of security protection includes not only big data itself, but also the knowledge and conclusions obtained through big data analysis. Handshake.uk.com, an online market platform in the UK, tried to allow users to negotiate the payment for sharing personal data with brands.

The construction of talent echelon has a long way to go.

Talent is the foundation of big data. Compared with the talents in other sub-sectors of information technology, the development of big data requires higher talents' compound ability, which requires not only mastering computer software technology, but also knowledge in mathematics, statistics and other fields as well as professional knowledge in application fields.