Data analysis capability model

In the previous incremental era, new fields and new markets were being developed every day. Especially in the dividend period of the Internet, e-commerce and other fields, it seems that as long as a single breakthrough, the market can be obtained. In this wild era, business operation is mainly driven by experience and intuition. For example, in the early days of cross-border e-commerce, with the advantages of the world factory platform, it seems that domestic manufacturers can sell well as long as they choose products by experience.

"Now" But with the maturity of the rules and the entry of more players, the market has changed from the blue ocean to the red sea and entered the stock period. The growth model driven by experience alone is no longer effective. Take cross-border e-commerce as an example. Due to the sharp increase of sellers and the saturation of overseas markets, cross-border e-commerce has entered the era of stock operation. There is no absolute blue ocean market, and there are many competitors in every segment. At this time, merchants are required to change from extensive management to refined management, that is, to use data analysis reports to determine whether the market is worth investing, to use data to select products, to use data for business analysis and to use data for inventory management.

Of course, it is not that pure quantitative data analysis determines everything, and experience is not important. It means that in the process of decision-making, data conclusions occupy a larger proportion than before, and business experience is also an indispensable part.

The "future" Internet has gradually become the future of "traditional industries", and data-driven industries such as artificial intelligence and metauniverse are increasingly dependent on data analysis. There are still many manufacturing industries in urgent need of digital transformation to increase the added value of manufacturing links in the global supply chain. In other words, in the future, data-driven services will be more frequent.

The essence of data analysis is "sand table drill": on the battlefield, the commander "deduces" the enemy-enemy situation in front of the terrain model of the headquarters and determines the operational plan; In shopping malls, management "infers" the development of operations through the operational relationship between data, and then makes decisions.

Based on this definition, we can know that the purpose of data analysis is to make favorable decisions for the current operation and development, so how does it do it? In order to answer this question, several key concepts can be deduced from the previous definitions: data, operational relationship, reasoning and decision.

The most common understanding is that data is stored information. From the application point of view, data is a tool to quantify things, and everything can be digitized: numerical values are data, as well as words, images, videos and so on. It is also data.

According to the field type, data can be divided into:

According to the structure, the data can be divided into:

According to the different properties of data continuity, it can also be divided into:

Isolated data often have no reference value, such as quantifying a person's height as 180cm, which does not mean anything. For example, users of Netease Cloud Music, the age of each user is data, and the age of the people who use the product is subdivided, such as 18-24 years old. The index of the proportion of people in this age group is valuable to Netease Cloud Music. The calculation process from data to indicators is the "operational relationship" between data, also called "indicators".

The function of indicators is to "measure" the development of enterprises;

These indicators (points) can be woven into an indicator system (line and surface) through a certain structure to measure local or even global business.

In the "sand table drill", the commander analyzes the enemy's situation and formulates the operational plan through the terrain on the military sand table, the fortifications of both sides, the deployment of troops, and the allocation of firearms. Data abstracts the real operation into a digital world, and helps operators to make decisions through index system and various analysis methods (business analysis, product analysis, user analysis, business analysis ...).

Zhao Kuo was familiar with the art of war, but he couldn't use it flexibly, so he became an armchair strategist. Therefore, after gaining analytical ability, you can't follow the script, and you should make decisions according to the actual business scenarios.

Data analysis involves process innovation, change management, and solving business problems with new thinking. But this process is not forced to change, and needs the help of business understanding and soft ability to make the analysis smooth.

Standing on the shoulders of "predecessors", we can go further. Based on many years' experience in data analysis, Biscuit precipitated a data analyst's ability model, followed it to supplement his missing ability, and finally formed an independent and grounded data analysis ability.

A complete data analyst capability system should include three axes: bottom cognition, business scenario and capability.

Before establishing data analysis thinking, we must first reach a * * * knowledge at the bottom.

What is cognition? It is the understanding of the underlying logic of things, the judgment of everything in the world, and the essence of cognition is decision-making. In other words, in order to help the effectiveness of each decision in data analysis (what indicators and analysis methods are selected? What to do next? Wait a minute. ), you need to establish the underlying cognition first.

In this step, we need to define the definition of data analysis: what is data analysis? Purpose/output? Analysis process?

In the process of finding a job, students will find themselves as data analysts, but the content of the interview varies widely, from examining professional abilities such as machine learning and statistics to market/industry analysis and product analysis.

At this time, some students asked, are these really what data analysis should do?

We literally disassemble, data analysis = data analysis, and further disassemble:

This is the cognitive bias: when some students think that data analysis means making tables with Excel, writing scripts with python and modeling with machine learning (in fact, these are only part of the data analysis ability), the job market has more complete requirements for data analysts.

Looking back, what is data analysis? The author thinks that data analysis is a process, and it is a process of using data ability to do analysis: from finding problems, analyzing reasons to making suggestions; This is also a process of "deconstruction": from the whole to the part, from the general to the special, from the surface to the line to the point, constantly drilling and analyzing to find the specific landing point.

After understanding what data analysis is, think deeply about a question: what is the final output deliverable of this process?

To answer this question, we need to go back to the essence of data analysis: solving business problems. In other words, before deciding the final deliverable, what is the need to go back to the business level:

The most common data analysis scenario is that the business finds that the sales volume drops, users are lost and the product jump rate is high, that is, there are problems that need to be solved at the business level. At this time, data analysts need to intervene to help mine the reasons from the data level and give solutions.

The analysis process may be to do some exploratory data analysis, statistical analysis, machine learning modeling, even AB test experiments, and finally deliver the analysis report, or deploy the model online.

Sometimes there may be no exact "problem" in business, and the more purpose is to improve the effect of existing business models and strategies by deepening the understanding of existing scenarios; For example, at present, merchants use the average customer unit price to divide customers into high and low groups for marketing. At this time, the data analyst gives a more accurate crowd division scheme through the insight analysis of consumers: divide customers into three groups by using customer unit price quantile, so that merchants can use updated strategies to carry out marketing design and improve the conversion effect.

The analysis process may be correlation analysis, regression analysis, or even unsupervised clustering to explain the status quo.

According to the timeliness of requirements, business requirements can be divided into temporary requirements and regular requirements. The first two are temporary requirements of the business, or special analysis requirements. For routine requirements, it is mainly to improve the efficiency of business processes. For example, in the commodity inventory management business in e-commerce operation, the operation needs to query the inventory situation in time and make up the order for low-inventory goods in combination with the sales trend; At this point, the data analyst can help optimize the efficiency of the process by submitting a "low inventory warning report".

The contents supporting diagnosis mainly focus on automatic reports and even the construction of business intelligence (BI) system.

If the previous analysis is based on known patterns, there is still a need in the business, that is, exploring the unknown. The most typical scenario is to give the strategy of brand and business growth after gaining insight into the market and consumers.

The analysis process is more industry-based and market-oriented, using business analysis models such as PEST, SWOT and Porter's Five Forces.

So far, we know what data analysis is and the final deliverable output, so how is this process realized? From the perspective of landing, data analysis is a process from divergence to convergence: business understanding-data exploration-analysis model-landing delivery-product life cycle.

Data analysis is a process from business to data and back to business, so understanding business is the starting point of data analysis.

Senior data analysts' suggestions that "there is no analysis without scenario" and "analysis without business scenario is hooliganism" all illustrate the importance of business scenario. The business scenario model in the data analysis capability model: user-product-scenario is designed to help readers understand the business scenario, so I won't repeat it here.

I wonder if readers have such an experience? Is it that the leader gives you a task, or a friend asks you something, and a person with strong execution quickly completes the task requirements, but in the end he is told that the result is not what the other party wants? This situation often happens to freshmen who have just entered the post of data analysis. In the final analysis, the reason is that the problem is not well defined!

After understanding the business scenarios where the requirements are located, we can use the logical tree tool to disassemble the problems, and the disassembly process should follow MECE and "mutually independent and completely exhausted" gold and pyramid principles as much as possible.

If the question defined above is what to do clearly, how clear is this step?

For example, in the face of the decline in sales, do data analysis and finally produce a data analysis report, or do you need to intervene in testing experiments and give growth strategies? If it is the latter, how much is the increase in sales worth? Is it anodyne 1% or should it reach significant 10%?

If we don't think about the value level and put it into action, it will easily lead to the soul torture of "where is the value" and face the risk of being optimized.

In the business understanding stage, we communicate with the demand side from the business level, but the core part of data analysis is carried out at the data level. Therefore, before formal analysis, we need to transform business requirements into data requirements, and this process is data exploration.

When you get the business requirements, you need the assistance of data: use data to see the business and judge whether the current situation is consistent with the description. For example, merchants say that the decline in sales needs to be analyzed, but who is this decline compared with? Compared with competing products, it decreased month on month but increased year on year, and decreased year on year.

This step is to use exploratory data analysis, or to analyze the data state through commonly used statistical indicators.

If the first step is to verify the validity of requirements with data, then this step is to really turn business problems into data requirements.

In addition, it is necessary to judge the data quality and the feature engineering that can be done. For example, the high missing rate of some fields will affect the construction of features.

After understanding the business and defining the data requirements, you can choose the appropriate weapon (analysis method, model framework) to fight.

To sum up, there are four analysis methods:

Whether the indicators are good or bad and whether the characteristics are significant can be achieved through comparative analysis. For example, the common attribution business scenarios, the essence of which is to compare horizontally and vertically to find out the reasons.

Analysis methods: such as t-test, variance analysis, year-on-year comparison, cohort analysis, etc.

Analyzing the correlation between variables is an important analysis scenario. For example, do you want to know whether increasing the advertising budget can improve sales performance, or even how much? This correlation analysis can find the best configuration scheme of ROI.

Analysis methods: Chi-square test, Pearson correlation coefficient, Spearman correlation coefficient, structural analysis, etc.

Both the prediction of enterprise sales and the prediction of user behavior can help improve business efficiency, such as the common analysis of predicting user churn, getting the list of people with high churn probability in time, and improving user retention rate through pre-marketing intervention; Common sales forecasts can help enterprises prepare for the supply chain. This kind of scenario mainly applies the supervised classification model in machine learning.

Analysis methods: linear/logistic regression, decision tree, time series analysis, Bayesian, etc.

The first three are all based on the analysis logic of the enterprise's known patterns, and there is also an analysis method-unsupervised machine learning model, which can deal with the analysis of unknown patterns. For example, you don't know how many groups the existing crowd should be divided into for marketing. You can do unsupervised cluster analysis on the crowd based on the core characteristics and get the boundary of effective grouping.

Analysis methods: Kmeans clustering, DBScan clustering, etc.

The best practice of delivery is to let data and analysis penetrate into the business from theory, change the process and improve efficiency.

Before delivery to the enterprise, it is necessary to evaluate the effectiveness of a given solution:

If the analysis involves the development and use of the model, the effectiveness of the model at the data level needs to be proved by AB test, ROC and other indicators. After the verification is completed at the data level, return to the business analysis requirements and evaluate the effective implementation of the delivered solution at the business level.

Data analysis revolves around business value, so in the final landing, we will discuss value and answer the way and degree of this solution to solve business problems:

A. Is the way to optimize the process (reducing costs and increasing efficiency) or the data (data system efficiency and data quality)?

B. to what extent can this method help solve the problem? For example, is the business promotion 10% or 30%? Is it the application of a single project, or can it be deployed into the daily process, which will affect the business in a longer time and in a wider scope?

C. In addition, what resources are needed to achieve this effect?

Project landing needs multi-party participation analysis. Even analysts with rich business skills cannot participate in every step because of the existence of process boundaries. Therefore, an important factor to ensure the effective landing of the project is whether it can reach an understanding with the business.

How? Tell a data story: Is the cause (requirement definition), process (analytical logic) and ending (important conclusion) fascinating (recognized)?

This process requires PPT reporting, business communication, and even cross-departmental speeches.

Whether it is a business model or an algorithm model, there is a process of "boots landing"-landing implementation. After the model test is effective and the business has reached a * * * understanding, the deployment and online stage of the model will arrive:

At the end of analyzing the life cycle, the product life cycle is analyzed: data analysis is viewed with product thinking, and the model application delivered to the business is the product. The process of data analysis is not static and single, but the process of continuous iterative upgrade of PDCA. (The definition of analysis products includes analysis services and data products. )

From the perspective of product thinking, the analysis conclusion falls into the business process, and the process reengineering improves the operational efficiency.

When the data analysis process is mature, a large number of repetitive processes can be extracted to form an automated product for service data analysis (mainly for data analysts, including operations), which is the data product. The analyst's conclusion model can be deployed to existing data products to optimize the analysis efficiency.

The reason why we should look at the data analysis process from the perspective of product thinking is that we should iterate the analysis model like iterative products: whether we optimize the algorithm parameters or adjust the analysis framework, we can get better conclusions.

In the first step of data analysis life cycle, "Understanding Business", we mentioned the importance of business scenarios.

Based on business experience, the author precipitated a set of easy-to-understand models: business scenario = user product scenario.

In other words, to understand the business, we must understand the users, be familiar with the products and analyze the context clearly. They determine the analysis objectives, processing logic and landing suggestions.

For more detailed discussion, see: Return to marketing theory and talk about what is a business scene?

After you have a basic understanding of data analysis and business scenarios, you need to have tangible "moves" to act: the three axes of thinking method, tool technology and project ability can form different moves to deal with changing problems.

People often say that data analysis is like cooking. If so, in the kitchen of data analysis, the tool technology is spatula, iron pot, spoon and other utensils, the thinking method is cutting, cooking, plate loading and other technologies, and the project ability is the last dish on the table.

Many people learn to cook, probably because they saw a food video in Tik Tok or Bi Li, and then began to prepare food according to the video steps. This process is also a process of learning thinking methods in data analysis. Data analysis is also a way of thinking, which can be said to be analysis.

When I first learn to cook, I usually learn the basic cooking methods of frying, frying, frying, roasting, boiling, steaming, stewing and mixing. These basic abilities are general analytical thinking in data analysis, such as statistics, correlation analysis and attribution analysis.

Just as food has eight cuisines to meet the tastes of people in different regions, data analysis has different "analysis" moves to meet different business needs in different scenarios:

After learning cooking methods, you can choose several handy utensils to improve cooking efficiency.

The reason why we don't choose utensils first and then study the cooking process is because tools are always tools. There may be many tools that can achieve the same goal. I can cook with the original earthen stove.

However, for some complex cooking needs, it is also necessary to choose specific utensils to complete.

Common tool technology and its application;

After cooking, you must do it in time, plate it and serve it, otherwise the delicious food will only be in the attic.

Project capability emphasizes the landing of data analysis projects. How can theoretical analysis methods empower business scenarios and reflect data value? This is a topic that many enterprise data teams are discussing.

Not strictly speaking, the project ability is like the last serving stage of cooking, because the landing ability is a soft ability, which runs through the whole process of analyzing the project:

Author: biscuit brother author